Embedded - 62: Costs a Penny to Go to the Bathroom

Episode Date: August 6, 2014

Josh Bleecher Snyder (@offbymany) joined us to talk about PayPal's Beacon, being acquired, the Go programming language, BTLE, computer vision, and working at a large company after founding small ones. Bluetooth Low Energy: A Developer's Handbook by Robin Heydon TI CC2540 BTLE module Learning OpenCV: Computer Vision by Gary Bradski and Adrian Kaehler Gatt is a Go package for building Bluetooth Low Energy peripherals (video description by Josh from GopherCon 2014) Card.io Machine learning's Theano Eigen Library for matrix math

Welcome to Embedded, the show for people who love gadgets. I'm Alicia White, here with my co-host Christopher White, and our guest Josh Bleeker-Snyder, Director of Software Engineering at PayPal. Hi Josh, thank you for coming on the show. Hi Lisa, thanks for having me. Could you tell us a bit about yourself? I'm a software engineer at PayPal now. I came in through an acquisition a couple of years ago,

a company called Card.io that did early mobile computer vision. We made a product that let you hold your credit card up in front of your iPhone's camera, and it would scan the information off and help with payments. Scan the numbers? Yes. Hmm. That's kind of cool. Yes.

I don't even have computer vision to ask you about, but now I have a bunch of other questions. Okay, okay, go on. And Cardio was acquired by PayPal. Yes, two years ago. And since landing at PayPal, I've worked on a variety of products, mobile SDKs, which were a natural follow-on to Cardio. And recently Beacon and a bunch of time I've spent with Go, the programming language. And so it was Beacon that I invited you on to talk about. What exactly is Beacon? Beacon is a small piece of hardware that a merchant can plug

in at their store and enables this sort of magic consumer experience of walking into a store and being able to pay for something without ever having to put your hand in your pocket or your purse to get out your phone or your credit card. And it's powered by Bluetooth Low Energy. It sounds kind of like stealing. Well, it's not stealing if you pay for things. It's facilitating the transaction. And the merchant is still involved at every point. So it's not like you walk in, you grab the stuff off the shelf,

and PayPal magically knows that you've picked up a bag of chips, and then you walk out again. Instead, you go to the counter, or maybe you order at the counter. And then when it comes time to pay, instead of reaching for cash or credit card, you're already checked in at the store, which means your face and your photo pop up on the merchant's screen, and they can confirm that it's you. So it's basically like any other regular... Okay, yes. So that makes sense.

People spend a lot of time trying to make NFC do similar things. This seems like it has a lot of advantages because you don't have to have a reader that you have close proximity to, right? Yes, exactly. This was PayPal dabbled with NFC, and we felt that in the end there wasn't a real clear story. And Bluetooth Low Energy makes it possible to have a much more exciting customer experience. And you can do other things with it as well. For example, you could walk into a hotel and be ready to be checked in.

The additional time also, when you check in, as you enter the store, as opposed to at the moment of tapping your NFC, you have the opportunity, for example, for a barista to fire up your favorite drink. All the automatic things that can happen. Exactly. And, you know, it's not like anybody really enjoys waiting for their check. No, decidedly not. Okay, so know, it's not like anybody really enjoys waiting for their check. No, decidedly not. Okay, so now with the technology. You said Bluetooth Low Energy. Yeah, BLE is the thing that makes Beacon possible.

And we've talked about Bluetooth Low Energy on the show with Internet of Things sort of bent. And I guess this qualifies. This is an Internet of Things, except it's your wallet that you're replacing. Before the show, we got to talking a little bit about Bluetooth and all of that. What makes it interesting for you? Well, BLE is really exciting because it has a sort of magic combination of characteristics that no communication protocol has had yet.

It is low power, so you can put it everywhere and have it always on on your phone, which is huge. Anyone who's fought against backgrounding restrictions on iOS or battery problems in Android? Yes, Android. I keep trying to say Chrome, and I know it's not Chrome. It will be soon, probably. Probably will. Knows how amazing it is to be able to run something all the time in the background without it destroying your battery.

So it's always on. It's ubiquitous at this point. And it provides ad hoc two-way communication. So I'm near you, I can talk to you. And those combinations mean that you can do things that you simply can't do with earlier Bluetooth, with NFC, etc. I'm not sure I agree about ubiquitous. Do you? It's getting ubiquitous. Certainly every iPhone has it and most newer Android phones do.

Not everybody has a smartphone. Well, but people who don't have smartphones aren't going to be interested in mobile payments. At least not right now. All right. And so you have to design for all of the phone platforms that are, I mean, you have to write software apps. Yes, we do write for iOS and for Android, not Chrome.

It turns out actually that the hardest challenge is not writing across all of the different platforms. It's about dealing with the OS-specific quirks. So there's a bug in, not just in 7, but in 7.1.2, they got fixed in 7.1.3, and how do we work around that? And iOS actually has a much more stable BLE stack on the whole than Android does, and Android tends to be device by device. The good news is we don't have to deal with fragmentation per se, because either it works

or it doesn't. And if it works, we're off to the races, and if it doesn't, we pretty much have to wait for the manufacturer or Android to get around to actually fixing the Bluetooth app for that particular chip for that particular phone. Because that's what you depend on most. Yes. BLE is the thing that makes it tick. Do you use anything like AppCelerator to do cross-platform development? We don't.

PayPal has whole teams dedicated to making wonderful, beautiful wallet apps. And we are just a piece of that. So we focus very specifically on all the Bluetooth communication. And that we have had to build per platform, but large markets like AppCelerator and PhoneGap and the like, they tend not to go into the deep integration

with the hardware pieces specifically because they're very platform specific. With iOS, you're tied into the deep integration with the hardware pieces specifically because they're very platform specific. With iOS, you're tied into the backgrounding framework, which there's no comparable equivalent in Android. With Android, you're dealing with the peculiarities of the stack. So there's really no sense in which an app seller actually is going to care or invest in making it easy to do this crop platform. So we've had to do it. But it's only two. It hasn't been too painful. Well, the test matrix alone is the

difficult part. It's all of the different varieties of hardware, all of the different varieties of software that run on all of the varieties of it gets exponential for testing. It's true, but we have the world's best fuzz tester which is a whole bunch of PayPal employees who walk around with phones in their pockets.

So we set up a beacon and we find out when it doesn't work. People tell us. And then we're like, okay, what are you running? When were you there? And it's basically a game of whack-a-mole. Why are you, what are you charging them for? Is this, it costs a penny to go to the bathroom or something? Because that would be hilarious.

That's brilliant. We should totally institute that. No, we don't actually. So what PayPal provides initially is this automated check-in. Merchants can then decide to charge, or as a hotel, you might decide not to charge at all. What we facilitate is this

secure presence. So we know for sure it's you, and we know for sure you're here. So we can actually have folks just check in as they wander by the lobby and not necessarily do anything with that. So that's a little scary. Good. I was hoping that I was going to get the, oh, creepy response. Because this is actually really near and dear to my heart. I generally believe that it's really important as an engineer that you feel proud of and what you build and that you think it's actually sort

of a moral force for good, or at least not a moral force for bad. So Hasty, who was the original progenitor of Beacon, who I worked very closely with, Hasty and I both feel very strongly that we do not want to creep people out. And this is one because we think it's important, but also for trust reasons. If people think it's creepy, they turn off Bluetooth and they miss the opportunity to have this awesome experience. And then we've built this thing and there's no point to it because everyone thinks it's creepy. So I can tell you, if you don't mind going to the weeds a little bit, some of the things that we've done thinking about privacy. Oh yeah, it's all weeds here. Go on.

Excellent. Yeah, this will head off some emails. Perfect. So the magic UI, the story goes, hey, we make the UX disappear. You walk into a store, you walk up to the counter, you say, I'm Josh, they give you your coffee and you're off to the races.

That's actually not quite how it works. Yeah, I saw this video where she was out running and then the coffee person like met her at the door with her preferred coffee. It was a little creepy. Yeah. So here's what it's actually like the first time you walk into your favorite coffee shop. You walk in, your phone buzzes and it says, hey, you know, you're at Faze, which is my favorite coffee shop in the city, do you want to check in? And at that moment, control is in your hands. You can either say, yes, I am willing to have them know that I'm here, at which point you've signed on and it's no longer creepy. Or you can say no, and I'll talk more

about that in a moment. Or you can ignore it. The ignore and the know both have interesting privacy challenges. So suppose that you ignore it. Well, if we know that you've ignored a notification at FaZe, and then you've ignored a notification at ByWrite, and then you've ignored a notification at Boulampi, we might be able to de-anonymize and be like, oh, well, there's somebody that's tracked, blah, blah, blah. So we are very careful to, we know that someone was there.

We don't know that it was you. We cannot even re-identify in an anonymous way that it was you. And actually, we have the Bluetooth SIG to thank for this. Part of the Bluetooth Low Energy spec is that your MAC address gets rotated to a new random MAC address every 15 minutes. So if you ignore it, someone was there, but Lord knows who it was. So there's a bit of privacy there. Well, every 15 minutes, does that mean it re-asks you?

No. So if you totally ignore it, yes, it will ask you again if you hang out in the coffee shop. So if you say no, that's a great segue. What happens if you say no? If you say no, we have to store that in order to be respectful, to not bother you over 15 minutes. But it'd be careful.

The natural thing to do is to sync it up to the server. We store it only on the phone. This means if you lose your phone, you're going to get harassed everywhere again. Like, hey, do you want to check in here? Do you want to check in here? But it's better than PayPal servers containing the information that you were there

and you said no, because then you've effectively lost that privacy. So we've been very careful about that. Well, and you've lost the privacy in places you don't trust, which is the worst place to lose it. Exactly. So there was a battle, but we fought it and we won. And it's very clear now, that sort of information does not leave the device. What other privacy protections are you working on? Let's see. So the key ones are around know and ignore.

Well, what about if I lose my phone? That kind of, I mean, now somebody can just go buy whatever they want. I don't even, I mean, I guess I call my phone provider, I call PayPal. How do I deal with that? Yeah, so I believe you can probably lock your account online. It is also the case that we're probably actually not the most valuable thing that you have on your phone. So the usual mechanisms you would for securing your phone, your PIN code, your passcode, please, everybody, if you don't have one, put a passcode on on there um only so when you get arrested they don't call your mom exactly um we have in mind um some features that um will let you facilitate smooth transactions when the dollar value is small um and then sort of step up to a

better authentication when it's larger um Those have not been built yet, but they're definitely something that we have have in the foremost of our front of our mind. I can go into the coffee shop. I can spend five bucks and that's always fine. But if I'm ever going to the coffee shop and spending 3000, please put up a warning. If you're spending $3,000 at the coffee shop,

I'm going to come help you anyway. I'll take everything. Hey, you heard about that Starbucks contest, right? Who can make the most expensive Starbucks drink? If you buy enough stuff, they give you a free drink, and it can be anything. People online are trying to get the most expensive, and it's up to like 50 bucks. By the time you put in all of the frappuccinos and the... It's also completely inedible well yeah

it sounded disgusting i mean the 45 shots of espresso alone was disgusting but by the time you add in the gold dust and the saffron okay so so there are limits but those are coming soon isn't all of this kind of coming soon is it yeah so we have some actual beacons out in the world where we can get foot traffic and not just on the PayPal campus. We're doing a slow rollout. And the slow rollout is for a couple of reasons. One, we want to make sure that we nail the user experience. We get it just right.

And if you screw this up in the field, you have a brick. And bricks aren't much good to anybody so why do you say that um well interesting question how do you set up one of these um well it's a little bit of setting up the beacon not not the phone yes sorry the phone is easy it has a screen and you can tap on it but the beacon is this like little black triangular thing and you can stick a you know a ballpoint pen in the reset button, but that's it. So in order to set one of these up, we use BLE, of course. But in order to use BLE to configure one of these things, BLE has to be working. The OS has to be

up. You have to be able to connect to it. All the services have to be running, which means that if we screw something up really badly, you can get to a state where you have an unusable very adorable triangular piece of um industrial design doesn't have a debug port or exactly did you consider an led one that's green and red we have an led it gives you statuses but um and we actually thought about having Morse code. Oh yeah, I've done Morse code in LEDs before. We'll just reprogram the firmware. Exactly. Call me in three days.

Yeah, the challenge is more making modifications than getting data out. And are you updating in the field? Of course. That's a tricky proposition. I guess you have to have Bluetooth for the customers and then some other back-end holler. Is it Ethernet or Wi-Fi?

Wi-Fi. We use the merchant's Wi-Fi. And this is because we want this to work everywhere. We want it to be that if you get a beacon and you set it up, it works. And if you're in, say, Grand Central Station in one of these underground shops, if you're in, say, Grand Central Station in one of these underground shops, if you're relying on the user's cell phone having some sort of signal, you're hosed. So we need some reliable way to get to the internet and the PayPal servers. And for that, we use the merchant's Wi-Fi. And so you have to set up the merchant's Wi-Fi on the Bluetooth low energy

connection that you have, but you can only do it when the merchant wants to and not when some strange person walks into your coffee shop and tries to hijack your beacon. Exactly. And that was a really awesome bit of cryptography and security design fun. Fortunately, PayPal has a pretty good bench of folks who think about security all day. So we got to tap them. But it's actually a really fascinating cryptanalytic challenge, which is we have like three parties here. We've got the merchant's phone, we've got the PayPal servers, and we've got this beacon. And you have sort of general distrust.

I think you have another in there. You've got the merchant who is setting this up the owner merchant and then you have the cashier who is not necessarily trusted yep okay exactly and in fact you can't trust the phone and you can't trust the beacon the only trusted party in this whole story is the paypal server and you might not be able to reach the server if wi-fi has changed um So you need to figure out a clever way to get signatures that are offline validatable at all times by all parties so that you can make a unilateral decision as the beacon, say, I got this request,

they want to change something, something, something. In order to be able to trust that, it has to have been signed in a reliable way by the PayPal servers. And the same thing is true for the merchant's phone. I can't go into the details with you, but it turns out you can. In part because we need a whiteboard.

Exactly. But thank goodness. Cryptography is really amazing. And you can do all of this. The real challenge is figuring out how to compose the cryptographic primitives in a way that doesn't leave you open to side-channel attacks and to replay attacks and the like. When I've done small amounts of cryptography,

nothing on this level, but it's always been about trying to come up with what all of the attacks can be, to be the bad guy. Yep. Do you spend a lot of time? Doing threat modeling? Absolutely. modeling absolutely that's the word yes threat modeling and uh the the other interesting part is figuring out which threats are important and which aren't which really requires you to think about the entire product

experience so here's a threat model that we actually worry about a lot um can i snoop on the traffic of the beacon and then um pretend to be the FaZe beacon in the middle of Times Square, causing phones to wake up and tell the customer, hey, you're at FaZe, do you want to check in? You've actually not done any harm in any sort of direct monetary sense, but you've really hurt the sort of user perspective on, is this reliable? Is it safe? Is it trustworthy? So that sort of threat we really worry about a lot. Other threats. I'm trying to think of an example of a threat we don't worry about. Can you say check in five times right in a row? Sure, knock yourself out. Yeah, it seems like that would be hard to come up with a way of monetizing the benefit there.

Exactly. So you think about what you care about. You care about reputation, you care about trust. Yes, and we actually have interesting denial of service pieces put in place that I can't talk about for obvious reasons. But with something I can say, one of the fun things about the denial of service is we were worrying about it initially from a security perspective. It turns out it's also really useful as a developer.

It's really easy when you're hacking on these things to DOS yourself. And I don't know how many times I have been saved by our own DOS protections. Well, I mean, you've got to have, you know, a whole little stream of cute little pyramids across your desk. And you're trying to get them all working

and testing them all with different things and different phones and here and there. That's always a problem with BLA projects. I've worked on an office building and there's hundreds of them. And this doesn't really work very well anymore because the radio spectrum is saturated. Manufacturing is also fun when you're trying to, to just test that they all can connect,

except you're not talking to the one you're looking at, because there's no other way to tell. So the discussion we had about security around setup, before we had that, in the early phases, people were constantly setting up each other's beacons, and it caused total havoc. And it was like, we need a Faraday cage. And finally, actually, I convinced them we need a Faraday cage,

so I think we're going to be building one. Because there's times when you just need a Faraday cage. Yeah I think we're going to be building one because there's times when you just, you just need a Faraday cage. Yeah. You need to be alone with it. In part, because I've had some clients who got so used to the it's everywhere environment that they forgot. Most people are doing it on their own and having a list of here are your

possibilities. And they're all in these hideous Bluetooth Macs. Well, and if you're random in these hideous Bluetooth Macs. Well, and if you're randomizing the Macs, you're totally host. How do I know which one was this? I have no idea. Back to Bluetooth, so we get off of security. You told me a little bit about GAT.

Can you talk about that? What is it? Yeah, so if you're a developer who's starting to tinker with Bluetooth Low Energy, the level of the protocol with which you interact is GATT. And GATT stands for Generic Attribute Profile. So there's a lot of... These are those XML files.

Depending if you've been working with some BLE systems, yes, they show up as XML files. They don't necessarily need to be, but yes. No, of course, they get compiled. But a lot of the transceiver vendors are putting them XML that you develop on your always-a-stupid-Windows machine. Yes. And then you compile them.

Although you don't need a stupid Windows machine, actually. If you have a Raspberry Pi and you can buy a little USB BLE thing to plug in. I wrote a quick open source project because I was really frustrated with people getting excited about BLE and not being able to do anything about it. So at github.com slash paypal slash gat

if you can write a little bit of Go and you have a Raspberry Pi nearby you can do without Windows. It's pretty easy and quick to get started. Anyway. Okay, sorry. So BLE is full of new terminology. So profile is BLE speak for a use case.

It's all the things needed to make something work. And if you think back to BLE 3 and before, it was all built around very specific profiles. I want to stream music. I want to receive phone calls, etc. BLE is much more generic, lets you build anything you want. And so, of course, we have a generic profile. And the attribute part is a nod towards how it's set up and configured.

So here's a bit more terminology. In the GAT world, you have peripherals and centrals so your stereotypical central is your phone it's the thing that um you think of as a client it discovers peripherals in the world your peripherals are your servers so your phone connects to a peripheral and it says um i want to peripherals are your servers your peripherals are like your fitbits not like your internet servers. Exactly. They're serving small amounts of data. Exactly.

Okay, sorry. Good. What your peripherals advertise are services. So I can do, I know about these sorts of things. And your service is nothing but a collection of characteristics. And a characteristic is just a value. It's small, it's like 20 bytes or smaller, and it's just a value it's small it's like 20 bytes or smaller and it's just a value so to take a stereotypical example if you think about a thermometer a thermometer might have an ambient temperature service and a battery service and then that battery service might

consist of the current battery level which is a characteristic it's a value and it might can also have say an on or an off characteristic. And you could write the value, you know, zero into whether it's on to turn the thing off. Of course, it would be hard to write the value one in because it would already be off. Examples are hard. Anyway, so you've got this sort of layering. You have services are bundles of characteristic characteristics or values. You have services, our bundles of characteristics, characteristics or values. You can read, write, or receive notifications for these values. And that's it. That's the entire ontology you have to work with. So when you go to build something at Bespoke that uses BLE, you have to figure out how to make it fit into this world. But you're building the peripheral.

Yes. But the peripheral advertises services. So we have to decide what services do we advertise, what characteristics are associated with them, and what do all of those things mean. And I can tell you a little bit about what we did with Beacon, which is actually an abuse of BLE, but it works great. A lot of my BLE experience is using BLE as a serial port, which as you talk about characteristics and I'm like, oh yeah, I bet each one of some meaning like your current heart rate and you just read it or the current time.

But we want sort of arbitrary communication back and forth. So we have a characteristic that is like, hey, you can write into this characteristic. And if you as you write multiple times, it concatenates them And you can receive notifications when this other characteristic changed its value. And that's a way to get full duplex communication back and forth. Of course, the tricky part is that duplex communication, it's a lot like UDP. You don't necessarily have reliability. You don't know anybody got it. Exactly.

So you find yourself pretty quickly building a network stack. Only you can't just reuse standard network stacks. Your MTU for one of these things is 20 bytes. Your TCP header is 22 bytes. So you clearly can't use TCP with these things. So you end up inventing your own network stack, which is an absolutely terrible idea. It really is a terrible idea.

But when you're counting packets and you want to get a whole check-in completed within three round trips or four round trips, you really are paying attention to every single bit. And all of a sudden, you are designing an ad hoc network protocol that fits your exact needs and nothing more. Well, that's what embedded systems are about. Exactly. Fitting your exact needs. And that's why we get to optimize them. Indeed. One fun thing I can tell you, though, and something that I'm actually quite proud of, is we built a traditional network stack on each side. And for high throughput cases, like check in, where you want to be able to check in people, you know,

123, as they walk through the door, we work at a fairly low level but in cases where we want flexibility like setup we want to be really easy for um as we work on the user experience we want to be really easy for the ios folks and the android folks and then also the firmware guys to make it um straightforward to experiment so we built this network stack all the way up to the application layer and when you work on um secure check-in as an iOS developer, you make an NSURL request and an NSURL connection. And we grab that and we handle the whole network stack down. And on the beacon side, which is written in Go, you write a regular old HTTP handler. And we have set up interfaces that handle the whole network stack so on each end you just write http and you don't think about the fact that it's not running over tcp ip

um and it just works no it's pretty cool let's ignore that part which is what users want to do they always want to ignore the tricky bits exactly not everybody's used to interacting with network devices over urls now for some reason so So might as well make it look like everything else. Exactly. Sockets, what are those? So you mentioned Go. That's a programming language. Yes.

People usually say that Go is a young programming language, and it's actually starting to not be that young anymore, which is very exciting. It came out of Google. I think its initial public release was in 2009, which I guess is a little young in programming language years. And Go was aimed to hit some of the challenges that you have as a modern developer. So you need good support for concurrency. You want it to be easy to write, say c++ but you also want it to be very efficient and cpu friendly unlike say you know a python or ruby or an interpreted language you want really good

tooling around it and a good ecosystem perhaps the most distinctive thing that people think of when they think about go is a go fumped which is an automatic formatter so there is no style discussion there is no argument there's also no human tweaking. You write your code, you run it through GoFumpt, and it's formatted in the one true canonical style. And that's that. It's absolute delight. It sounded like it was an experimental language initially as a derivative of C, that was a different path than C++. Yes. So I think the original story was Rob Pike was like, we want a better C.

It clearly has its roots in the C family, although it pulls in aspects of other programming languages. It looked a little Python-esque, too. It uses CSP for its concurrency model. It's object-oriented-ish. It's object-oriented in the sense that you can associate functionality with data, but there's no inheritance. There's nothing of what you think of as traditional classes.

So it sort of sits in this funny space. It's more focused around, if you think from a Python world, it's more around duck typing, statically enforced duck typing, than it is around a full OO stack. Do we really need another language to learn? Yes. Hey, Apple just added another one too.

Everybody's doing it. It's really delightful. So Brad Fitzpatrick, who is on the Go core team, the thing he likes about it, he says it's 90's really delightful so um uh brad fitzpatrick um who is on the go core team the thing he likes about he says it's 90 perfect 100 of the time you never have to think okay i'm gonna write it in c++ it's gonna be super efficient um but oh man i'm gonna have to maintain it and you never think okay it's gonna be really fun to write and i'm gonna write it in you know python ruby pearl blah blah blah and but what

happens when you know i hit high concurrency and um my you know my cpu is churning all the time there are none of those trade-offs you have to make it's almost always good it's always good it's maybe never perfect but it's always good and it turns out that's a really nice trade-off 100% of the time it's 90% good actually is a good trade-off it is better than having it be 90% of the time it's 100% good because that other 10% is what kills you I find, personally, maybe this is just me, but I find that embedded folks, maybe just me, would rather use C or even C++ if we're forced to. But higher level software engineers are more willing to change languages depending on the problem at hand.

And so, I mean, I know a little bit bit of python i used to know a lot of pearl awk said let's not go with fourth and four tracks i'm probably better at those a couple of assemblies and yet i still am resistant some of that is because when i see the python people sometimes they're super nice and inclusive and sometimes they're're like, you should learn our idioms. And I'm like, what the hell with you? You should learn what a pointer is. Well, okay, now Go has the worst of both worlds. You've got pointers, and you should learn the idioms.

I thought the point of all modern programming languages was to get rid of pointers. Well, they don't allow pointer arithmetic in Go. Okay. So a lot of the the increments that go bad is is disallowed right exactly so you don't have pointer arithmetic you have memory safety um you have type safety um so you can't just willy-nilly um cast things back and forth and um you also have garbage collection so you don't have one of the other sort of pain points around around pointers um but pointers let you express at a low level, very important data structures. And actually,

one of the tenets of Go is the code that you write to sort of going and writing code should run very efficiently. But if you need to squeeze performance out of it, you have the tools to. So with Java or Python, everything is boxed, and there's no way to get around it. But if you just really need values, for example, you are dealing with a highly concurrent system, and you want to keep all of your data in cache local to a processor, you can't go chasing pointers everywhere. So with Go, you have the tools you need to write really efficient code, which actually is one of the things that made it a great trade off for the beacon. Embedded ends up being a lot of concurrency, Go is very good at concurrency.

But of course, we're running on like a really, really low powered ARM device and they keep underclocking it. They're like, can we just turn down the CPU a little bit to help with thermal? And then a week later, like, okay, we turned it down. I'm like, all right. And so far Go has kept up with it.

And it's specifically because in the performance critical sections i can write code that compiles down to very very tidy machine code i mean you just simply can't do that with a python even though go doesn't have as big a user base there's still people writing compilers that are good enough to do this and is it because it's c-based good enough is the keyword so um goes code generation is not at the level of gcc or llvm and it's because of a very conscious trade-off so the go team thinks that one of the really important aspects of a developer experience is compile time and llvm and gcc spend a lot of cpu cycles squeezing every last bit of performance um go has a non-pessimizing compiler so what as opposed to a heavily optimizing compiler

it doesn't do stupid things all right so it's a non-pessimizing compiler it compiles promise not to break your code but exactly what was the word word again? Non-pessimizing? Pessimizing. The opposite of optimizing. Exactly. So you can compile very large Go programs very, very quickly. You know, seconds. Which might take minutes or hours.

But on the flip side, you know, you might run 10% slower because we haven't squeezed every last instruction. Well, it's not just speed either, though. Optimizers optimize for space as well. I mean, so for an embedded solution, that might be a real problem. Absolutely. It was one of the early conversations we had around using Go. Go binaries notoriously start large,

and it's because they're a single statically linked binary. It contains its own runtime, its own garbage collection. The great thing about this is I can cross compile on my laptop and SCP a binary over and I am independent of what version of libc is there and what version of the kernel is there. It just works, which is delightful. But on the flip side, the binary starts at a couple of meg.

Well, by the time you add some significant stuff in. And so we made a decision early on that, well, people say that hardware is cheaper than software. Software engineers are notoriously expensive to hire, but... Yes, and sometimes getting another couple meg of flash or RAM is worth it. Yes, exactly. And we made an early decision that said, you know what? The important thing is to have it work, to get to market.

And if it means an extra 50 cents on your bill of materials, let's pay that. We can always optimize that out and tweak it out later. Yes, and it is better. I mean, if you want to get to market, then you get the expensive hardware, you prove your market out, and then you say, okay, all the nickels that are sitting on this board, we're going to start taking them off, giving them to software engineers. Exactly. And the idea that someday there will be more nickels than software engineers.

And you pay attention. Some of those nickels are right next to the steamroller, and some aren't. So you pick up the ones that aren't next to the steamroller. Yeah, you take the safe ones or the good ones. So you were talking about how Go is very good at concurrency. What are you doing that requires concurrency, really? Well, let's see. We've got a BLE chip.

We're talking to it. So first of all, the beacon runs Linux, the firmware that is in Go runs in user space. It's a regular old program. It talks to the BLE chip. It talks to the Wi-Fi chip. It manages multiple connections on both halves. And so you are effectively designing

a heavily concurrent system. And in fact, the beacon is relatively dumb. It has some intelligence around these network stacks and a minimal understanding of the domain. But for the most part, its job is enforce enough security and then get the data back up to the PayPal servers. A, they're the ones we trust, and all the important data comes through encrypted anyway. And B, it's much easier to update a server on the fly. So the beacon's relatively straightforward and simple as a code base, but its whole job is to manage a bunch of these things

and make sure that we keep it all straight. It is one big exercise in concurrency. All right. All right. I get that. So going back to Bluetooth again, because I keep wandering off. What do you use for Bluetooth tools? I mean, do you have a good sniffer? I've done a couple of Bluetooth projects and I haven't always been thrilled. And so often we end up as serial ports, which is a lame way to use Bluetooth. Totally. Yeah. We recently got a really fancy sniffer, but for a long time, we got by very effectively. I believe that the TI2540

is the tool, and you can plug it into, yes, a Windows box and fire up this free bit of software that talks to the 2540 and turns it into a sniffer, and you can see every packet that comes over the air. And if Windows is not your gig and fighting with UIs is not your shtick, you can export them in this sort of fixed width format and then you can do a bunch of manipulation. So our sniffer effectively consists of a Windows VM that you plug in the 2542, you gather a whole bunch of data, you dump it,

and then you run these sort of hacked together Go scripts that parse out the interesting bits you want and present them to you. And having a good sniffer has been absolutely critical to solving real problems in the field. Partly because the phone stacks, Android and iOS, don't give you a lot of insight into what's actually going on. And then on the other side, if you have a problem on the beacon side, it's just as likely that whatever your firmware is reporting to you is wrong. So you have to actually see what goes over the air. Once it's broken, debugging is toast.

Exactly. Wow, I like the idea of using this little system on a chip. It's a Bluetooth system on a chip with an 8051. Man, those are awesome. So yeah, I will link to the CC2540. And do you have any other advice for people who are starting projects looking at Bluetooth and thinking, hmm, how should I get this started? Yes. So one of the frustrating things about BLE is it's a delightful technology. It's really exciting. It has this big future ahead of it, and there's nothing on the internet about it.

There's no blog posts. There's no stack overflow. You go to learn it, and it's really impossible to get started. First of all, for folks who like to read, there's a book by Robin Hayden called Bluetooth Low Energy, a developer's guide. It goes into glorious and excruciating detail about everything about BLE down to, you know, the hardware frequencies and how they designed it to be robust against RF interference. But it also goes through a lot of the design of BLE in a way that's very relevant. So if you want to read a full book, you'll be up and running after reading that.

Another thing is get a hold of like a $15 BLA USB dongle and play with it on a Raspberry Pi, play with it on sort of an open hardware where you're really focused on what is your service and characteristics and write very simple, straightforward code. This is on the peripheral side. And on the client side, on the central side, on the phone side, there are generic central apps.

I think I use LightBlue a lot. It's a very good, very simple tool that you can use for exploring. You can also use it on your laptop if you have a recent OS X laptop. But if you can get the core of your communication sorted out in these cheap, easy-to-run-with ways, then you can go and fight with the details of the core Bluetooth stack and go and figure out how you're going to take this thing

that, okay, now it works on a Raspberry Pi, now I have to think about hardware and embedded, and that's a very long process. But you want to, for most people, if you can just get up and running and have, you know, a little tiny box that sits in the corner of your house, that's enough. And you can spare yourself a lot of misery fighting with versions of Linux. And can I, you know, set up a Yocto build system? Or do I use Arch Linux or this or that? Hey, that all sounds really familiar. All yak shaving, totally not necessary if you just want to tinker with BLE on a Saturday morning.

Good advice. So changing gears entirely, you were at Card.io and you were a founder, right? That's correct. I was the technical co-founder. And was the other person the non-technical co-founder? Yes. That's kind of cool. It's delightful. He took care of basically everything that didn't have to do with the tech.

I took care of everything that had to do with the tech. And our overlap was around product. And we'd worked together a very long time. We trust each other very deeply. And so it made it very efficient to go and build things and raise and give you an idea of the level of sort of mutual trust. I gave him a build at some point.

I was like, okay, you can go raise with this. He said, okay. And at some point he came back and he said, here's a term sheet, do we want it? And I said, well, do we want it? He said, yes. And I said, great, done. That was our entire fundraising conversation was he figured it out and i said well do we want it he said yes and i said great done like that was our entire

fundraising conversation was he figured it out and i said yes that is nice how did you find him we worked together at a previous company and um he's about the best hard-working man out there so you've been a founder at multiple companies yes um uh all of the other ones folded. So can you count how many of the other ones existed? It's not 10, right? It's hard to tell when a company starts to exist. You're moonlighting and then you decide to stop moonlighting. The only one that we put a non-trivial amount of effort to measure on the order of months was an ads company.

And it was a great relief when it folded. I was like, oh, thank goodness I'm collaborating. We're not doing that one anymore. It was a disaster. So did you have any that folded after you hired people beyond founders? No. That is always so very, very hard.

Yeah. I guess I'm now an old school type because I tend to think that your job is to take as much risk out of the company before you raise and to really understand your market and understand why it is that you're raising money. So you're not like,

oh, I have an idea. Somebody give me money. It's like, okay, I have an idea. I've demonstrated the tech. I understand my market. I know that we can take a good run at this. I know that there's a backup plan. And if things fail, we're going to have an acquirer waiting for us.

You get all your ducks in a row before you go raise money. Nobody likes that anymore. Yeah, I don't understand. And also, people don't seem to like starting businesses that make money. Wait, you're already making money? We don't want to fund you. You said people don't like that anymore. You mean VCs don't like it when you have your things together?

Kind of. I've heard through the grapevine that people who have their stuff together versus people who just have an idea tend to do worse lately. In the fundraising market? Yeah. Actually, maybe it's not surprising, right?

If what you want to do is invest cheaply in lots and lots of companies, then it's fantastic to get to invest really cheaply because they haven't proved out their market and you just take a flyer on a bunch of things. And they're too dumb to know how to do this properly, so you're going to take a huge amount of their company and they're not even going to notice.

Exactly. We were fortunate enough, well, I was fortunate enough that my co-founder picked very forward-thinking, sane investors and they were a delight for us all along. And then you got acquired by PayPal. Yes. Was that always your goal? No.

So the goal was to build something interesting, and we did that, and the acquisition was sort of a nice icing on the cake. To build something interesting. Do you think that's what your partner would have said? I mean, as the non-technical co-founder, was this interesting to him or just a way of doing business? Oh, no, I think he'd agree. Because the product was cool. Yeah. And you know, you only have so many keystrokes in your life so you should do something fun and worthwhile with them

I totally agree with that what was being acquired like? it's a fascinating experience I probably am not allowed to talk about it in great detail oh man it takes a lot longer than you might think like it's a it's a many month long process yeah and there's a lot of uncertainty um and it can it can fall through at any time absolutely so uh a friend of ours who talked to us a lot during the process said you know big

companies they're they're sort of like moby dick like they show up and everyone's like oh my god oh my god something's happening and then they submerge and it might be like two months. And they show up again and you're like, what happened? Sounds very familiar. Yeah. And Lord knows what Moby Dick was doing in those intermediate two months when you're scanning the horizon obsessively like Melville. And so it's best not to worry about it too much. You just have to sort of roll with the punches.

Did your VCs, did your investors help you help the partnership, help the acquisition process? Or were they like, yeah, you're done. We'll just take your cash when you're done. Oh, no. Our VCs were wonderful. They were really supportive. They were actually, they were very clear at all times that we were driving the process. We were in charge.

What we wanted was going to go. But when we made it clear what it is that we wanted, they helped facilitate all of those desires. I can't recommend them enough. The good VCs are worth it. Yes, absolutely. Can you tell us who your VCs are? Oh, yeah. So Michael Deering from Harrison Metal was the lead, Manu Kumar from Canine Ventures, Jeff Clavier, and a couple of angels who are quieter.

Very cool. And did you have it in your contract that they would wheel in a bathtub full of cash? And did you choose coins or paper? Because I've always been kind of, you know, coins would be neat because there's all this, but paper would be better. You know, actually, I have, I think if you... What was the most ridiculous thing? I think if you do the math on 20s or 100s,

you'd find that acquisition prices really won't even fill a bathtub. Oh, you're killing the dream. You know, they sell these like, got gold pirate coins online. You can totally go Scrooge McDuck with that. Even better. Then you get to eat them afterwards.

Makes a bit of a mess. Okay, but you've been at PayPal for a couple of years now. Yes, two years. And this is the first time you haven't been a founder. I mean, this is the first time you've... This is the first time I've been in a big. I mean, this is the first time you've... This is the first time I've been in a big company. Yeah. I was an early employee at AdMob.

That was still a startup, though. Yes, it was a fascinating transition going from a startup to a big company. What was the most difficult part for you? Oh, my God, there are all of these people. What do they all do? And then I want to do something. I've been going through that, too. And so somehow, out of all of these people, there are they all do? And then I want to do something. I've been going through that too. And so somehow out of all of these people,

there are some people that I have to talk to in order to get these things done. But who are they? You can't just do it. Exactly. And you go to just do something. And by the way, totally just doing things is absolutely the right answer.

But at some point you're like, okay, well, I need to actually deploy this to a server somewhere to get it integrated. So was the the biggest part was just finding how to get stuff done and to navigate the organization um and you do even if your job isn't management or political you still kind of have to figure out where where the paths are because you don't want to bash your head against something. Absolutely. Although, interestingly,

and this was not something that I had expected, PayPal internally seems to be very, very open to people doing things. Like, you just go off and do something, everyone gets really excited, and you're like, oh, you've done something, let's figure out how to make it big. One of the crazy experiences,

maybe the sort of most absurd thing, was working on this beacon. And then, you know, getting it to be a point where like, okay, it's pretty stable. And then there was this launch. And all of a sudden, there was this party. And there are like 3D, like 10-foot tall beacon replicas. And I was like, who did all this? How did all this happen?

I didn't do this. And I'm used to a startup. Like, if you don't do it, it doesn't happen. It's all those people you don't know who they were. Yeah, absolutely. It was pretty incredible. That is pretty cool. Are you enjoying your time at PayPal now? Yes. I spend a lot of my time working on Go at this point, which is really, really fun. On the compiler side? On the compiler, which is really, really fun. On the compiler side?

On the compiler, the tool chain, the ecosystem. Go is actually designed with building tools in mind, so I spend a lot of my time now building tools that treat code like data and try to do interesting things with it. But that's all open source. Yes. So you got somebody to pay you to write open source code. Yeah, hell of a deal, isn't it? It's pretty cool.

I know that a lot of open source developers are being paid by companies, but it's still strange. Well, you can, I mean, here, we'll, you know, I don't know what the right metaphor is, we'll reveal a little bit. The trade off is, I now spend more time being public. I give talks at OSCON. And I go spend time interacting with the world. And so it's PayPal, historically has had a sort of terrible technical reputation. I mean, there's no way around it. We treated developers badly. And we are acutely aware of that, and we're trying to fix it.

And so having me contribute to open source, be a public figure, and demonstrate to the world that there is actually, and there really is, there is interesting tech going on, is a really good use of PayPal's time and effort. So there's one way to do the calculus. It's nice of them to recognize that there is karma. Oh, yeah. And it really makes a big difference. Yes. difference yes so you are a director at paypal of the beacon system or your title titles what's your title really mean i have no idea in fact i think it was supposed to change when i switched from being a manager to a to an ic and and it never changed i i don't know what my title is i

mean i can tell you what I work on. I split my time between Beacon and Go and sort of internal efforts to improve our underlying tech and push forward innovation. And go to conferences. Go to conferences. Which was, as a founder at startups,

at tiny companies, going to conferences is kind of a luxury. Oh yeah, I never did it. Because you don't have time and you don't have money. And so now you're like, oh yeah, OSCON. I was there. I gave a short talk, but you know, it was fun. Did you have fun at OSCON?

Yes, I did actually. That's the Open Source Conference. It was like a week or two ago. Yeah, last week. Did you see anything neat there that you want to talk about? Well, the thing I'm most excited about that happened at OSCON, but it's not maybe terribly new, is Docker and the container movement. I think that it very much looks like the future.

This brought back such a flashback of me going to OSCON one year. I have no idea what you're talking about, and that's what the entire conference was. I don't know what Docker is. It's pants. Yeah, pants. Pants. Developers wearing pants.

Are they going to start putting LEDs in pants? Because we had a last episode was all about putting LEDs in clothes. So if Docker is, I mean, really, that's cool. because we had a last episode was all about putting leds in clothes so if they're if dockers i mean really that's cool given given what i do to my clothes i think leds in them is a terrible idea for one thing i wash them oh no that was all covered okay good good good clearly i'm behind so so so in all seriousness what are these things what is docker yeah um so stepping back um what are containers oh yeah so we're not talking about container ships i assume no although actually i have written software for those awesome the motivation for um docker is taken from container

ships so once upon a time um if you wanted to ship something maybe a bunch of rugs and you've got a ship that you've hired to ship these rugs for you you pile all the rugs into the cargo and you squeeze them into every corner and this is all well and good until your product that you're selling doesn't fit very well into the whole of a ship and each time it's this one-off loading and unloading so containers famously came along and said you know what let's standardize if it fits in a container you can it. They're easy to move from ship to truck to step to step. So Docker said, you know, look, we actually have this same problem in software. Yes, it runs on my laptop, but I'm running OS 10 and we're running this particular

flavor of Debian out in production. How am I confident that I can take my full environment and my full code and transport it from one place to another? So they said, let's take the container ship as our metaphor, as our inspiration, and say, okay, Docker provides containers. You write your code, you get it running in this container, it's standalone, and you can pick up this container and move it from underlying OS to underlying OS, underlying hardware to underlying hardware, and have confidence that it's going to behave exactly the same. So it's a standardized component you can move around.

Sounds like virtual machines for applications. Yes, sort of. Although they're actually lighter weight than virtual machines. So everyone's really excited about VMs. There's a problem with VM, which is everything is virtualized, and all your I.O. goes through a VM layer. They're heavyweight. So containers tend to use functionality built straight into the kernel. So it's not actually a VM, but you're in a jail. You can't get access to things you shouldn't get access to.

Control access to the file system and the like. It's mediated, but it's not mediated in a way that's as heavyweight as a VM. It's kind of like Mac OS's sandboxing, but generalized. But deeper and generalized. And better than running Java machines. Yes, although you can of course run Java inside a Docker container. And the really cool thing about these is you can start to... Totally going to run a VM with a Docker container running a Java machine, running a simulator

for a processor. I have nothing to say to that. Just as well. But it is really neat. You can do fun things by breaking this tight tie between your underlying system and your application. These actually can start to be portable. So you can actually move a container with a running application to a totally different machine.

And if you can, at your DevOps level, reroute all of your TCP traffic and you can freeze the machine, you can actually have these things at your DevOps level, reroute all of your TCP traffic, and you can freeze the machine. You can actually have these things that have a lifetime independent of the underlying hardware and move them around. And in fact, this is quite common. Well, I mean, that certainly sounds like the dream. Yeah, the dream is almost here.

Cool, it's about damn time. So if you want to Google stuff, Google Docker and Kubernetes. It's an open source project recently out of Google. And there will be a link on the show notes. No doubt. So, camera vision. Yes. Camera vision.

Yes. What did you use for camera? What did you use for processor? How did you get started? I've been playing with this silly thing over here on my Raspberry Pi. Yeah. I have so many questions. Oh, I probably have no answers.

But here's, so we wanted this to work for all users. And this is the card IO where you're getting a visual of the information associated with your credit card. Yes. So this was clearly going to run on an iOS or an Android device using the built-in camera that's there. So we had the camera available. There are camera APIs that were available. So the challenge was, how do we make an appropriately responsive real-world computer vision system that runs on mobile? Notoriously slow processors, underpowered. an appropriately responsive real-world computer vision system that runs on mobile.

Notoriously slow processors, underpowered, and at least in the era that we were working, pretty slow GPUs. They've gotten a lot better since. And edge detection is only part of the problem here. Yes, edge detection is like stage one of the problem. So the interesting thing is, you can't do anything fancy on mobile, because you don't have the compute cycles for it. So your challenge is really how do you take like decades old algorithms, and figure out how to bundle them together, optimize them. We spent a lot of time

writing assembly for the neon co processor to get the performance we needed out of it. How do you take these things and tie them together in a way that will help the user help you? So it turns out we're really good at vision. Computers are really bad at it. We can get the user to help us a lot if we provide them with a really responsive, really fun UI that can sort of train them without them knowing they're being trained, can train them to do the hard work for us. And it sounds silly, but this is actually what we do. The way that we made this work was a lot of low level optimization, a little bit of magic, but surprisingly little magic, and a lot of attention to the UX. Oh yeah, the magnetometer people will tell you

that the UX is absolutely the most important part because they started out trying to convince us to do a circle, and then finally they have that figure eight, and somehow we all kind of know how to do that one. Totally. And so yeah, training the user to do something. And we had interesting user challenges. One of the first companies to pick up Card.io

was a ad hoc hotel booking app. And so you can imagine that a lot of our users were drunk and in a bar. So it's dimly lit. Nothing can go wrong with this plan. Exactly. People joke about pretend your user is drunk. I was like, my user is drunk.

Now what do I do? And the great thing about it is we learned very quickly, you couldn't tell them anything. If you tell them anything, you know, like with a text or an icon or something, they totally ignore it. So it has to all be intuitive. And perhaps most importantly of all, it has to be extremely responsive. If you want folks to learn quickly and interact with the system, your reaction time has to be really, really small. So we spent all this time optimizing specifically so that we could process frames

before the camera could deliver another frame to us. So we were maxed out at the frame rate. And this insight was actually provided to me by a guy named Octavio good who made word lens i don't know if you ever saw it it's another mobile computer vision it does yeah ad hoc translation really cool yeah that is really cool yeah and so what he told me is i mean he has some serious tech in there but he said the thing i realized is if you refresh really quickly users will figure out what to do to get the output they want. And WordLens actually, basically there's this like

maybe second and a half training as a user. It doesn't work. And then we figure out how to make it work. And we constantly adjust to make it work. And that happens specifically because it's very responsive. So we took the same insight and applied it to Cardio.

Neat. That is really neat. So people want to get into computer vision. Do you have resources, suggestions? Yes. So there's a book called Learning Open CV. And it is totally readable. And if you read it from beginning to end,

you will know about half of all you'll ever use about computer vision. It has all the basics in there. And you can get up and running very quickly with OpenCV, both on your laptop, which is a great place to experiment, and on your phone. When the time comes that you want something a little heavier duty on the processing side, there's an open source library called Eigen that does lots of matrix multiplication. And it is this crazy C++ templating library, and it takes forever to compile. But man, it makes amazing, amazing code. So we needed that for a lot of our very efficient work that we did on the phone. And the other recommendation is a Python library called Theano. And I thought you were going to say SimpleCV. No, no, no. So it turns out a lot of computer

vision is actually machine learning. You've done some work, and then at some point, you just, you know, you need some help from a model. And the interesting work going on in the sensory-oriented parts of machine learning is mostly around deep learning at this point. And Theano is a pretty awesome library that makes it easy to experiment with deep learning. They also have very good tutorials to let you get to try out different deep learning models. And it comes out of the University of Montreal, where a lot of the interesting deep learning work is going on. So it's a quick way to dip your toes into deep learning. And that has a little bit of looks like integration with NumPy, which is Yes, exactly. So then I think everybody should get to play with.

Absolutely. And really importantly, it also has good integration with your GPU. So when you go to train one of these models, and it takes like five days, and then you go buy a fancy gamer GPU for a few hundred dollars, and then your model trains in five minutes, you start to think, wow, it's really nice that I can write my code and it takes care of having good GPU kernels available for me. Yeah, using your GPU to do some of that matrix math is the way to go. It's absolutely the way to go. And Theano makes that all transparent. And they also do things that, as someone who's new to machine learning, you might not think about.

You might be like, okay, well, you know, I'm doing neural nets. I have to do some back prop. But there are optimizations you can do to make it both faster to do the computation, but also to provide numerical stability. You can transform mathematical equations in a way that makes your output,

avoids problems with floats around very large and very small numbers. Theano does all of that for you. And so it's like basically having a graduate student in machine learning available, fixing your code for you, and you just write at a high level. Ooh. Yeah, totally fun. All right. Well, I have just a couple more questions for you then,

because I want to go play with this. You've done a lot of deeply embedded, you've written assembly, you've gone to deeply science. But PayPal isn't embedded. I mean, you're working on Linux now, and Linux and Go sound very cool, but do you miss the hardware? Do you miss the twiddly bits? No, no. In the end, all of this is a means to build something cool. I mean, people who get into embedded, yes, they love the, you know, the data sheets and stuff, but ultimately it's because you want to build something cool and there's no way to build it other than by going through Embedded, other than by going down to this next level.

So for me, it's all about like, what can we build that's really amazing and interesting and unique and pushes the envelope? And you go exactly as deep as you need to, and no deeper. All right. Well, any last thoughts you'd like to leave us with? No, it was a lot of fun. Thank you. Well, thank you for being here. My guest has been Josh Bleeker-Snyder, Director of Software Engineering at PayPal, here to tell us about Beacon, although I'm happy we arranged all over the place. Thank you also to Christopher White for co-hosting and producing.

I am Groot. His loquaciousness is noted. If you have questions, comments, or suggestions, email us, show at embedded.fm or hit the contact link on embedded.fm.

We do like to hear from you. And if you like the show, tell a friend. Thank you. Thank you for listening. Our final thought this week. Let's see. Oh, this one's for Christopher because I know it's really important.

It's from the Dalai Lama. Sleep is the best meditation.