Embedded - 155: Foot-Seeking Bullet

Episode Date: June 8, 2016

Jonathan Bradshaw spoke with us about working with hardware engineers, schematic reviews, and FPGAs. At the end of the podcast, Jonathan made a pitch for folks to submit proposals for the IEEE Southern Power Electronics Conference in Auckland in December. The FPGA boards Elecia mentioned were the XLR8 board and the Papillio platform (more on the latter in show #66). By the way, The Amp Hour is our “enemy podcast” but we actually like their show quite a lot. It is a joke. But do feel free to tweet their shameless advertising tweet with the link replaced with one to our show.  And weta are neat! (Image, wiki)

Welcome to Embedded, where we plan to talk about sea otters today. There's a good reason for that. Our guest expects to talk about electronics. Who do you think is going to win? I am, of course, Elysia White, and my co-host is Christopher White, and our guest this week is Jonathan Bradshaw. Hi, Jonathan. Good morning where you are. Hi. Yeah, it is morning. Thank you.

All right. That's true. Jonathan is in New Zealand. Is that right? That's right. Well, why don't you tell us a bit about yourself? Okay. I'm an electrical engineer. So once upon a time, I went to university and studied for electrical engineering. And then I worked for a couple of years as a power systems consultant. I found in that one it was a bit top-down and high-level, so I went back to university and did a PhD. And after that, I worked in ABB Switzerland for a few years as a corporate research scientist before moving back home to Auckland.

And now I work as a senior hardware engineer, developing what might loosely be called consumer products and learning a whole lot about actually manufacturing things. Cool. Well, we have many questions about those things and a few others, but let us go to lightning round where we ask you questions and want short answers. And unless it's about sea otters, we want your answers to be relatively limited.

Chris, do you want to go first? Sure. Favorite electrical component? IGBT. You're going to have to help the software engineer understand what that is. Insulated gate bipolar transistor. It is a high power switch switch really okay big mosfet

all right least favorite electrical component leakage inductance it comes for free and it's really hard to get rid of it wait a minute that's not a component the virtual component all right. Do you listen to podcasts? Yes. What is your favorite, excepting, of course, this one? Oh, that is a bit tricky. That is very tricky. I think I'm not going to answer that.

It's because it's the enemy podcast, isn't it? No, no, no, no, no, no. I've caught Dave Jones telling sheep jokes, and I feel that earned some marks off. And I would like to point out for anyone who's heard about the infamous sheep jokes that no matter where you are, you can just swap Australia and New Zealand and the joke still works. Favorite fictional robot? I think it has to be Marvin the Paranoid Android. Good choice. Would you rather explain API design or ground loops?

Ground loops. Is Australia up or down? Australia is left. I see. Beach or down? Australia is lift. I see. Beach or mountains? Come to New Zealand, you can have both. All right. Yeah, New Zealand and California have a lot in common.

What is most important to your job? A whiteboard, a soldering iron, or a keyboard? A mouse. This comes from last week's episode. Favorite type of flux? I'm going to be different. I'm going to be different. And I'm going to say a medium weight non-rosen flux.

Because rosen flux you have to clean or eventually it'll soak up some moisture and become acidic and attack the rest of the board. Okay, can you give examples? Are there chemical names or something? We're still learning about flux. I don't know if you caught the last episode where we discussed flux in great detail. I can't think of any specific brands. I just fished the syringe of paste out of the fridge at work. Fair enough. Everybody who wants that should go to your paste out of the fridge at work. Fair enough.

Everybody who wants that should go to your work and rob your fridge. Right. And remember, if it's in the fridge, it's good for eating and soldering. Oh, no, we have a separate glue, solder, et cetera, fridge. That seems safe. Yeah. Pinnipeds or cetaceans yeah you're either gonna have to let me use google or or explain those ones i'm with you man pinnipeds are seals and sea lions sort of marine mammals and cetaceans are whales and dolphins

sort of marine mammals oh i'm gonna have to go with cetaceans are whales and dolphins sort of marine mammals. I'm going to have to go with cetaceans. That's it. We're out. I'm getting a note. Should we bring back the dinosaurs? I think that the environmental niches to support the dinosaurs are gone

and what we should instead do is genetically engineer giant cavalry weta. So if you just go into Google Image Search and look for weta, you'll see what I mean. Oh, oh no, I know what those are. Don't, don't, don't search. You won't like it. Okay. No, I need to know.

I mean, at least a high-level explanation here. Very large insect. Oh, oh, yeah. Right, right, right? Yeah, yeah. It's like a punk cricket. Yes. With long feelers and hooks and spikes on the legs. And the cave ones, the bodies get to about 100 millimeters long and then 200 millimeters of feelers out the front.

Quite remarkable. Yeah, I won't search for that. I'm going to go back to my pinnipeds. Okay, this one is more detailed. What is a day in the life like for you? A work day? A typical day, I guess you'd put it together from some components. So, some typical things I'll do are, boring as it sounds, documentation chase-up. So, chasing documents and

writing documents for um certification procedures which is a very important but not very exciting part of my job um they'll also be liaising with the firmware team about uh what what we've got coming up what what is available what we can do in terms of low power modes and stuff and board bring up then there will be component research and finally there will be design which runs all the way for me from component selection to schematic capture to board layout, getting the boards fabricated and assembled, and then finally whatever hardware hacking is necessary to actually bring the things up. So kind of choose three.

Very much a hardware engineer then. Yeah. But you did mention liaison with the firmware team What should firmware engineers know about working with hardware engineers? When we hit compile, it takes six weeks and costs several thousand dollars And that makes sense I mean, because you're dealing with real actual protons and electrons. Yeah, the protons are really heavy.

And there's all these neutrons. And they just hang around and do nothing. Nothing at all. You wouldn't like the universe without neutrons. They're very much like those sea otters we saw today. Well, the early universe probably didn't have a lot of neutrons in it. And if you wait around, something interesting might happen. Okay, so compiling takes a long time for you.

That's right. What implications does that have? Well, I guess there's sort of two aspects. One is in production, when you're ordering a thousand pieces, you become truly paranoid that you've done something wrong. Because one mistake is not so bad. One mistake repeated a thousand times is very annoying. So for production orders we try to do minimal deltas, we try to do a prototype first and we hope we get it right. When it comes to the prototype development, I try to review all the schematics, make sure everything's connected, review the boards, hand the board step files off to the mechanical team who check they're going to fit the plastics, and review, review, review, cross fingers.

The review is the hard part. Software engineers often do review as a post-test stage. You know, we write our code, we compile it, we test it ourselves. We may give it to somebody else to unit test. But it's only after all of that that we start to review it, which may not be the best way. I mean, design reviews are great too. But our final, before we ship something,

we may not review the code until after we ship it but for you it's got to go the other way yeah yeah i mean here's where it gets a bit tricky because every circuit board is is custom and what that means depending on what technology of components you're sticking to the board what what the actual innards of the board are like does it have innards or is it only two layers there are some things we can we can hack and we can cut tracks and solder bits on and solder bits of wire and all sorts of things and there's some things where you can be like yeah we can we can knock that up that'll be fine and there's other things where you can be like, yeah, we can, we can knock that up.

That'll be fine. And there's other things we're just like, oh no, no, no, no, no, no. That's not going to happen. And unless you've designed the board, you don't really know what box things are going to fall into. When I do what I call hardware design, it usually involves buying a bunch of breadboards from Adafruit or Sparkfun or wherever I can find them. And sticking them together with jumpers, maybe eventually soldering actual wires together.

But I don't do the sort of hardware you do. And I don't know how to make that step. I mean, if I just copy the Adafruit schematic and do it myself, then I end up with a bunch of extra components that I don't need, but I'm not confident enough to take out. How do I get over that barrier? Well, that's quite a big question. That's good, because Chris and I are really exhausted from kayaking, so have at.

So, I just talk for a while and you have a nap. So there are some components like decoupling capacitors where it is unwise to throw any away. Realistically, you've probably got too many, really, if you're interested in squeezing the last sense out of the design. But the problem is if you don't have enough, the symptoms are weird and frustrating. Like, oh, it doesn't work in the morning, but then after lunch it does because it's warmed up a little bit. Or, oh, it works on this supply, but not that one. Or it works when it's plugged into the computer, but not when it isn't.

And it's just so frustrating. So don't skimp on the decoupling caps but there's other things like for example esd protection which you you really only need it on the outsides of the of your circuit so if you're sticking multiple boards together and they've all got esd protection and they're all going to be in the same box eventually. That's a waste of effort. Where it gets a bit tricky is all those interesting little resistors and capacitors in the signal paths. What are they doing? Why are they there? Can I get rid of them? And what I'd say to you there is you can buy zero ohm resistors. So just leave all the resistors in.

And then if you want to see if you can get away with it or not, stick in a zero-ohm. And that's a cheap way to configure a board. Do they have zero microfarad capacitors? Yeah, it's called do not populate. Right, because, okay. Okay, yeah. I'm cool with that. It's called air. It's called Do Not Populate. Right, because, okay. Okay, yeah. I'm cool with that.

It's called Air. It's called Air. Okay. Anything else that I should know when I'm, you know, putting things from schematics I get off the web into actual custom boards? Generally, I'd say don't freak out about getting it perfect the first time because you almost certainly won't. I've made many circuit boards over the years and only one of them was right first time.

Most boards you're going to need to cut a couple of tracks and solder some stuff on, that sort of thing. So if you're doing a two-layer board, you can get at it. You can chop things and change things, and that's all good. If you're being bold and making a four-anth or four-layer board, what I recommend is putting... You've got your two outer layers, which you can get at and chop and change. And then you've got your two inner layers.

One of those should be ground and the other one is probably going to be power. And that way all your signal layers that you might have to fiddle with are on the outside where you can get at them. That makes sense. Yeah. There are a few places where you can get trapped or caught out, like if you're trying to stick 5-volt and 3.3-volt components together. Yes, and that is something I end up doing a lot when I get off-the-shelf breakout boards

because they're all at different voltages, and I have to fuss with that. Yeah, and I'm sorry to say something inflammatory, but 5 volt is dying. So for all those people who are like, why can't I get this? Why can't I get this, you know, 600 megahertz processor in a dip with 5 volts? And it's like, no, you can't. You never will. Give it up um on usb i i have a a 30 second

digression on usb power for those interested in a in an informative rant go ahead okay so usb is five volts allegedly now if you look at the spec what's coming out the side of your computer should be between 4.75 and 5.25 volts, i.e. plus minus 5%. Once you get down the cable, your USB device is meant to work all the way down to 4.4 volts. So you're not really 5 volts anymore. Also, the cable might be long and skinny. So you've got 4.4 volts at your device now. You're allowed to draw up to 500 milliamps, but you've got to leave like 10% margin. So you probably design your circuit or your battery charger to be 450 milliamps so you don't violate the spec. And then your battery charger chip has got a pretty

inaccurate current limit so sometimes it's going to be 10% low and you're only going to have 400 milliamps of current available at 4.4 volts and then if you multiply that by your buck converter to get down to a nice clean 3.3 volts, efficiency 85%, suddenly instead of 2.5 watts of power available, you've got like 1.5 watts of power available. So USB power you need to think about. Yeah, of course the losses are going to add up, but we get so used to powering everything with USB chargers, and that's good advice. with ordering a bunch and having the errors propagated throughout the hardware.

Is that an experience professionalism sort of dichotomy? Or is that you should only order three boards initially, or you should maybe order a thousand but only populate three? How do you do that? Right. Well, that is a big question. And effectively, hopefully, you get to pick your level of risk. So, I would recommend you start off and you order 10 boards and you populate three of them. That would be my suggestion. hypothetically, someone in sales has promised someone else something on very short notice and you have to go straight to it, then you're going to have to go straight to production quantities,

then you better review, review, review, and then suck up your guts. And as a contingency measure, you may wish to interview for jobs in Western Australia in case you have to flee the country. Yeah, it depends on schedule. My preference is often, let's be honest, a lot of products these days are a microcontroller with bits stuck on. My preference would be to bash something together with dev boards like you do, because you can get them. And professionally, your time has a lot of value. And as a hobbyist, you've got to ask yourself, does my time

have value? Well, and do you want to do the mundane parts as a hobbyist, or do you just want to skip to the fun stuff software exactly wait a minute forgiven definitions of fun there we go there's there's putting the boot in for you um the thing is most hardware these days wouldn't do it wouldn't do a thing without the software so um it's fair to say my job is preparing certified hardware platforms to run software on um yeah so ultimately the do you do you review the heck out of it and try to get it right first time and order a lot versus do you prototype your way up and expect failure depends on your planning and your business and your skills and your history and whether or not this is a totally new product or you're just reusing pre-verified blocks out of something else.

And your tolerance for risk. I mean, that's what that all comes down to. Yeah, yeah. There is a time when throwing money at the problem in hopes that it all comes back good is worth it. I mean, if you've got a short market window, you'd better get to it. But if you've only got a little bit of money before this company has to close, don't order a million of anything. No, no. And then you run into lead times uh so so sometimes you

have to sort of make a hybrid approach like if you said we need this fancy feature and this processor is the best one to do it with um you order a dev board you check it actually works and then you might have to stump up and order a reel of two and a half thousand straight off the bat because they've got a 12-week lead time. And that is life. And every now and again, you're going to throw away a reel of components. Sorry. As software engineers who don't have to deal with the physical part as often, that's a tough thing to really understand, the idea that these things cost money and it's sometimes okay to throw money away.

But that's what investment is. You do your best on this and then you go on. Speaking of going on, actually going back, reviews. You do a lot of reviews to make sure that what you're putting out actually is what you really want. But as a new engineer coming from software, those schematic reviews were sort of intimidating. Because I was just learning to read schematics. I didn't know what the electrical engineer really wanted from me. And so I spent, I mean, I spent hours and hours trying to figure out all of the little resistor networks,

only to find out that what he really wanted was more along the lines of, did he choose the processor with the right amount of flash? And is the pinout not as objectionable as I... Do you have access to all the debug stuff you want? And are my spy lines on spy-appropriate pins? It took me a while to understand what a hardware engineer wanted from the schematic review process. And I'm not sure, even today today that I am doing everything they want, because if they want those resistor networks checked, I don't do that anymore. What would you want from your firmware engineers

when you're doing a schematic review? This is a useless answer, but it depends. Sorry. The important thing is to realize that you're not sure and then you're going to have to go and talk to your talk to your hardware engineer and say what did you actually want me to check what is really important here and if they say everything you can say um i'm not screwing around double checking all your filter calculations because that's not my wheelhouse um but the things i'm interested in in designers checking for is

stuff like and software engineers checking for is stuff like um have i put the right frequency crystal on is it the right chip does it have the right chip? Does it have the right, you know, does it have the right version with the right flash and RAM? And indeed, have I connected ports to it appropriately? So, oh, you use a particular UART for bootloading? Well, you better check that one's connected. And, oh, you need these to be the same across the whole product line. Well, we'd better do that one's connected and oh you you need these to be the same across the whole product line well we'd we'd better do that you know get some pins the same so we can use common factory

software because if you need a custom software version for everything the production team is not going to be happy with you um and then it depends on the product, but it can get even more interesting than that. Like, I routed some interrupt requests into this micro. I might have screwed up. Those ports I've put them into might not support interrupts. Well, they might support interrupts, but only if the processor is already awake. And you meant for that button to wake up the system. Oh, sleep modes.

Yes, sleep modes oh yes sleep modes and and sleep modes is a whole different topic of design review it's a it's it's really interesting it's interesting engineering but it can get nightmarish um in terms of the the the zandhine wake-up circuits and oh well this process has only got one wake-up pin so you'll need to multiplex stuff and oh by the way uh when it's in the steepest sleep mode um it forgets what to do with all the IO pins and um they all go hi-zed sorry um so that kind of thing it's a real hardware software issue because the initial software bring up or the then lack of lash up on the dead on the dev board won't show that so really what i the most important things i need from a software engineer for them to tell

me things like this board doesn't do interrupts these um these particular iopens need to be on the same ioport or can't be on the same ioport uh these you know this spi bus oh this doesn't work right on that port of that chip so we need to use the other one. And yeah, the low power stuff, if they can think about it too. But low power modes are one of those things where they really fall down the crack between hardware and software. Like we had a product in the field working properly, produced quite a few of them. Then a customer said, oh, the sleep current is terrible.

And we're like, what are you customer said although the sleep current is terrible and we're like what are you talking about the sleep current's great and then it turns out that the hardware the soft sorry the firmware team have been a bit rushed and not implemented the final power save mode and then they were saying oh the hardware doesn't support it and we were saying the hardware does support it and it was just in that in that gap well and it it can be this sort of thing where you have you know you've got your processor running maybe you've got you've got a mode where your processor runs at a slower speed so that it takes less current then you put your processor to sleep and there's lots of ways for it to wake up either from a timer or from multiple gps or even from DMA, from communication methods.

But then there's this other sleep feature that a lot of people, a lot of processor vendors are implementing. It's a deeper sleep, a hibernate is what I usually end up calling it. And that's much lower power. And sometimes you get to wake up with a clock, but sometimes you don't. And you can only use a GPIO to wake up with a clock but sometimes you don't and you can only use a gpio to wake up and only one gpio maybe only the rtc clock not any of your other clocks and so it's a very deep sleep and really nice because you aren't sucking a lot of battery but yeah you have to talk to the hardware engineers about, well, in this mode, everything is really stupid

and I don't get to do anything until you give me an interrupt here and it has to be from one of these six buttons. Yes, hardware, you have to do that. I can't do that in software. So that level of organization does fall through the cracks a lot. Yeah, yeah. And you can usually kind of paper over it a bit, but it's never going to be as good.

I believe you have been working with STM32s? Maxim chips also have the same. I can neither confirm nor deny, but Maxim chip and Atmel chips have the same levels. That sounds like a good firm.

And Maxim is also infamous for having long and inexplicable lead times. Long, inexplicable lead times with like, no, like they never deliver. I was so shocked. I had a client recently say, well, we want to use this Maxim processor with blah, blah, blah, Cortex, blah, blah, blah. It's super cool. And I'm like, you can't use Maxim. I like put my fingers up in the shape of a cross and held off the Maxim distributor.

No, you can't do that. They swear they're better. And we haven't had problems so far, but we also haven't started mass build. So, yeah, Maxim. I would have changed the name. Once upon a time, I was at university. And some of my colleagues ordered some free samples off a semiconductor company, which shall remain nameless. And a few weeks later later a box turned up from

from company x and they opened up and they're like oh it's our samples sweet and they're like these aren't the samples we ordered these are random and they looked through and they looked the packing list like well it matches the packing list but this isn't this isn't what we wanted the samples we bought and then they looked and then and then they had this horrifying moment when they said oh no i know what's happened someone said these are the samples we ordered last year yes yes yes yes lead times um okay going going on um Are there any tricks that you use on your boards that we as software engineers or firmware engineers can then ask our electrical engineer about?

Oh, there's some good tricks. Zero ohm resistors are really cheap and can be quite small. And that can let you do cool stuff like, oh, if we just pop off these zero-ohm resistors you can interrupt the serial port and stuff in another board that already works or hey we've got some weird peripheral um put some test points down so i can get a scope onto it. Now, you may have to find out what kind of snack food they like, but a skilled hardware engineer can actually solder wires onto the side of a 0.5mm QFN. I usually get charged with either chocolate or beer on that and it it's not cheap uh it it is a total pain in the

backside um but if you can handle it at schematic time and say i probably will want a test point here they can pull it out to the edge and maybe make their lives easier. Oh, yes, so much easier. Even just putting a little one-millimeter round dot on the board, it doesn't take up too much space. But then you can just go and solder a wire on. It's very easy. Well, in the complicated things that I have to deal with as a software engineer, the communication buses or even looking at particular signals going to and fro, those are often the same signals the hardware engineer needs to do manufacturing testing.

Oh yeah, there's often quite a lot of overlap. So I guess to call back to a previous topic, uh the important thing is it's hard it's hard because it's not in anyone's job well it might be in someone's job description but who cares it's hard because it's not in anyone's regular job and you're not going to get promoted for it and if people notice they're probably going to think you're actually misspending your time but the important thing to do is to stump up and actually go over to the double e's desk and talk about what you're going to need to do and i'm going to say to any double e's that are listening this applies in reverse as well

you need to stump up and you need to take your block diagram and your schematics and you need to go and talk to the firmware team about what they're going to do because remember however much it might feel us first then you are all on the same team yeah and you can get a lot of points by um phrasing it uh i wouldn't say correctly differently than you might for yourself like i have learned not to ask for test points for software debugging, but instead to suggest test points for manufacturing debugging, which is a lot more important to an electrical engineer, but they're the same test points, and so that's cool. Ah, well, that's sneaky.

I myself am pretty happy to put down software debugging test points because that saves me from soldering wires onto the side of a QFN. But the hardware engineers can go to the software engineers and ask for software to debug the boards, which just happens to be very, very similar to software that would be needed to manufacture the boards. And it's a phrasing, like, what's important to you and how can I help you do your job and also get you to do mine at the same time?

Sort of sneakiness. Yes, sneakiness. Oh, yeah. I mean, sadly, you actually need an incredible variety of software to fully test out a board. It's kind of crazy. For the certification process on a product I'm finally prodding across the line right now, there's like 10 different test

software builds for the damn thing. It's crazy. It's ridiculous. Let's count how many radios you must have. No, let's not. We're not supposed to talk about work. Well, actually, that's without the radios. Wow. Yeah. That's got to be complicated. And then people are like, wait, wait, wait, wait, what's that shit for? And I'm like, oh, I've had this discussion several times. It always fails.

Just going back briefly to the talk about schematic review and what software engineers should provide as information to the electrical engineers, it seems to me that a naive software engineer just coming into firmware might look at a data sheet for a processor and say, things that we talked about the limitations of okay this port can be used in this way this is what happens to this port when you're in a different sleep state this is what you know what can be shared sometimes um and i think in order to have that conversation with an electrical engineer the firmware engineer has to have a deep understanding of the part. Well, then I didn't understand that I was supposed to partially represent the data sheet. Because the data sheet is so electrical looking that it is hard to get through my head sometimes that, yes, I really, really have to understand this deeply. I can't just surf to the area that gets my job done. I can't just surf to the area that gets my job done. I can't just go look at the spy chapter of the reference manual and tippy type my little

driver in and then make it work. I have to understand when I go to schematic review, spy has to be on one of these ports on one of these pins, because if we get that wrong and you end up with a spy clock coming out of some random pin then i'm totally hosed as a software engineer and our product will not be shipping on time you can't fix there's always bit banging never there's never no no i'm sure you can bit bang an i squared c port with clock hold and I2C high speed mode That'll be fine, right? Sure, as long as that's all you want me to do Flashing back to bad memories of bitbanging spy and I2C off of an NI digital out port

Oh, oh, I feel your pain Why we did that, I don't know Yeah, the reality is that if you know your design is going to be pushing things a bit, for example, you know you want to do lots of sleep tricks, or you know you're going to be using up lots of I.O. resources, and this can be really painful, you're using an external memory bus, I'm sorry, but you're going to have to read quite a lot of the user guide for all the candidate chips. You're only going to use one.

So you're looking at five different candidate chips. Well, 80% of your reading is ultimately going to be not very useful. But engineering is a profession, and that's just something you have to do sometimes. It's useful for the next job. Yeah, yeah, yeah. Um, and the other thing I'd say is please look at the errata. Your electrical engineer should look at the errata too.

I still miss the errata. I just don't think that way. Even though I know about bugs, I just think chips should be bug free, even though it's shot me in the foot a number of times. So, I mean, there are some classic ones I've run across. Like, one is from the Texas Instruments C-2000 series, the series which later became known as the Piccolo. You know the ones. That's so confusing to me. I still, yeah, C-2000 is great. Piccolo is You know the ones. That's so confusing to me. I still, yeah, C2000 is great.

Piccolo is a flute. Yes. Yes, I know. Marketing. Marketing. It's like, yeah, so if your ADC sample window is less than 70 nanoseconds and you use the auto scan feature, then the first sample on the auto scan will be corrupted. So you can either go on the software and put a dummy scan in, or you can just change the acquisition window and you're fine. And you're only going to find that if you read the errata.

Oh, and a lot of the chips have errata around sleep states. Oh, yes. And so if you are sleeping, you really, really need to get that errata and get the latest. Oh, yeah. MSP 430 FR, basically anything. So that's the ferroelectric RAM, non-volatile RAM thing. They look really awesome. I wanted to use one for a product.

I was like, oh, wake up's good. Yeah, look, it can run off a 32K crystal and keep the UART active. That's awesome. I mean, it's low power. It's easy. It won't be lots and lots of handshakes stuffing around for the firmware team. Don't say I don't think of you. And then I look in the errata and it's like, oh, by the way, in these two low power modes, it may not wake up.

That's all. And you're just like what the hell how did how how could you release a chip that was that bad because it had to deadline they had market windows too yeah and then it's like oh we're up to the rev f silicon and i'm like okay and then i look in the router and like oh it's still there yeah so read the errata that one that one is code pmm24 um and one advice that i have dragged from from place to place is if you have spare gpios allocate three of them to indicate hardware board revision then when they mess with your GPOs and they put them in different places, it's easy to fix.

Oh, that's a good thought. I do a little bit of pinstrap programming, but my firmware team, see, every engineer is going to be different and every team is going to be different, that whole culture thing. So my firmware team, I was like was like well i can knock up a i can lash something up by by kludging this dev board and this half this existing product and half that existing product here they're all glued to a bit of wood with some hot glue oh yeah hot glue um and get everything connected but all the pins are going to be in the wrong place. And they were like, oh, we'd rather just wait for the first prototype

because it's so annoying to remap all the pins. And I'm like, okay. And then I give them the first prototype and they're like, oh, some of these connections aren't the best. Like, could we have an extra signal here? And, oh, we're a bit concerned about this and i'm standing there and i'm thinking but not saying because there's no point it's just going to upset people um then you should have let me give you a lash up and you would have found out

before i'd ordered the boards see six weeks and several thousand dollars to hit compile. Yep. It comes back to that every time. Yep. We'll get there in the end. So FPGAs. Ah, they're good. What have you done with FPGAs? So I did my PhD thesis sort of on FPGAs.

Now, it should be said I wasn't doing anything outrageously fancy with them. More obscure. So, shall I give a canned summary of what an FPGA is for those who have heard the term, but not too much more? Sure. As long as you can get past field programmable gate arrays, then that would be great okay so uh the basic idea of an fpga is that sometimes you would like an asic an application specific ic but making one of those is outrageously expensive really getting really expensive yeah like like start with a million dollars expensive um so on fpga the concept is it's a configurable ic so you feed in a special configuration bit stream a la code and it opens and closes switches inside to wire up the innards to make a custom logic circuit, which you can program to do what you want.

So the attraction there is you can get custom logic without spending a million dollars. Yeah. Depending on who you buy from. Oh, yeah, yeah. I mean, if you buy the high-end, latest-gen, smallest nanometer, highest frequency, all that stuff, you can spend $20,000 a chip. The attraction isn't really cost, although I think it's getting better. The attraction is definitely speed for certain applications, right? Well, the attraction is not low power. I'd let him answer that i don't know well the the fact that it's even hard to ask the question is kind of informative about the real problem which is um they're not the best at anything uh they'll be the best choice sort of in aggregate. Like, if you can do it well with just a boring microcontroller, do that.

It's going to be quicker to engineer. It's going to be cheaper. But if you need something that you just can't do with a micro or can't do with a standard chip, for example, you need a special custom serial protocol. Then you're saying, well, we can't do it with software. We need custom hardware and we're not going to make a hundred thousand units a year. So we'll use an FPGA. They are not the cheapest, except when they are. They are not the lowest power, except when you're racing them in heavily numerical cases against a general purpose processor.

They are not the smallest on the board, except when you're racing them against a giant kludge of logic gates wide up. And they're not the quickest to engineer, except when you're pushing microcontroller technology to the very end and hand-optimizing everything in Assembler. So this is why every time I come across an FPGA, usually I come across the FPGA dev board, not come across a problem. And they say, well, use it for, I don't know, adding floating-point logic to Arduinos, because I actually saw that recently. And my response to that is, why don't I just use a real processor,

you know, one that has floating point logic if that's what I need. You save the Arduino for when I need 8-bit things. And then I have people say, I mean, you said actually, a specialized serial protocol. And my first response was, why don't I just put a 30-cent micro out there and have it translate to something useful? I keep ending up with that with FPGAs.

It's why don't I just do this with a microprocessor? It seems simpler. I think those applications these days are easier to do with a microprocessor. I think, in my experience, the things that FPGAs are good at now are, well, we could do this with a dedicated micro, but that would be all it gets to do. You know, you have a Cortex-M3. We need a Cortex-M3 level thing to do this image processing or this sensor fusion kind of thing, or to deal with signals that are at a very high rate that an M3 might not be able to keep up with. It's those sorts of things where you have a very specific, very high

speed thing that you don't want to burn a general purpose thing on. But if it's so high speed, then it's got to have throughput, which means it's got to go somewhere. On one end it might, but not on the end to your micro. Yeah, that's true. Okay, you're starting from a really good

place there, which is skepticism skepticism i has it yeah because an fpga is often not the best answer but there are some cases where they are just awesome and i will i will try to present a couple which might show you where you might have the edge and actually step up to one of these things. What say you need to do an incredible amount of fixed point math or just a lot of fixed point math in a very short time? Let's say you've got some weird ass process going on that you need to service a control loop every five microseconds. If you're running a big ARM A7 with Linux on it, you've only just gotten into the interrupt routine when your time limit has expired. If you're running a little bare-bones Cortex unit, then you might have a couple of hundred ops

before your time limit expires on an fpga you can go crazy parallel so you could have a clock rate of 100 megahertz is generally quite achievable so five microseconds times 100 megahertz you've got 5,000 cycles, right? You could deploy 100 multipliers on this problem, say. Multipliers are often a good metric of numerical problems. So you could deploy, what did I say, 100 multipliers. So you could deploy 500,000 multipliers on that sucker in 5 microseconds. So that's one case where an FPGA is going to be a real candidate.

Another one is if you need to do something interesting and custom in serial, in access protocols, like if you need to stream data out of an ADC and onto Ethernet, and for some reason if you need to stream data out of an ADC and onto Ethernet, and for some reason someone really wants to stream 60 megabits data stream through 100 megabit Ethernet, you're going to have trouble doing that with a micro, because it's quite close to the limit. And by the time you're waiting for interrupt routines and stuff, you could have burnt too much time. but that one's a bit of an iffy case by case thing. Um, the other thing where FPGAs are fantastic is where you need like deterministic,

precise timing. Like if you're doing radar or radio or test equipment and you need it synchronized to the clock then doing that in an fpga is a matter of attention to detail whereas if you've ever tried to write code for microprocessors that is cycle accurate through all the branches and if statements and function calls you'll know that you spend days with the microprocessor reference manual counting out instruction durations. And then you have a cache miss and you're screwed. Well, and so this comes back to me for things like Chris mentioned sensor fusion and synchronous sampling is sometimes really, really important

because if there's even 100 nanoseconds between when you sample the X gyro and the Y gyro, if you're moving, you're not sampling the same thing anymore. And in vibration environments, if you sample the gyro and the accelerometer at different times, they aren't telling you the same thing at all. And so you need synchronous sampling. And that is hard to do in a microprocessor. I mean, sometimes you can get ADCs and you can toggle them all at the same time. Not that it's impossible, but also at Schatzbotter, we needed synchronous sampling across the different microphones so that we knew exactly when things arrived

because when things arrived was backtracked to where people were. And so the FPGAs are very helpful for that. Chris, you said they were in the VR system. Mm-hmm. Probably because there's a lot of sensors that need to happen, that need to get that synchronous sampling at the same time so that you are positioned correctly in the room. Yeah, and I think the micros on them are Cortex-M zeros.

So they have a baby micro and the deaf PGA doing heavy lifting. Plus there's the P. I mean, the programmable aspect is really important. And I think, you know, especially if you're doing some sort of algorithm that's high speed or synchronous. Image processing. Image processing, things that need to be changed

or could be upgraded. I remember we used some high speed DAC boards at a previous job and they had an FPGA doing a lot of the heavy lifting, but it also, you could put new stuff in it.

Like, oh, I want you to do the FFT of this data before it even leaves your silicon and goes to my software. I want it to be FFT. So you could drop in other things on the FPGA. Yeah, math blocks. So, you know, and those were expensive. I mean, those were $5,000 to $10,000 a pop.

So putting FPGA is not going to be a cost driver there. But for small things, yeah, that's when probably you should start thinking, why do I need this? Yeah, yeah. I mean, FPGAs, basically they deploy more transistors than they need because you've got to have all those configuration switches but being able to reconfigure them is awesome um because you're like oh let's it lets you rapid prototype your way through hardware as it were like you can say okay first up let's just make the adc go and just stream out some raw bits. Now let's add the filters. Okay. Now let's add the digital down

converters for our radio unit. Okay. You can do that. Whereas if you're trying to do it with hard silicon, you're back to the very, very expensive to do a spin. You'd better hope it's right first time there is a thing called partial reconfigurability where you you reconfigure i.e reprogram part of the fpga at run at runtime yeah i've had to do that in my microprocessor to take stuff from this flash and send it out to the FPGA. Yeah, yeah. I won't say who, but I've heard that described as a foot-seeking bullet. That's a nice metaphor. Yeah, so that's the other thing. And I remembered another use case where you clearly want an fpga which is say you're

trying to do network packet analysis but you don't want to add any latency to the system any appreciable latency to the system and oh it's a it's a gigabit backplane you can do that with a big fpga you can you can stick it you know you you can you can pipe a gigabit signal through without sacrificing any throughput and with just a little bit of delay. And then as those packets go through, you can verify the checksums. You can look at the routing information. You can apply basic to medium analysis. You can do statistics counting. You can divert and copy packets with specific signatures.

Like, there's a lot you can do, and you can do it on the fly, and you can do it transparently. Whereas with software, you'd always be worried that it would get held up a bit somewhere, and then all your timing's gone. So FPGAs have repeatable timing. But getting into the world of FPGAs is non-trivial, in part because of these projects that I look at and say, well, I don't really want that board, I could just get a bigger processor. Unfortunately, they are difficult to get into. And I think to some extent that's inherent

because of what they are and the way you try to program them. What they are is hardware. So they care about arcane stuff like flip-flops and logic gates and lookup tables and set up times and hold times and eye-open standards and all that stuff. The way we program them is we feed in HDL, hardware description language. Now, I believe in North America, it's usually Verilog. And in Europe, it's usually VHDL. And in New Zealand, it depends because we are, as always, split between our British background versus American cultural import. But they do the same thing and what they what they do is you tell the computer i would like some logic that does this you're not telling them

what logic to use and then the computer runs huge and ornate and probably terrifyingly hairy if you get down into the source code algorithms to actually turn that into the magical bitstream which configures all the switches. So yeah, you need to think about it as logic and you need to think about clocks and resets and all your pipeline delays, if applicable applicable are exposed and you have to think about them it's on you and concurrency in a way that programmers probably don't think about uh yeah i mean at risk of at risk of triggering a flame war um i feel a flaw of verilog is it looks a bit too much like c so you put a statement you put a line in verilog is it looks a bit too much like C so you put a statement you put a line

in Verilog and it kind of looks like a function call but it's really really really not like really not because everything happens at once all the time every clock and that's awesome if you want to

monitor 10 rotary encoders and drive 10 brushless DC motors and servo loops because you can write the code for one and then get this. You can declare a for loop in your code, a for loop, and that in software, a for loop is in time because you'll come back and you'll do it multiple times you'll do it once you'll come back you'll do it again you come back you do it again you come back in verilog and vhdl with a bit of care a for loop is in hardware so if i say four channels zero to nine it's actually going to put down 10 copies of your servo hardware.

And they're all going to run at exactly the same time. And they're not going to interfere with each other. As long as they fit, as long as they can be fit into the silicon and meet the timing requirements, they will go. Which is awesome. Except they use the same exact terminology or syntax except in Verilog when you look at something and think it's a function call but it's not

because when you're going through software it's sequential execution but that's just not an FPGA you do end up with a lot of state machines, though, which do execute sequentially. Right. You have to finish the state before you go on to the next one. Yeah. All of the things involved with the state have to finish before you go on to the next one, which is why you have state machines. Yeah. Except when you get down into the,

so you've got your switch statement and then you get into a case and then everything in that case happens simultaneously. And then the next clock, it's going to look at the state and do something different. Only it doesn't actually look at the state. It fuses the state register with all the inputs and logic that the computer spits out to work out what the outputs are going to be and what the state's going to be all at the same time, all in one clock.

I'm used to my compiler going from C or C++ to assembly, which I can read in a list file, to machine code, which I can't personally read. You could with a reference manual on some paper yes but with this fpga i write it in say verilog or and and then it it does magic to get it to be what is essentially machine code at that point except it's not exactly it's gates it's yeah it's odd it's i think it's more monster than magic there's there's three steps to it and none of them is compiling so the first step um is oh it's been a while it's been a while the first step is to synthesize it which is to say you take the

allegedly high level hardware description language it's not high level and you turn it into valid you turn it into a collection of valid Boolean logic statements and

also call out to predefined blocks in your FPGA. Because once upon a time, people thought, hey, let's just make a sea of gates and they all kind of connect. And if we want RAM, we'll just configure gates as RAM. And if we want multipliers, we'll just configure gates as multipliers. And it turns out the performance just isn't good enough. So these days what you find in an FPGA in the logic part is there are the actual bits of logic, which is like a lookup table and a flip-flop. So you implement logic functions in the lookup tables

and you store stuff in the flip-flop. So far, so good. But next to that might be a big block RAM for just storing RAM. And then next to that might be a digital signal processing slice, which is a multiplier with some trimmings. And that's just an optimization to get the speed up. For example, if you fork out $400 for an Aria 10 chip from Altera, those suckers have about 150, the small ones, have 150 DSP slices,

which can do IEEE 754 32-bit floating point at 450 MHz. And there's 150 of them. so that is a lot of flops i feel like that part of the movie was it maverick where she says is that fast that looks fast that's very fast yeah altera do white papers on our stuff is the best. Yeah. But, you know, everyone does that. Stepping back a bit. So you've run your synthesis and you've got some, let's call it, call outs to real hardware blocks and also Boolean logic. Your next step is going to be to translate translation, which is to take those Boolean statements and translate them into configurations, which we're going to stick up and

stick into the lookup tables. So instead of saying, I want an or gate, you say, the truth table of this lookup table should be such and such to mimic an orgate. So what you've then got is you've got a collection that says, I want lookup tables with these truth tables in, and they need to be connected according to this netlist. And that's not too bad. That's the kind of process you can get your head around. Where it gets really, truly ugly is when it is the final step,

which is place and route, depending on your accent. And what that's doing is you've got a big, horribly complex problem where you've got a great big list of all these lookup tables you want to configure and connect, and these RAM blocks you want to configure and connect, and these DSP blocks you want to configure and connect. And they've all got to connect to each other according to the specified connection table,

which came from the translation stage. And they've all got to meet these timing constraints, which comes from manufacturer-provided tables, which say how long any particular function or logic block or multiply or connection takes, because remember there's time of flight in the wires. And then you've got to place all these logic functions in the silicon, like in x, y coordinates, such that they all fit, they all connect, and they all meet the timing requirements. And that problem is horrendously dimensional and iterative and time-consuming and awful um and it gets worse um because as you really start filling the thing up once you go

beyond about 80 full of logic um it gets harder and harder and harder to wire them up properly and your your synthesis times just go up and up and up and you may get to the end and you might come back in the morning and it says no solution sorry yeah my compiler never says that yeah sometimes it'll say out of memory well i guess my compiler says that sometimes yeah but but it won't say no solution um that being said the good thing about the fpgas is they when you've specified the timing you want once you've generated that bit stream want, once you've generated that bitstream, sorry, once you've actually passed through the hell of place and route, then you've actually got a guarantee

that it's going to meet timing. Whereas with software, it can be a bit unclear if you're guaranteed to meet timing. Like there's some non-deterministic stuff in there like tracing linked lists or cache mishes and stuff. Well, I wish me timing. It's just whose definition of timing?

Exactly. Exactly. And then once place and route is done, you know where everything is going to be in the FPGA. And then you can generate a bitstream, which is quite straightforward. Yes. That all sounded very straightforward. So the data, basically you have source files and you click the go button and go to lunch. So that, let's call it compilation process,

takes two minutes to overnight, depending on what chip you've got, what desktop computer you're using and how full it is um and yeah sometimes you just can't do it other times it's easy and you'll you'll get a feel for it um so it's it's really much less clear how your source code turns into bits than in C or C++ or Java or what have you. And that's the nature of the beast. So the two big things are the abstraction layer is both less powerful and staggeringly difficult. And fundamentally, everything happens at the same time. Asterisk and the asterisk is sometimes you have different clock domains inside your fpga

so you might run a vga output stage at 27 megahertz while your processor runs at 100 megahertz and what that means is that those clocks kind of run past each other because there are different frequencies. So the relationship between them is going to change all the time. And you're going to need to make that work. Yeah, that gets challenging so what would be a good beginner project um so that someone like me didn't just keep thinking gosh i'm just going to get a different processor the the one that i i mentioned they do floating point on arduino is the xlr8 board um and it's sort of cool, but their other feature that they do is NeoPixel control. NeoPixels are the WS2812s from Adafruit. And so they're the controllable LEDs

that speak a form of SPI. And they can be difficult to control because that form of SPI is not necessarily the same as what you would expect. And you can only put a certain number on a pathway. So if you're making a large screen with these, A, find somebody to do the power for you because that's too hard, but B, you need a lot of output to it. What other projects should I be looking at for FPGA that won't make me want to give up? Output to NeoPixel is actually a really good one because if you send some malformed data, it's not going to blow up. I would also suggest stuff like direct digital synthesis waveform generation. So you wire up, say, an SPI DAC, and then in the FPGA, you can make a lookup table for a sine wave,

and you can feed that through some gain and offset stuff, and then you can take those words and have your own SPI master that drives that out to the DAC with specified timing. That's quite a good one. It's a bit more complicated, but you can go to something like a Zinc, which has got, honest to goodness, ARM cores in it. Or you can do what's called a soft core processor. Yes, the name is inherently a bit odd.

Where you use up a chunk of resources to implement a processor in the FPGA. Those are the ones that make me crazy. Because I'm just like, if you want the processor, get the processor. It's not that hard. Go get another processor. But I do understand that that means you can put the IO and the algorithms and the processor all in one part and not have to deal with a whole bunch of little parts. Yeah, you're dead right that using FPGA resources to make a processor is wasteful. And a good way to put that is to say, if you've got, say, a Zilinx zinc chip, on the same chip, you can have a 600, 700, 800, maybe a gig if you fork out for an expensive

premium part. So you can have a 700 megahertz arm core and then next to it on the same silicon process you can have a 100 meg soft core microblaze that that's the zilinx brand one soft core processor so from the same silicon process and you you can have a 700 megahertz beast or a dinky little STM, sorry, or a dinky little micro blaze, which is, imagine a Cortex, Cortex M2, M3, but with an optional memory management unit,

but only goes at 100 meg. So in that regard, the soft core processor is a bit wasteful but on the other regard if it saves you a whole chip that's good and also 100 megahertz doesn't sound very fast but think about interfacing something with spi you might get say 20 megahertz 20 megabits a second is a pretty good connection rate when you're inside the fpga you can have a 32-bit bus at 100 megahertz so that's 3.2 gigabits a second versus 20 megabits a second on the spi i mean that's co-located with your processor oh and by the way you can make write your own peripherals and memory map them and wire up the IRQs properly.

So that's why you do a microblaze in the same FPGA. Well, sorry, a soft core processor in the same FPGA because it's already connected. Whereas if you put a processor down next door and interconnect them, you're either going to have your hardware engineer possibly raging, maybe crying, or it's going to be really slow. Whereas if they're on the same silicon, job done. If you need more grunt, you can step up to one of these Zynx from

Zylinks and there's a similar one from Altera. Dual core 700 meg ARM A7, have a nice day. That's quite a lot of processing power and then you can put your weird custom peripheral stuff in the FPGA section. Yeah so one thing I found really nice when we we were using a zinc as a control unit at abb um we had a custom or we had an interesting an interesting industrial ethernet this controller done on the logic part while you've got the logic part you can do very nice stuff like it can wrangle all your adcs and it can do over sampling and fir filtering and then just give you an interrupt when uh can just give you an interrupt or or trigger a dma when all your adcs are sampled all your input scaling is done all your FIR filtering is done and everything is nicely lined up in contiguous word aligned memory blocks

so you don't have to screw around with SPI, it's just done Yeah, and it'll work the first time too I achieve pretty good rates of something happening first time but I tend to simulate the heck out of my FPGA designs. That makes sense because it is a bit of a difficult... Well, all right. So you've convinced me that I should go play with FPGAs more than I have. And I know Chris has got a board around here, so I should go do that.

Maybe more than one. Probably more than one. I think we are about out of time. I can tell because my stomach has started growling and um yeah well because because i'm yeah kayaking really cool lots of otters in the monterey bay but a little tiring if you've never done it stop looking at me like that i'm not crazy christopher do you have any last questions? I do not. Jonathan, thank you for putting up with us. It's been a pleasure.

Any thoughts you'd like to leave us with? Yes. Yes. I do have a thought. And the thought goes, drumroll please. He'll put that in later. Yeah. You aren't the only one with problems. And you should try to be kind to people who make mistakes, especially when that person is you.

Good sentiment. That was a great final thought, Jonathan. I hear you also have a plug for a conference you'd like to say. And if you happen to be a power electronics type person, just a reminder that the extended deadline for the IEEE Southern Power Electronics Conference, SPEC 2016, is on the 20th of June. We look forward to your submissions.

My guest has been Jonathan Bradshaw, senior electrical engineer. Thank you so much for being here. Oh, my pleasure. My pleasure. The downside is going to be, of course, that in a few weeks when I listen to the podcast, I'm going to be like, ah, that in a few weeks when I listen to the podcast, I'm going to be like, ah. Well, there's three options. One will be like, hey, this guy's saying a lot of stuff I agree with.

That's great. And the other option is going to be like, why am I on a podcast? And the third option is I'm kind of going to have an off week for Embedded that week. Well, there is some value to not listening back to your own voice because it's pretty painful to most people but if you get if you do listen you'll get to that final thought which says you know partially be nice to yourself too so it's good to end there thank you also to christopher for producing co-hosting, and paddling.

And of course, thank you for listening. We are getting ready to do t-shirts. I'm so excited. The new logo is so cool. You'll love it. So check out our blog and newsletter for information on that. And of course, I'll announce it here. You can find all of that on Embedded.fm, the website, along with a contact link if you'd

like to say hello, or if you'd like to, you know, tell me that I should have this kind of shirt, because I know you have opinions about it. We will be here next week. In the meantime, a final thought to tide you over. Oh, the puns, they never stop. This one's from Tolkien. Apparently, I was a little focused when I was writing this. Folk in those stories had lots of chances of turning back, only they didn't. They kept going because they were holding on to something. That there's some good in the world, Mr. Frodo, and it's worth fighting for. Embedded FM is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not

put them there and do not receive any revenue from them. At this time, our sole sponsor remains Logical Elegance.