Embedded - 3: Plenty of Candy, No Guns

Episode Date: May 29, 2013

Elecia White and Phil King of Weekend Engineering talk about things a hardware engineer wants software engineers to know. Drifting a bit from topic to topic, they touch on interviewing, oscilloscopes, ways to light hardware on fire, why they work on projects at home and writing novels. Some links from the show: Phil works at Lytro making amazing cameras. Elecia and Phil have worked at Leapfrog and ShotSpotter together. Very different products. Phil's oscilloscope (the one Elecia borrows) is a Tektronix DPO4034. At Phil's instigation, Elecia wrote a space opera novel for NaNoWriNo a few years back. (If you contact us, you can have a PDF for free. But really, she wrote it in a month, what do you expect? Buy her real book to get the good stuff.)

in Terrabang. Welcome to Making Embedded Systems, the show for people who love gadgets. This is Elysia White, and this week I'll be speaking with Phil King, one of my very favorite electrical engineers. The plan is to hear what a hardware guy thinks software engineers should know. Hi, Phil. Hi, Elysia. Welcome to the show. Hi, Elysia. It's good to be here. I know you've worked at some neat places, made some really cool products like children's toys for LeapFrog and a gunshot location system for ShotSpotter. We did those together. What else have you been up to? I've worked on educational laptops for kids at a company named NetSchools that was doing infrared networking. I have worked on network boxes, which was not

particularly interesting in the looking at it sense, but there's lots of fun problems to solve moving network traffic around. I worked very briefly at Amdahl on mainframes. Seemed like a good idea at the time. That was my first job out of school, and that didn't last long. Oh, and I've been a game programmer. So I actually have done software a little bit, but not enough to call myself competent anymore. But now you're at Lytro. They make a camera, right? Yes. Lytro makes the first consumer planoptic camera, which is to say a light field camera. The idea is that instead of capturing a two-dimensional array of pixels, we capture a four-dimensional array of light field information. It has both direction as well as intensity of light at each location. So you can do things like refocus the picture after you've

taken it. So I can't really take out-of-focus pictures anymore using the Lytra? At least not in the depth sense. You can still get motion blur and other problems like that, but as far as focusing to depth, yes. If you use it properly, you can refocus to any depth in your field of view. Well, that sounds like a pretty cool gadget. It is. What do you do there? I am a senior hardware engineer, and since we're a consumer company, that of course means that we do our manufacturing in China. And so I work with our JDM partners who are doing basically implementation of our design ideas. And so I spend a lot of time in Taiwan working with engineers there.

And then some of my time in China overseeing things at the factory. And when we worked together at ShotSpotter and I was a manager looking for new embedded software engineers, looking for more minions, really, you were one of the best people on my interview team, helping me weed out who would make a good team member and who probably wouldn't. And even better, you could articulate what you liked and what you didn't. What was your secret question? One of my favorite questions is to start with a really simple, and this is looking for software people, a really simple coding problem. And that is literally the simplest coding problem there is, which is I give someone an editor and a compiler and I say, please write hello world, which is literally

just a program that prints the line hello world to a console. So void main void printf hello world semicolon. Exactly. And first step is to see if they can actually type a few lines

of code with someone staring at them, which is an inherently intimidating, nerve-wracking situation. But I want someone who can actually go through the whole process of creating a trivial source file and compiling it from the command line. And then you hit on some of the interesting stuff right there. You said void main void. And that's not exactly right, is it? Well, it depends.

I start to investigate exactly what gets handed into main and what is happening around the execution of that program. Because that's where it becomes really interesting from an embedded systems point of view. Yes. What exactly happens before you hit that P and printf? In an embedded system, that's especially important. What do you look for? How do you get them past just typing? Well, generally, one of the interesting questions, I always try to have a small suite of different editors available because you quickly run into the, are you a VI or an Emacs person?

Or sometimes there's people who just want to use DOS edit. I don't think anyone's ever tried to write their source in Microsoft Word, but you could. But basically, once you get something that they can feed as an input file to the compiler, then we're on our way. It's perfectly fine if the compiler barfs the first time and they realize, oh, there's a few typos, and I want them to go back and fix their typos. And preferably, another important part of the process is talking through what you're doing. Obviously, I'm less interested in the fact that someone can write hello world than in the fact that they know what's going on inside the computer. One of the great things about this

question is it very quickly separates people who are pure CS background from people who are embedded systems, because a pure CS person will tend to view the computer as this idealized box containing some mystical glowing light that executes algorithms. An embedded systems person will view it as a chunk of hardware with a whole bunch of setup requirements and power constraints and thermal requirements and so on and so on and so on and so on. And so, as you said, asking what happens before you get to main, what's happening in the system. And if you didn't have an operating system, obviously they're working in the context of a console. But if you didn't have a console, if you didn't have an operating system, if you didn't have any external support around this,

what do you as the embedded system code designer have to do to get this code to run on a piece of hardware. And there I'm looking for understanding of all of the sort of setup requirements, you know, knowing what goes on at least. Where printf even goes. Exactly. I mean, do you Morse code it out to an LED or put it onto a debug serial console or push it over USB? Exactly. Exactly. And seeing, first of all, if people think to ask what the constraints on the system are. Ideally, someone should want to know what the constraints are. And then seeing how they chase down that line of reasoning. Okay. So once we've typed hello world and they've admitted that there is more to the whole world than just that, what's next? Well, to extend the questions about what's going on, I'll ask them about what if there are other processes running on the machine?

What if there's interrupts that are going on while you're doing this. I will also ask about some fairly simple extensions to the program, like add, change the code so that it takes a command line input and outputs a name, hello world, blah. And that requires that they understand what argc and argv are and where they're coming from and the proper syntax for handling them and so on. And just extending from there. This is a very C-heavy question. If someone can't program in C, well, they're not going to get so much of this. They're welcome to do it in assembly language. If someone says, well, I want to write this in C++, then I'm going to start investigating, well, do you understand the

implications of all of the sort of object support that C++ implies? The best example of this was when I was working with someone writing test code for a simulated processor that was running on an FPGA. So it was running at very, very, very slow speed. And originally, all of the test stuff was written in C++, which means that you spend most of your simulated processor cycles running constructors for all of your C++ objects, which is really silly. Well, that just sounds like a pretty bad design. Yes. And for the most part, embedded software engineers tend to write in C.

Certainly, if somebody comes up and say, can I write this in Python? You're going to say, well, what processors have you worked on? That wouldn't be kind of cool. I do like Python. And there's some cool integratedava engines and stuff like that but i've i've not got any real experience with those i don't think i understand those words cool integrated java well okay it was cool 20 years ago you always really cared about making sure we hired good software engineers uh why i mean i know why it's important to me to hire a good team but as a

hardware engineer you don't usually have to work with them. Absolutely, I have to work with them. The software people are the ones who make me look good. I mean, first of all, you can create the best hardware in the world, and then a bad software designer can utterly strip it of any elegance or functionality. On top of which, you know, the software, a lot of the hardware optimizations now depend on software. Everything from proper compiler optimization through power management, all of that is hugely dependent on the software

making good use of the hardware. And in fact, you can even end up designing hardware that can conceivably be destroyed by software oh oh i have done that yes i try to design things so that that is not possible or at least very difficult to do but you know whenever you're moving motors or moving power around you know there's always the possibility that someone is going to run a motor past the stops or stall it. You tried to design everything so it's safely interlocked, but you can't guarantee that's going to happen. Well, no. I mean, a software bug that has an infinite loop that ends up hitting a

motor stop and the motor keeps going and going even though it stopped. Yeah, that one kind of required a fire extinguisher. We don't need to talk too much about that. And EPROMs and flashes, you can't just keep reading and writing to the same address where it's going to be destroyed. Instead of having a device that lasts for five years, you have one that lasts for six months and suddenly you have a huge recall. And that does make the hardware engineer look kind of like an idiot. And on the test bench, good about the the e-proms you know there are some e-proms that have a hundred thousand cycle uh hundred thousand write cycle life and and if the code is is just smashing on the same block you can tear through a hundred thousand write cycles pretty quickly you can actually run through the entire usage life of the hardware in an afternoon.

Oh, less than that. Much less than that. I won't admit to that one, the motor thing yet. So early in my career, I once tried to learn more about hardware. And it wasn't with you. It was before. I mean, this was really early in my career. And I built a little barcode reader when you could do that out of a kit. But I was overseen in my job by an electrical engineer who was great. And I showed him what I was doing. And I would sit at his bench and solder and

then we turned it on. Actually, we went back to my office to turn it on. I should have been suspicious at that point. I had asked, you know, is this how I put everything together? I had laid it out and I'd soldered it together and every capacitor was on backwards. And they were the capacitors that it matters which way they go. Were they tantalum or electrolytic? Did they pop or did they flare up? They popped, each one, like a little series of fireworks as they popped and went over

my cubicle wall and to the software engineer next to me. And it was just pop, pop, pop. And yeah, that one didn't require a fire extinguisher, but it was at that point when I realized software was more my career than hardware. Every hardware guy has a day like that. There was one of my fondest memories of almost exactly the same problem, and that's why I asked if it was tantalum, was I was working with a senior engineer when I was very young, early in my career,

and he was trying to emphasize for me the importance of putting the tantalum capacitors incorrectly as polarity, he attached it backwards while holding it. And it burst into flames in his fingers. And he managed to let go of it before he singed his fingertips too badly. But it always stuck with me that as he was explaining the importance of the polarity of the capacitor, it burst into flames in his hand. Yeah, thinking back to some of the chemistry lessons we had in school, certainly the ones that involved fire were far more memorable. I still don't remember the chemistry, but the lesson was awesome. Exactly. Blowing stuff up is fun.

So I totally agree that software engineers can do a lot of damage to hardware. And hardware engineers can do a fair amount of damage to software. So many times I hear, well, let's just fix it in software, as though we have any more time than you do. Please, please explain yourself. Well, one of the problems is that we're often trying to design for constraints or features that are not well defined early enough in the process to get into the hardware. The other problem is that, you know, hardware design and software design are exactly the same, except that our compile time is three to eight weeks. You know, you can do a series of experiments in software in the course of an afternoon.

You can breadboard things in hardware in a day or a week. But if you actually have very dense electronics where you have to spin a PCB, you have to do a schematic change, a layout change, get it into fabrication, get it into assembly. And you can speed up the fabrication of a new PCB by throwing money at it to a degree. But there's some point at which no amount of money will make it take less than some number of days, depending on the complexity of the board. And then likewise, you have to get it assembled, and that takes some amount of time just to get all the parts put down, get everything soldered together. So you can just change things faster in software. And so, yeah, it is often considered a perfectly acceptable solution for the hardware guy to

say, well, we'll try to provide more capabilities than will be needed, and then we'll iron it out in software. Give us some examples of when you've done that. Oh, let's see. When you and I were working on gunshot location sensors, we went through multiple rounds of discussions about where the audio front-end filter should be. And we ended up putting some... Part of the problem of gunshot location is getting the gunshot noise or the parts of the gunshot noise that you need to locate on and none of the ambient background noise that tends to lower your noise floor or raise noise floor. Ambient background noise, to be clear, includes birds tweeting, even ones right on top of your sensor, which is loud.

And helicopters, which sound a lot like AK-47s if that's all you hear, and you're a computer, and even backfires. Yep. And so what we tried to do is filter some of that out, take load off the DSP that was doing all of this audio processing by putting analog audio filters on the front end. And we ended up with some of that, but not nearly as much as we had originally intended because, or as we originally discussed, because we came to the conclusion that it's better to have the ability to dynamically change that filter in software rather than have it fixed in the hardware. Exactly. If we need to pass more of some frequency or halt some other frequency, changing that to be an FIR filter or even an IR filter in firmware is far easier than trying to change a thousand boards that are on top of buildings or in the field. And getting the boards back and changing each one, it's much more difficult. Exactly. And getting the boards back and changing each one, it's much more difficult. So, yeah, I do recall that.

And I totally agree that software was the place for a lot of it. But we had to put some in the hardware or we ended up with noise. I think maybe that's a whole show about signal processing. Let's just say Nyquist is your friend and maybe leave it there. And also one of the things along those lines, Nyquist, remember, always remember, Nyquist is the absolute minimum sampling rate and you can demonstrate how sampling at the Nyquist frequency will still give you no signal if you happen to be aligned with the zero crossings in a sine wave, for example. I think we need a whiteboard. Yeah, that doesn't

work on radio, does it? So you and I have gotten along. We shared an office for four years and didn't manage to kill each other. Amazingly. You did leave occasional notes around the office like, guns and candy may not be stored in this office. I didn't think it was appropriate. That was at ShotSpotter and not at the toy company. At the toy company, there were no guns, I promise.

Plenty of candy, no guns. So going back to where we started with the interviewing and what hardware engineers look for in software engineers, it's more than just the coding question. Absolutely. What do you look for for soft skills? I mean, obviously, communication skills are everything. You could argue that coding itself is just a form of communication. You're communicating your intention to the compiler, but also you have to be able to communicate your intention to your customers, both internal and external. I need people, you know, I like working with you because you and I communicate very well

and have enough shared context that we tend to understand each other fairly quickly. And when we don't, we have the patience and the ability to generally explain what we're trying to get across. And the trust that the other person will wait for the answer to become clear. Yes. I've worked at places where everybody assumes whatever you do is malicious, and it gets pretty horrible quickly because if you can take, I mean, you can take anything wrong, and in an email you can definitely take it with the wrong tone. I know I have sent you emails that just say, fix this,

and you just assume that what I meant was, good morning, Phil. Nice to see you. I was in this morning and it was a little busy. I seem to have broken X, Y, and Z. Could you please take a look and maybe see if I have? And if I have, could you see if you get it fixed? Pretty please. Thank you.

But you accept that because there's a built and shared relationship. Yeah, and that also raises the other side of things, which is I'm definitely looking for people who don't take things personally very quickly, people who are fairly even keel. There's an interesting sort of dichotomy, or there's a problem there in that people who are in engineering are often very passionate about making the things they make uh and and that

passion comes across as an emotional engagement that is occasionally seemingly out of scale with the situation at hand. And so people who don't escalate quickly or unnecessarily are always a good thing. I saw a tweet this week that was, if you want to feel like a complete idiot and then a complete genius, every five minutes, go into software engineering. I think in hardware engineering, because you do have the slower cycles, you tend to have a longer time between feeling like an idiot and feeling like a god. Well, and I don't know, but the highs are higher, but the lows are definitely lower, I would argue. Because if a piece of software doesn't work, you know that you are, maybe, you're ideally one minor bug fix and recompile away from a repair,

or maybe some refactoring, or worse comes to worse, you've got some major algorithm rework. But there's never this horrible feeling of, oh crap, I have a large pile of hardware that I now have to throw away and replace because it just isn't ever going to work. Software isn't usually money that gets tossed. And that's an interesting point. It actually is, but it's not so obviously real money. The money is hidden because you're paying the people who are the money is in the form of the time of the people who have to fix it. And it's just as real money, but it's not,

but there's nothing that ends up in the trash can that you can look at and cry over. So that about covers my questions for you. Do you have any questions for me? Sure. When you're interviewing hardware guys, what are you looking for in your, in your double e's it's funny my technical questions aren't very difficult and uh and the secret is that i often am not 100 sure of the answer uh the question i think i last asked had to do with h bridges and

motors and the truth was i didn't understand the question. All I really wanted to know was, could he explain it to me? Because I needed to know the answer at some point. And I wanted to make sure that my hardware engineer could tell me what it was he knew about the circuit so that I could wander off and actually work with it. And if a hardware engineer can't tell me about a difficult circuit, then either he can't communicate with me, because that's critical, I completely agree, or he doesn't know hardware engineering. Either way, I don't want to work with him. So how do you discern, though, between someone who's giving you a good explanation

and is just really good at making up crap? I know enough hardware engineering to just barely make that. Because I often don't start with, this is going to drive a motor. And if you have an H-bridge, chances are it's doing something along that lines. And the guy who comes up and says, oh, okay, so this has to do with a thermistor or a motor. Then I know we're in good shape. Okay. And then something else that I always like to ask people,

and so I'll ask you as well, is do you do any sort of engineering projects for your own amusement? And if so, what do you like to create on your own time? I do. Lately I've been trying to create a podcast. So we're learning a lot about that. I read a book, I just finished it last week, that was called The Half-Life of Facts. And I think you might enjoy it. It wasn't a great book, but it was pretty good. And the idea is that the information you get while in school has an expiration date, kind of like uranium half-lives. And like uranium isotopes, you don't know which ones are going to expire. Half the information you got in school is out of date and useless at this point. And unless you keep up and keep thinking, you're not going to be able to get new facts to replace the ones that go out of date.

You're not even going to know which ones are out of date. So I tend to want to do lots of different things. Part of me wants to work on little gadgety things at home, and I do. And you've helped me with those, so I know you know some of them. But I want to learn all sorts of stuff, whether it's reading nonfiction, in addition to my hideous and horrible fiction addiction, or whether it's taking a class on how to do stained glass, which actually improved my soldering skills just a little bit,

or even doing the podcast. I've learned a lot about how to set these things up and some of the software features that I need to deal with. But technical projects working at home, you helped me with the shirt that I was working on that was a posture improvement. And it had a series of motors and series of sensors along the back in a T-shape and would help people sit up while they were working at their computer so that their static posture was better. Okay. And then, of course, you're also working on all sorts of interesting writing projects.

You've got your O'Reilly book. I do. I have Making Embedded Systems, my O'Reilly book, coincidentally naming the podcast as well. And I did NaNoWriMo one year. I was working at ShotSpotter with an electrical engineer who came in one August and said, you know what I want? I want to write a novel. You know, you really shouldn't take it face value when someone says they want to write a novel. You know, you really shouldn't take it face value when someone says they want to write a novel, because maybe they like the idea of writing a novel more than the actual process of it.

Or maybe they were busy that month and fell behind by about November 5th. I said, I will if you will, and I am still waiting for my book. I've read your novel. It's very nice. No, no, it's not. So software has a lot of process. You said that a second ago, you just said the word process and it made me think about it. You know, we've got the agile process, which is, well, I don't want to talk about it. We'll talk about it some other time, but there's a waterfall and agile and all of these named processes. Are there hardware processes like that and if not how do i

convince somebody that that's a good idea oh boy uh there are there are a series of steps that i go through in the course of a hardware design i don't know of any formal process name or description for them. There's the product lifecycle and the hardware development lifecycle that we go through. And it's named differently in different companies, but it always is the same sort of progression from original collection of requirements through initial designs, engineering validation, followed by design validation, followed by production validation, culminating in mass production. This is very much like the waterfall process. Okay. Because, I mean, you think about water goes from one spot to another and then to another. And sometimes you have to loop back,

you prototype and that feeds into the requirements. But for the most part, you're going in one direction, and you have specific, I would say, punctuated equilibrium places where you stop for a second, waterfall doesn't go up by an order of magnitude each step in the way because it's not like you're you're suddenly in when you're doing a product development as you as you get toward mass production you produce more and more and more of the hardware and so every incremental step costs more and if you screw it up you have more and more hardware to throw away and redo. And that's a good reason for not using the Agile process, which lets you make your decisions very late in the process. So, yeah, okay. That makes a lot of sense.

That said, early failure on – I do think fail early and often on a hardware project is perfectly valid up front when you're in a breadboarding stage, when you're in a lab stage, where you're doing experiments. If you don't know what's actually possible, you there to try and build a little front end circuit or a motor controller or something along those lines. That said, it's getting harder and harder to do some of that because more and more of the packages, the component packages, are so completely unsuitable to anything but final assembly onto a real printed circuit board. You can take a 20-pin dip package and you can solder leads to all of the wires and you can literally build a wire harness with no board at all, no substrate holding anything together. And there's some really brilliant prototypes that I've seen people build up that just looks like a ball of wire with components sort of glommed into it um that is what most of

my soldering projects look like do theirs work afterwards because mine don't they do but but the problem is that you can't do that if one of your components is a 142 ball bga package that's eight millimeters on a side with you know know, half millimeter ball pitch. You just, there is no way short of a, at least a breakout board and probably a full up printed circuit board for your circuit to, to do anything with those parts. So it's, it's, it's definitely made certain types of prototyping a lot more difficult. Well, what about home prototypes for you? Home gadgets? What are you working on? Oh, I've got a whole bunch of different things that are sort of in the, gosh, wouldn't this be neat stage.

One thing that I built recently was a shutter timer test, a shutter timer rather, which is just a sequence of LEDs that get cycled through one after another at whatever clock rate you want. And the idea is that you take a photograph of this, and then you can tell how fast the shutter on your camera is from how many LEDs you see lit. Because you know, for example, if each LED is one millisecond, and you see five LEDs lit, you know that your shutter time is five milliseconds.

Now what's interesting, though, is if you take two of these series of LEDs, and you set them up so that they're at the top and bottom of your field of view if your shutter time is five milliseconds uh or say if your shutter time is one millisecond you will see one LED lit but it will not be the same LED at the top and bottom because you're not actually capturing the same one millisecond section of time because on a lot of cameras if you have the shutter in the focal plane the shutter is actually a traveling slit the shutter can't move fast enough to be all open and then all closed in one millisecond it instead has a essentially a one millisecond wide gap moving down the imager is it moving i mean i'm like thinking about

motors and and moving things it's really it's spring-loaded usually uh but but for example if you take a dslr camera and you take a picture of the a good way to to see this uh manifesting without building up a timer timer like this is just take a picture of a fan. And you would think that the picture would freeze the fan blades in one place in time. But because the moment that it captures at the top of the image is not the same moment at the bottom of the image, you get distortion where the fan blades actually appear to be twisting. I'm going to have to try that. It's kind of cool.

I think that's better than taking apart my husband's camera. Probably. But using the LEDs you can actually quantify it and you can say, well, the shutter exposure time is one millisecond, but that one millisecond is actually four milliseconds apart between the top and bottom of the frame. Neat. Yeah. Last week with Jen, we were talking about oscilloscopes and voltmeters. And I mentioned that I often borrow an oscilloscope and it has shaped what I want in my own oscilloscope, the one that I'm pining for.

Yours is the scope that I almost always borrow. And I have to say, I really do appreciate it. I love my new logic widget, but a scope sometimes helps you figure out the big problem instead of the little tiny one. It is a Tektronix, right? Yeah, I have a Tektronix four channel scope. Do you know the... I can't think of the model number off the top of my head. I think it's... It's an MSO something. Yeah, and I'm not remembering exactly what number it is. It's their sort of medium higher-end scope,

and it's fast enough that it will catch most reasonable embedded-type signals. It's not something you'd want to use for super high-speed serial stuff, USB or MIPI or HDMI or any of that sort of stuff. But you're right that using a scope will often catch things that you're just not going to see otherwise. Transients on power. For I2C, it's a protocol where one side kind of lets the other side gently pull up the signal. If you're using a digital logic analyzer, that looks like a hard line.

But if you're using an oscilloscope, it looks like a soft little hill instead of a cliff. And knowing how long it takes to get up that hill makes a big difference for how you configure your software. And it will also show you if the logic value is sitting somewhere in the middle because both sides are trying to drive it. If you look at a signal, the logic analyzer will resolve it to a logic value one way or another, even if it's not actually a logic value. Because it only knows ones or zeros. Exactly. The scope will show you what the actual voltage is. And so if you were buying a scope, and you kind of know what I want, a little bit of digital, a little bit of analog, where would you start? Are you a Tektronix lover? I am biased toward Tektronix just because I really like their user interface.

It's, to me, a little bit more usable than some of the Agilent scopes that I've used lately. And I think part of my gripe is just that Agilent, several years ago, went toward all embedded windows. And I don't know if this is a fair characterization anymore. I must preface this by saying that I haven't actually used an Agilent scope anytime recently. Our producer's making gagging sounds and faces because he has used an Agilent scope recently and it was brutal. But when I used one of their high-speed scopes about a year

and a half ago on a trial basis, the problem that I had with it is I would turn the zoom or pan knob and there's a notable lag before it does anything and so I'd sit there thinking it hadn't gotten the command so I'd turn the knob again and then it would update for the first turn and then it would go shooting past what I wanted to look at so of course then I turn it back and I overshoot the other way and it's just it's it was it was a little it was a little laggy now that, I'm sure there are people who feel exactly the opposite and love Agilent and dislike Tektronix. Yes, I think to some extent it's what you grew up on. It's whatever your first scope was. It's like Apple with getting into educational sites.

It's important to put your scopes in colleges because that's where they get indoctrinated. Absolutely. And what about LaCroix? I have only used LaCroix equipment a very little bit. And then there's a couple of Japanese makers that I've had very little exposure to. One of the things that I found telling, though, is that even in Taiwan and China, where I've spent a fair amount of time in the last couple of years, the two big brands in use there, even though, even though there are a lot of Asian, uh, scope manufacturers, Asian test

equipment manufacturers in most of the labs that I've been in, in both Taiwan and China, it's all still Tektronix and Agilent. So there's definitely an international, they've got a good international footprint. Interesting. Yeah. So you mentioned going to China. Yes. And I remember from our leapfrog days, a penny mattered. Yes.

Oh, you're probably looking for some of the, there are lots of funny stories from China. A penny matters, especially when that penny gets multiplied by a million units. You know, if you, if you can knock a penny off the bomb cost of a product, you're going to ship a million units on. That's an appreciable amount of money. That's a million pennies. That's $10,000. Do you ever wish they'd just give that to you? Ah, yes. And that goes straight to the bottom line. But that also means that part of the reason that China has been a desirable place to manufacture things is because labor is very cheap. You got a lot of people, the cost of labor, which has been going up dramatically in the last few years,

but for a while it was very cheap. And so things that can be done with labor instead of hardware, you always throw people at the problem. And this was brought home to me actually about 10 years ago. The first time I went to China, I was there and I had sent ahead some instructions for a test fixture that I needed to have built for a charging circuit we were working on. And at one point I specified that I wanted a 10-ohm power resistor. I think I said a 10-ohm, 10-watt resistor. And of course, when I say a 10-ohm, 10-watt resistor, what I'm envisioning is an aluminum housing, they're usually gold-colored power resistor, which is designed to dissipate either 10 or 25 watts, basically a power resistor. And when I got over there,

I realized that power resistors are expensive, but telling some poor tech that he has to assemble 100 1K resistors in parallel to build a 10-ohm resistor out of 100 smaller resistors is perfectly acceptable. It was plus or minus 10%, right? So you really only had to do 90. Exactly. So I just laughed because it was this giant bundle of resistors that were very carefully soldered together. It looked beautiful, but it was a good illustration of how my thinking was, yes, you spend $3 and you buy a power resistor.

And their thinking was, we already have smaller resistors and we have some guy who doesn't make much money and he can solder things together well so we'll have him build a power resistor another related story along those lines is that margins are so thin in a lot of chinese electronics manufacturing that they will there are corners that will be cut that you wouldn't even think someone would think to cut. For example, I was having some prototypes built a couple of years ago, and we were having problems with one of our solder processes because we were using some special components required to low-temperature solder.

And so I ended up over at the prototype shop, and this was a shop in Taiwan. And I did a shop floor audit walk having them walk me through the entire process and at one point they i said well i want to see the solder paste you're using confirm that it's the proper low temperature solder paste they get out the can of solder paste i look at it it has the right name on it the right part number but i noticed that the name of the company was misspelled now Now, I have never known, and it was a Japanese make of solder paste. I have never known a Japanese company

to misspell their own name. Well, I've done it. Well, and then there were a bunch of other misspellings on the can of solder paste. And so I quickly realized that it was in fact counterfeit solder paste. Now, the fact that anyone would make counterfeit solder paste now the fact that anyone would make counterfeit solder paste was kind of boggling to me but there it is you know

there was there was money to be made somewhere by taking i suspect what they did is instead of using low temperature lead-free solder they just took leaded solder which has a similar thermal profile but is by the way chock full of lead um and just scooped it into cans and stuck fake labels on it and we were able to sell it for a dollar less than than the expensive japanese made real low temperature solder paste because when you're making a million and you use a half a can of solder on a hundred that does add up. Yep. And so, of course, the warning that falls out of that is you really have to watch out for that, especially right now. During the economic downturn of the last couple of years, a lot of the part manufacturers

did not ramp up their production capability. And so as the end product manufacturing cranks up, there's going to be more and more part shortages. And I have a feeling that that's going to draw more and more counterfeit stuff out of the woodwork. As the availability of components goes down and the prices go up, there's more and more incentive to create fake products. And so I think that's going to be another interesting problem in the next couple of years. And it'd be fine if the fake products were almost as good as the real, but that never happens. It really doesn't. And if you look online for counterfeit components, you will find some truly hilarious things like electrolytic capacitors. Someone will

have a 3900 microfarad capacitor, a 3900 35 volt capacitor, and inside it is actually a 1000 microfarad 25 volt capacitor. So if you just measure the capacitance, it comes in way low, but it's also underrated for voltage. And so it will, it may in fact work if your circuit is designed, is well over-designed, but it may in fact not. And if your circuit's that well over-designed, there probably was a reason for it. Yes. Well, you know, fire is fun in the right circumstances, but not in all of them. No. And then there's also the sort of borderline where products,

there's a lot of products you'll find in Asia that are, they're not counterfeit, but they're designed to be intentionally confusing, especially if perhaps English is not your primary language. One of my favorite is the Panasemig battery. These are batteries that look remarkably like Panasonic batteries, except they are extraordinarily cheaply made. They actually, the outer jacketing metal is so thin that you can squeeze the batteries with your fingers and the electrolyte will gush out the ends. And they have about 10% the battery capacity of a real AA, but they're hilarious. As long as you don't need them for anything. Sorry, have I wandered completely far afield here?

No, no, no, no, no, no. Someday we're going to have story time. But I think we probably should wrap it up here. Alrighty. And I do expect we will talk some more in the future. Excellent. Well, thank you for joining me. And thanks to our producer, Chris White, and to everybody tuning in. Please leave comments and questions at embedded.fm or email us at show at makingembeddedsystems.com.

We love to hear from you. Next week, I'll be speaking with my Logical Elegance business partner about the wonderful world of embedded software consulting. The good, the bad, and the ugly. Until then, have a good one.