Embedded - 25: Thunderdome for Antennas

Episode Date: October 30, 2013

 Jen Costillo surfaced briefly from her startup-induced blackout to share her wisdom about manufacturing consumer products. They discussed new product development and working from (and making modifications to) Hacker Dojo in Mountain View, CA. Jen and Elecia pined for this (probably not really a two pack) microscope.

Welcome to Making Embedded Systems, the show for people who love gadgets. I'm Elysia White. Today, Jen Castillo is here to talk to us about what she's going through with her startup, as it's going to be manufacturing soon. Jen, it's good to have you on the show. This is only take four. I had to come up from air. So much startup stuff. you've been really busy i mean it's a startup induced blackout was made a lot more sense mostly it's just i i'm

i'm up to here and i'm a very short person and it's at least six feet of of stuff that i that i have to wade through each day and prioritize the best advice that i got before i jumped into this was make three piles the stuff you need to do right now the stuff that you're never going to do and what's the middle pile the the i don't i don't know which way it goes but you need to do that pile first to figure out which which the triage pile yes the no no it's not even triage it's like i don't know which way it goes, but you need to do that pile first to figure out which... Oh, the triage pile. Yes. No, no.

It's not even triage. It's like, I don't know whether this is important or not right now. And I need to file it quickly and then get back to that first, that main pile. So you've been ramping up on manufacturing. Yes. What's been on your mind with that? Well, what I've really been reminded of repeatedly is it's really a garbage in, garbage out situation. What do I mean by that?

Well, if you're not very clear and specific about what you expect to get back from your contract manufacturer, because that's, you know, that's typically what people are doing, then you will get garbage out. If it can go wrong, it will go wrong. Yes. And not just in the normal cosmic sense. They will find incredible ways of making it go wrong. Well, their profit margin is so small that they will take shortcuts on every single part. Yes. And in order, I mean, they will reuse

resistors if you're not up front with what your expectations are. Yes. Yes, absolutely. Well, that wasn't an issue that I've noticed yet, is the reusing of resistors but phil talked about it last time he was here i was pretty appalled but if for example you know i'm working on a wireless device and the antenna is so important to this

product that and it's such a small product that you know're not going to talk about the actual product. We're not going to talk about the actual product. When it ships, we're going to talk more about it. When we get closer, we will actually talk about it. But it's wireless and it's tiny. Yes, and it actually has three antennas on it. Three?

It has three antennas on it. Okay, that's a lot. That's a lot of antennas. Do they fight there is a thunderdome for the antennas and two going at a time they are seated awkwardly but um but i mean gps will uh fight with like2.11 and other protocols. If you've got three different things, you've definitely got something in your flow.

I am very lucky. I have an incredible hardware engineer with not one decade plus of experience, multiple decades of experience in GPS. And in RF antenna design? Yes. But you brought that up as a manufacturing issue. So here's the deal. You know, you have a small enclosure and you need to make sure, since it's a consumer product, you need to make sure it's sexy. Which means we got to hide

that antenna in some interesting ways. Yes, because if you hide it on the outer edge of the device, somebody will touch it and then you'll get all kinds of bad press. We're gappy. Yes, because if you hide it on the outer edge of the device, somebody will touch it and then you'll get all kinds of bad press. We're gappy. Yes. Thank you, Apple, for that demonstration of what not to do. Yes. So we don't do that. Good. However, you know, we have a separate board just for the antenna. And so it's very important to make sure two things. One, that it is, one, connected to our main board such that the antenna data actually gets back to the main PCBA. But the other thing is to make sure that it fits nicely within the enclosure. Why? Because if you have an air gap, that changes how the antenna works. It's all acoustical transmission media related. So if you have an air gap between where the

antenna is and your plastic enclosure, or whatever that is, that changes the characteristic and how effective your antenna is. So you need it to be snuggled up. I mean, really touching snuggled up. So that's why on some, you know, some, some, let's say some of your cell phones, you'll actually see the antenna built into the back casing of the phone. Right. If you take it apart to get out your SIM or to do battery stuff or just to see what iFixit has for you to look at. You'll see that. Yeah.

I think the, I can't remember which Nexus it was, but one of the Nexuses had, I don't think it was the GPS or the GSM antenna was there, but I definitely remember the NFC antenna was right in the back in particular. Well, and that's near field, so it needs to really be as close to the outside as you can right um so anyway so one of the things that we thought was interesting is so so the way that you

attach this antenna board to our pcba is surprisingly complex and if you poorly document it you will get poor outcome uh so the board was fine. The mechanics was okay. The connection was okay. As long as you built it, as soon as you asked someone else to build it, they did something a little different and made it non-functional, less functional. Right. So, so for example, so what you would need to do, we were using a mixture of conductive epoxies. Now then, as a startup person, I've had no experience with conductive epoxies. Usually I would have a mechanical engineer, a double E, someone else, materials engineer, someone who would be more knowledgeable about these materials, procurement, durability, reliability.

But suddenly now I need to look at data sheets for conductive epoxies. Oh, and conductive foams. Yes, yes. Well, that's the beauty of working at a startup is you get to wear so many hats. It's such a learning experience. It's a firehose of information. That's also the downside of working. I have to wear all the hats.

You have to wear all the hats. Um, so, so we, you know, to, know, to make this long story much, much shorter, the way that you attach the epoxy, you know, just like you would attach any epoxy, you need to rough up the surface, you need to clean it, rough it up, apply the epoxy, and you need to apply enough of it. Well, they weren't applying enough of it, and they certainly weren't prepping the surface beforehand.

No, because that's time and that is where they make their profit. If they can cut steps, they will. Yes. So the helpful metric that you should always keep in your head is how long is it taking you to manufacture one device? And you should always assume that manufacturing costs will always be at least one dollar per minute so think about that when you're putting together anything so we've already we've

already made it more complex in addition then there's soldering that has to be done so there's solder pins that go in through the the the pcba and the antenna board and those have to be soldered even at a dollar a minute for soldering that's still cheaper than putting a connector in there yes i mean much cheaper yes i don't even know how you would do a connector there quite frankly too tiny yeah um so you would just you would just solder these in the problem is is that if you don't use enough solder remember the antennas are big sheets of copper if anything goes to ground that's another big ground plane that's difficult to warm up it's very prone to cold solder joints ah but that means i mean this really indicates you need more manufacturing testing whether it's it's checking for connectivity or transmissiveness. So are you doing all the automated tests?

Or is your CM doing them? I don't believe, for this particular part, I do not believe that they're doing this type, like a flying probe test. No. Well, even once you have the board all together and the system packaged the can you see a a signal a signal at a at a good enough rssi to a signal strength uh to ship the unit right so so we'll we'll talk about that in just one second but i wanted to follow up

the the you know you can't even do that test until you do final assembly which means when you have the final you know uh casing also on because otherwise your antenna won't necessarily work because it's tuned to assume that there's going to be a plastic layer on top right okay so the other thing that's interesting is that when you have those solder, you know, those solder to soldering the antenna, those extra solder gloves, if you put too much solder and it's on top, that means that the casing won't lie flat against. Too little, it doesn't work. Too much, it doesn't fit. Right. And so the final thing is then what we ended up doing to deal with that because you don't know how big these solder lumps may be. They may be perfect solder, you know, perfectly smooth solder,

but if any raise means that it won't lie flat against the casing. So we actually had to mechanically adjust for that. So, I mean, this is a design for manufacturing issue. It is. DFM is one of those acronyms that the electrical engineers and the mechanical engineers have to deal with a lot more than the firmware engineers do. That's true. But I, you know, as leading the engineering team and not having necessarily a mechanical engineer local, then this is the acrobatics that I get to go through. Oh, this is definitely, I mean, yes, this is all aboutrobatics that I get to go through.

Oh, this is definitely, I mean, yes, this is, this is all about being a tiny, tiny startup. Um, having to do all this yourself and, and having to learn everybody else's job so that you can ask the right questions at the right time. Yeah. And you weren't there at the very beginning of the startup. So some of these things you might've asked, you might have realized, hey, at every other company we had a DFM checkpoint. But you were a little late for that. Yes. So we're making changes as we go along.

So part of it is we have a distributed team. In fact, I would say on a daily basis, I see maybe one other engineer. The distributed team is pretty interesting because it's not just one other engineer. Your CEO is 400 miles away. Your electrical engineer is 2000 miles away. My electrical engineer is actually not too far away, but he has another job. Right. Again, tiny startup. Yeah.

And then our CM is in China. Most of our logistics are in China. And you've been without a permanent office for many months. I mean, since the beginning there has never as far as i know there's never been a a an office for the startup that was one of my first goals when i started was to find an office how's that going well attention hardware startups. If you would like to use co-working space because you cannot afford actual office space, you will have an incredibly difficult time finding co-working space. Why?

Because you solder. And damn, you're soldering. Damn, you're soldering. I'll get you, my pretty. And I'll burn down your office too. That's the concern. That's the thing is you can't just go into one of these places that you can rent two or three cubicles because you have no lab space and you can't solder on your desk because everybody hates the way it smells or they're afraid you're going to burn down the office.

They're afraid I'm going to burn down the office. But you've been at Hacker Doja for a while. Yeah. So the economical, I mean, so, so our, I mean, going back really quickly, there was coworking facilities and then there was places like Regus, which lets you off, you know, rent out off at larger offices and they would let you solder. But man, there is such a premium. That is a racket. Yeah. If, let me tell you, if I had the time and inclination, I would rent out an office building and turn it into a co-working space. Because as far as I could tell, it's more money, more money, more money. Because they charge, they charge you for beverages. There's a beverage service charge.

Well, I mean, in the Bay Area right now, in Silicon Valley, the real estate is getting hot again. We went through a period where there was plenty of open office space. But it's getting expensive. Yeah. So I am... Okay, so that's a co-working space. That's where you share an office with a whole bunch of other small companies. A hacker space is where you don't necessarily share an office with a whole bunch of other small companies. A hackerspace is where you don't necessarily share an office.

It's kind of an open building. It's a community center, effectively. Oh, that's a good point. For the wayward geeky. And Hacker Dojo is one of the first and one of the best known of the hackerspaces. And they have, if you become a member, which you don't have to become a member to use it. No. But if you become a member, you get a key, you get access 24-7.

You get to be part of their vote on what's going to happen to the building committee, which I would pay not to have to be on that. But there's soldering there. There's oscilloscopes. There are lots of tables, a couple of rooms. Microscopes. Oh, they have microphones. They have Dremel.

They have a 3D prototyper builder. Oh. They have a MakerBot. They have lots of hardware stuff. They have a hardware space. It is not the best hardware space in the world, but if I just need to solder one thing. It's better than going to the tech shop or even going home to your home soldering system so it's if you if you don't have that

particularly if you're trying to work with a team the time it takes for someone to go home solder come back is is is is punitive if you just need to tack on a wire. Yes. And at Hackard Dojo, if you're a member, you can reserve spaces for like, I've been there for a couple of coding events or classes. But if you're not a member, you have to talk a member into doing this for you.

Yes. You need someone to, what's the word? Be the host? Well, host yeah i guess that would work but you need you basically need someone to speak on your behalf and vouch for you and and and take up your cause um so it's a hundred dollars it's a hundred dollars a month a month which is for one person and you can kind of bring in other you can bring in other people they just need to sign at the desk, same as you.

And the other people don't have a key, but they don't cost any more. And in fact, I've gone up just to meet with someone, realizing that if they had no space, we wouldn't be able to meet, but it was at an off time. You'll always be able to find space, even when it's crowded. You may not be able to find parking, but you'll be able to find space. So the issue I was having is, you know, the people that I needed to meet with,

most of them were kind of down in the South Bay in the near Mountain View. And then one person was in San Francisco without a car. So I need something close to public transit. That was the other gating item. And this solved all of it. And public transit here in the bay area is becoming way more popular as traffic gets bad it's even bad at like 11 45 i mean i i would love to take public transit all

the time but but you live in it takes it takes too bloody long it takes too bloody long so so let's go back so hacker dojo hacker dojo so so so you've been using that as an office so once so so yay you know you get there you do your thing everyone can hear all your conference calls all your big business and everybody wants to come and talk to you and ask you about your startup and tell you about theirs everyone wants to come talk to me what are you working on we don't get many hardware hackers here it is primarily software yeah it is primarily software but i and i don't know how you feel this i i personally get very irritated by the being called a hardware hacker especially when this is my startup but

i don't have a problem with hacker but to tell me that i'm hacking on something and when this is my professional work it is maybe hacker dojo though it is so i mean maybe if you were at like startup dojo they would not call you that no they would still do it yeah um so so one of the interesting stories i have for you about being at hacker dojojo is, you know, we were there and, you know, once again, wireless device. And we're not talking like a local wireless device. We're talking something that, you know, we've already mentioned GPS. So GPS in particular, we were not getting it. It's kind of impossible to test indoors.

Yeah, it is really hard um i mean here at the office i used to put my gps window uh antenna in a window but i had a big antenna and i still was i'm only seeing like a quarter of the sky there right and it still would take you and 45 minutes to lock so even with with some of the assisted methodologies that we have within our chipset, it would get better, but that was only a later revelation. What was interesting was,

since we were members of Hacker Dojo, or at least I was. This is where that $100 starts paying off. This is where that $100 starts paying off. They have a fund for improvements discretionary fund and we decide you know i talked with my hardware engineer he's like oh yeah we can put in a gps booster which we would have never been able to do at a co-working facility yeah when you can't have a saturn iron you definitely can't run an antenna wire to the roof.

To the sky, right. And back into the building. So I basically said, hey, we would like to do a GPS booster. And we're just going to put it up in the skylight. Doesn't require us to go through the building. You didn't have to drill in this. Didn't have to drill. We just needed to get it up above the grating into the skylight.

That's in the main portion. And what was nice about it is that we're basically providing a benefit to everyone in the building. Anyone, I don't know who else is working on GPS anything, but at least they could work on location-based startup ideas now. It's much easier. Yeah. That required us to, you know, it went through very quickly. We supplied all the materials, we supplied the antenna and everything, and it remarkably improved our situation. See, that's cool. It would have

been, I mean, the times I've had to install GPS in a building we rented, we didn't usually ask permission. And we had to drop the cable and it, and it was always a pain. The GPS boosters are pretty cool because it's an antenna on the outside, and then it kind of just reflects to the inside, right? It isn't a hard wire. So if you were doing iPhone development and you wanted better GPS connectivity, you've helped everybody at Hacker Dojo who does that. Yes.

That's really cool. It's nice that you can actually help others as well as make your life so much easier. Absolutely. Absolutely. So we went through that. I thought that was interesting because how exactly do you deal with working with wireless RF issues when you don't actually own the office space? And Hacker Dojo made it very easy for us to do that. Most of the spaces that have been like this, that I've been to, have been very open to that sort of thing

because they recognize it might be them who wants it next. And I'm thinking like tech shop. Yeah. They're pretty open with, I really want a microscope. And it only took them about a month to get one. Yeah. At which point I didn't need it anymore. But now I know where there is a really good microscope

and that's going to save me from putting one in my garage. That's much better than what I did with my startup, which is I got on Amazon, and I got those goofy jeweler eyepieces. Not the monocles, but the glasses that you put on. And you flip them up? Yeah, you can switch out the lenses, so the lens power. And so I will have two of those, one for my hardware engineer, one for myself when I go to china so i can see what you

know so i can verify my first article is that what you're gonna wear for halloween because that would be cool well they do have a lot of the bug person okay they're very stylish they have not come yet so maybe if they don't come in soon then so i can't ask you how well they work because i've looked at those and wondered i have you know reading glasses that are far beyond what I need so that I can look at boards. Well, I would hate to give, I would hate to be remiss and not give someone their just dues. But there was, you know, I was able to finally use a piece of swag that I got at the Embedded Systems Conference yesterday. And it was like this delighted... You got it at the Embedded Systems Conference yesterday. And it was like this... You got it at the Embedded Systems Conference last spring.

I think it was... You used it yesterday. Yes, I used it yesterday. It was somebody... Maybe it was ThreadX. I don't remember who it was. Gave me a magnifying glass. Was it the monkey?

Oh, yeah. I don't know what I'm going to do with the monkey. Thank you. Finally, I used it for this monkey. Magnifying glass with a light on it. my god cool help me help me help me do some verification earlier today because when you're when you're dealing with really tiny parts and polarities i've had this uh this this uh microscope in my amazon wish list for god it must be about yeah when are you getting that so i can come over and use it well it's 429 and since i put it in that's it they've had yes it retails

for 2000 this is two count i know it says two count and usually if it says two count that means there's two in the box maybe they mean two eyes i i i just you know if i could get two microscopes for 400 i'm sure i could sell the other one to you. Look, why don't you just ask the vendor? Oh, all of the other information. I think the two count is just a bug, but it cracks me up and it makes me want to buy it even more. That is a good price. Oh, the 430 is a great price and it's a great microscope.

Let me see, which one is it? You need to send that back to me so I can get my CO to buy that. The Amscope 3.5X to 90X with light. Light's important. You know, we deviated heavily. We went from talking about manufacturing. Did we have a point? I don't know, but your producer is eye padding surfing the web it's all like wrap

it up y'all okay so okay so so i got those glasses so i could do first article testing which is for those of you who are mostly firmware engineers that is the first what they're going to do when you when you go to your cm and you're and you're just starting you go over to china and you go to the cm you don't have to up the line. When you go over to China and you go to the CM. You don't have to go to China. No, no, because there's CMs here. I've been to those CMs. Yeah, it feels like a different world because it feels like the overhead's not so bad.

So first of all, they're going to run through and they're just going to do one board, one panel maybe. And all they're going to do is they're going to stuff everything and they're just going to make sure, hey, did we assemble this correctly? And that's when you need to remember with your brain, you know, what the polarity is, what the orientation of everything is. And you probably also want to do some power ground plane testing just to make sure. Like all the stuff that you would usually do when you get boards fresh, you want to do again. Even if you've already gotten PCBs once and tested it out in a prototype manner. That might have messed up. Once you get it on the line, you need to do all those tests again.

OM out, making sure that power does not connect to ground or that all of your connectors connect where you want them to connect. So, okay, so you OM it out and then you pop some software on the right well unless it's well okay so you say own it out like for me i'm just like i need so one of the things that's that's very interesting about the fact that you know with consumer products we've really shrunk down things so small that one you can't even find you can't even stuff a reference designator on the silkscreen anymore. Oh, no.

You have to know where everything is. You have to know. And the test points aren't even there for you to touch, really. They're for machine things to touch. Right. So flying probe has suddenly become negligible in some ways because there's not enough test points for them to necessarily do anything. Basically, it's the Vs. Yeah, so you're basically,

how many times can I say basically? We could say it five or six more. Should we just chant it? Well, basically. Oh, sorry, sorry. Let me get my thesaurus out. Essentially, essentially,

you need to find like how you can you know what the most critical test points are so those are usually going to be around your regulators if the regulators aren't right then you don't have power and if you don't have or you have the wrong power yeah um so what those are usually bigger components yeah Yeah. So this, yeah. So this, well, not on this thing. This is not very on these ones. Oh my God. Everything's so tiny. I wish I could show you, but you'd sneeze and it all blow away. So what I spent time with my hardware engineer, because there were some instances where they had things backwards because it wasn't clear what the polarities were even from the original

silkscreen because everything's so tightly packed together is that we had to go through and augment the assembly drawing with basically a test drawing that would tell you what to do as you were getting the first assembly the assembly drawing will this this new improved assembly drawing will have more dots. Clearly marked dots. More dots, yeah. And so not only is that supposed to help the CM, but that's supposed to help my dumb ass when I go out there. Because I did not design this board, but I'm sure it's all going to be the one there.

And with boots on the ground, you were the first line. Yep. And you're in the time zone, which means you'll be asked everything, even the wacktastic mechanical questions that are probably furthest from what you do professionally. Right. But as long as you, I mean, I'm by no means am I going to do I wanted to mean mechanical engineers everywhere. I have a lot of respect for the fact that you make my product look awesome. But in many ways, if you give me closures together and I can kind of, you know, I can fit a puzzle together and I can kind of see how things fit together or don't fit together. But if you're going to ask me like how durable it is or whether this is going to be waterproof or not. Well, can we take this shortcut, which is what the manufacturer is going to be asking you.

Then I'm not as useful. Okay. So we get through first article testing, then we will start, we will, you know, usually you will. Well, you've only tested the board so far. You've only tested... Now you have to put it into something. Well, so let's go back to the first thing I said. Garbage in, garbage out. Okay. If, you know, you should already know when you get to China or your CM what the manufacturing process is.

And you should know all, you should have all the documentation for all the tests that are going to be run. And that includes the automated software tests, but also if there are any pogo pin tests or anything like that. Yep. Each, each test station, you should know what goes into that. And you should have a very clear idea of what it's testing and how it's testing it. What's interesting, at least from someone who didn't originally design the manufacturing process for this particular product, was how simple it was.

But what I could quickly find out is that, you know, one, there's not a lot of flying probe tests on it, and there wasn't a lot of functional tests. So usually if you can't flying probe everything, you can't get good test coverage there, then you must do more functional testing functional test is what you put in the software yes so it would be that means that either you need to have your firmware needs to have some sort of

diagnostic back end or something that's innate to it that will allow the test station, usually a PC, running Windows, to query it or to observe some interaction. And that's why those debug serial ports are so critical to firmware engineers. Don't have that. That's how we're going to be doing manufacturing if we can. Nope, we're doing wireless. But you can't.

Oh, you're doing wireless. Everything has to be wireless. Some of the ones I've worked on have all been self-contained. So you have to push the buttons in a certain special order and then the unit will cough up using whatever UI mechanism it has. But doing it wireless means that in order for it to come up at all, your processor has to come up and your wireless has to come up. And if you don't't then you have no additional

information on what's wrong yep so so some of it can be done so some of this is more oh how do they say so some of it is is very dependent on what you have in there so for example if you just want to test hey can i reach each one of these attached components like for example an accelerometer which I know you've covered in previous shows quite thoroughly hey if your design isn't using the accelerometer right now but plans on doing it in the future you sure as hell better test oh to make every single component, everything you're connected to, whether it's I2C, SPI, ADC, GPIO, all GPIOs have to be moved in order to verify they aren't tied to the one next to it. Just talk to it and understand if there's any calibration that

needs to be done. If it's, if it auto calibrates itself or if there's something that you need to keep track of for later. Well, things like Mac addresses in wi-fi units and uh and making sure the mac address is somehow attached to the serial number that's on the outside of the product so that when you register the device it works all of the registration information is a pain um and calibration like your touchscreens often need calibrating to, so that the, where you touch is where it looks like it's instead of just like off to the side. So we're testing things. Make sure you test everything. The other thing that, um, for most people who've never done manufacturing before, imagine

doing the same test a thousand times in one day well that's that's that's the dollar an hour right so imagine doing the same thing ten thousand times a day and it sucks i mean it's a terrible job and you you know if your manufacturing tests are not completely obvious when they fail they're going to pass something that fails. It needs to have a giant red screen that comes up on the PC that says, don't ship this one. Yep. Because if they can click through, you know.

They will. If the little drinky bird. Blinking, flashing lights. Yeah, alarms should go off. I would say alarms could work, except for the fact that many times the manufacturing floor is noisy so it's not necessarily it's not necessarily people talking in fact it's not people talking go on it's machines it's the drone of the machines

um and your mileage may vary there but i don't know the fans the just i've never met a quiet cm area it's almost there's like a heavy white noise in the air, and you feel a little desensitized, a little aural fatigue. Well, when I worked on audio products, we had difficulty qualifying it in the factory because the background was so noisy. We couldn't tell if the microphone was meeting its specs because we couldn't get into a system.

So what I've seen people do... Anechoic chambers. Well, not exactly even anechoic. hell if the microphone was meeting its specs because we couldn't get into a system so so so what i've seen people do chambers well not not exactly even anechoic it's just like ink box enclosure baffled with foam okay the poor man's anechoic chamber the producer disagrees no um well i think one of the other things you need to know if you're going to the cm probably long before you go to the cm are all of the acronyms i mean it's been acronym soup lately and everybody seems to be using the same acronyms at leapfrog we had our own set and then about two years ago everybody started using the same there's there's theoto and then the DVT, which is the design verification test. Is that what you're going for? We are going for... And then there's

EVT, which is the engineering verification test. I think at this point we are gearing up for DVT. Okay. I'm saying that very tentatively. It could go either way. I mean, you don't always do each one separately. So ideally, when you get to DVT, this is your dress rehearsal for production. You should not be making a lot, if any. No major design changes at that point. If you're going to do anything, it's maybe a bomb change.

A component change. And usually limited to discreet. Yeah, and discreet. Discreet. I would genuinely say discreet are what you're aiming to change. I don't know if that's DVT. I think that's EVT.

Really? DVT, you're allowed to spin things a little further and you don't necessarily need manufacturing. DVT is the first. You do proto and then you do DVT. And then EVT, the engineering verification, that's verifying all of the steps. That's verifying that your manufacturing process is as you intend. And getting ready to make all of it.

All of the tests should already work at that point. For DVT, if your tests don't work, then you get to try again at EVT. Okay. So, and this may be a function of my startup situation. I'm having to overhaul many of the tests that we're putting into, even at DVT level. You may be doing DVT2, which is...

Oh, I don't know if we can afford DVT2. DVT6 is bad. Yes. I mean, ultimately, at the end of the day, you can call it DLL7. No, I like the other two. But the fact is you're wasting money But the fact is you're wasting money. The fact is you're wasting money. Okay, so DBT, Design Verification Test, Engineering is next, so EBT.

Oh, no, actually we do it. Everywhere I've been, it's been reversed. It's all EVTs, then you go to DVTs. Really? Yep, and then you go to PVTs. When was the last time you manufactured? Oh, sorry. Well, since I've got EVTs coming back next week, and I'm pretty sure, I mean, all right.

A little bit of Wikipedia is going to go really fast, right? That's fine. That's fine. Okay, do it. Do it. And then I will blather while you type. Yes. Okay.

So typically, that it, do it. And then I will blather while you type. Yes. Okay, so typically that has been my experience. EVTs come way before and then you do DVTs and then you do PVTs, which is, you've made it. Success. Oh, you're right and I'm wrong. Totally. You know what? You've already paid me in COCO. Ah, totally. You know what? You've already paid me in cocoa. Ah, yes. EVT, engineering verification test, is what you do to get the tests all correct,

to test that your bring-ups are good. Yes. And that's the first time. That your grand idea of connecting the who's a what's it to the microchip controller was correct. Okay, so EVT and then DVT. And then PVT. And then PVT, which is production verification test.

That's where they actually use the correct tools. Up to that point, they may be making your enclosures out of, you know, boxes. Star foam. Star foam. Children. Children, not children. And then after PVT, where they've tested their tools.

This small consumer product's really big. It doesn't fit on my hip very well. There's mass production. And it winds for food. MP. Yeah. And this is all part of the new product development, which must have come from some book,

but I can't figure out which book it came from. The Book of Life. Thanks. I don't know. Okay. And the even better part of if you go to new product validation in Wikipedia, at the very bottom, there's the food portal. Wait, what does food have to...

Nothing. Maybe this is how they make Cheetos. They had so many iterations. I'm totally amused by the internet being wrong today. It's often wrong, but I'm not often amused by it. So you went to Wikipedia. So I went to Wikipedia, the new product development page, and it goes through the steps and all of the blah, blah, blah.

And then it references food. And then on the bottom where it normally would say like engineering portal or science portal or technology. It says food portal. I feel like you're going to have one meeting, or not one meeting, you're going to have one show just on research and reliability of sources.

My producer now asks, do you think this is what people tune in for? I'm sure people tune in to listen to me surf Wikipedia. I have no reason, I have no idea why they wouldn't enjoy this part anyway let us deviate from this particular portion of the show and let's get let's let's go back to testing okay so here's the other thing that's interesting you know it used to be if you were the you know if you were the new kid on the block in the company like for example one of my first firmware jobs was you know design this whole

product oh and since you're you're the newbie you get to do all the manufacturing tests you need to figure out how we're going to test all this stuff and make sure you don't take too long because we have 10,000 of them waiting for you so I actually step up from maintain the old product, but yes. No, it was a startup, so that's what I got. So that's how I became very intimately aware of all the crap jobs, but important crap jobs that there are

in engineering. And different kinds of manufacturing mean different kinds of tests. There's a big difference between millions of units versus tens of units. So I had to write the manufacturing system for my product. And I had to figure out ways to make it fail, to make sure it was robust, that would catch things that would fall through the cracks. And then I had to deploy it to the manufacturing site, which was luckily in Fremont, because this was back in the days when you could actually do

manufacturing. There are nice CMs in Fremont. There are. back in the days when you could actually do many things. There are some nice CMs in Fremont. There are. I still use them, but they're in the tens of units sort of. Right. And so I learned very quickly, like you said, bright flashing red, green for pass. Using pictures, not words. Yeah. Because they, many times they cannot read the words on the screen and quite

frankly you don't want to really have to read them the other thing you want to do for automation is you want barcodes qr codes automatic logging automatic you need a database yeah to say what tests pass what tests fail what time, and ideally, who ran the test. So that you can log in later and say, I see that 20 of the units failed. Where are the 20 units? They're with Bobby. They should be in a pile,

and you should be able to give me 20 of them. And if there are 15 of them, that means there are five units out there that are going to get awful reviews and may cause fires and it's just bad so so so going back so on your test station duh you want to make sure that not only can it tell you what the error that there's an error and there's something wrong and it shouldn't go forward but what kind of error it is so it can get routed to the correct person for either reworking, re-evaluation, and so on. Or trash.

Or I don't know anyone who trashes. Well, some things are so cheap that if they fail in a certain way, you just toss it and start over. Fair enough. Yes. Even those, you can recover housings and whatnot. Because you're, I mean, you really, your goal is to get to 98% yield. Because that's what we're talking about here.

If you throw out too many units because your accelerometer isn't mounted correctly or it's a little canted or whatever. Cantilever. You're going to lose money. Yep. But if you take 10 minutes to run the test instead of two, you're going to lose money. Cost too much. It's a balance. And if you have your engineer take three years to write the tests,

not only are you going to lose your product window, that costs money too. And I mean, I can't overstate this enough is, you know, if you write tests that basically rubber stamp everything that goes through your ceo will kill you when you start getting thousands of units back in rma it it's yes it is better and cheaper to test and test well than to do a recall yeah which is even worse than having them all rma'd although rmas will's will give you crummy, crummy Amazon reviews. I'm looking at you, you little product.

It was annoying. Oh, sorry. So what else have you been doing with manufacturing? What haven't I been doing with manufacturing? So what we're talking about here is something that, one, not all startups are going to be able to do. Especially something that's size of three or four or five engineers on a team that's about all you've got isn't it i have less than that wow and i get to talk to everyone so so that's the other thing you know at least at least from my point of view i i effectively have

to count myself as half an engineer because I spend so much time on the managerial and logistics and talking with people. Well, and fundraising too. And fundraising. Yeah. So half an engineer. So I'm effectively half an engineer. That's great for your self-esteem. Yes.

So what do you do when you're a startup and you don't have all that much time to put into manufacturing? What can I tell you to help? First of all, try to leverage as much of your bring up testing infrastructure that you already have. It's important that you focus on testing everything that's coming off the line as thoroughly as possible. But given the volumes that you may be working at, you may not be able to have a huge database infrastructure. Understandable. But it is important that you test every component that comes there,

particularly if you plan on using it later. Some people will suggest statistic sampling, and that's fine as long as you're willing for everything not tested to not work statistics being what they are especially when you're trying to build a whole lot of something they may choose the first one manufactured each time which means that the second through ten thousandth may be broken yeah um so you got to be careful with statistic sampling and yes you have to test every component on the board that you ever plan on using even if the software right now doesn't work otherwise don't stuff it and don't waste the money and and your your board bring up tests the things that you use to initially test

your board you should think about those as manufacturing tests from day one because that's what they're going to grow up to be. Yep. What else? Yeah. Basically, don't try to be overly clever with your testing. Don't try to outthink what, you know, just to reiterate, if you designed it, you test it. Don't try to sidestep any of this just because I've seen it bite me in the butt. I think as a grown-up engineer who, who finally has taken responsibility for getting a system shipped to getting a system completed,

you have to take the responsibility for testing your code. It's not somebody else's job because there, you know it better than anyone else. If you can get into a, you know, the pair programming thing where you test each other's code back and forth. Yeah. Awesome. But you know the pair programming thing where you test each other's code back and forth yeah awesome

but you know what we heard about an engineer recently who had done the prima donna thing and had threatened to leave unless they hired a junior in a junior engineer to come and test all this code that wasn't lunch maybe you were talking to someone else um yeah and i was like so some poor guy has to try to figure out all of your code, and he's never going to do as good a job testing as you would have. I'm tired of hearing about that. Test-driven development is important, and even if you don't do all the unit tests, think about testing as you're writing the code.

A sanity check. So now then, have another engineer on your side. Um, sanity checking you is always good. Oh yeah. Do not. So if you have a CM, do not use your CM as your sanity check. Some CMs are great and will do sanity check, but it depends on how busy they are. If they're super busy, that's what falls. If you are a startup, you are not important enough and you will lose face. They will think that you do not know what the heck you're doing. So do not, do not skip, do not pass go. You will go to CEO jail if you, if you skip internal reviews of your schematics, manual, mechanical enclosures. Do all of those checks internally before you

release to the CM. Make a strong and really get people to focus on it. And particularly if you're writing up procedures for rework or manufacturing test, you guys really need to look at those carefully and really focus on, you know, get everyone's undivided attention on it because mistakes like that are extremely costly. And diagrams are good. They're better than text as long as you can make them clear. And English, if you're writing it in English, it should be what you would find in a computer science textbook. It should not be if the unit is recalcitrant that sort of thing is fun to write i think i saw that in a computer science textbook you really might have written it so uh and yeah it's it's important to not dumb it down just make it clear and i would also advise that

you know depending on who's writing your manufacturing test, whether it ultimately is the CM or you, make sure you document it. I can't overstate this enough. Just spending the five to 10 minutes to draw up a diagram and put it somewhere in a wiki or somewhere where it's documented will save you and everyone else a lot of time. Yes. Make your documents easy to find. Well, I think we're about out of time. I'm not sure we're out of topic, but I know you have to get back to fundraising or maybe doing some engineering work.

I think I have to put together product today. Ah, yes, the fun put together product. I like it when you get to see it all. So I actually have one last thing, which, you know, while your target audience may not be able to help me explicitly here,

if there's anyone out there who's interested in being a front end or front end slash back end server architect, I mean, we're looking for a guru here. By all means, email. Email show at embedded.fm and I will connect you with the startup.

Since we're not talking about the startup today, I'm happy to be the go-between. Or you can contact Jen on LinkedIn. You don't know my LinkedIn. Oh, no, you're right. Contact Jen on Twitter Twitter at robots on fire and that's where all of the

vowels are numbers it's in the show notes or email show at embedded.fm you can send us your comments and questions as well or hit the contact link on embedded.fm and that will pop us an email as well thank you so much for being here thank you for having me or hit the contact link on embedded.fm, and that will pop us an email as well.

Thank you so much for being here. Thank you for having me. I'm going to go gasp for one last breath and then go back down. Yeah, I think there's a little more hot chocolate there. And thank you to Christopher White for producing this show and for being patient with our many, many tries on getting it started. And thank you for listening. I hope you've enjoyed our little ramble through manufacturing. So now we're going to go sing the happy bug song,

since Jen wanted to sing earlier. Unfortunately, it's licensed, so we can't do it on air. Actually, given my singing voice, you should be glad about that. Really glad.