Embedded - 352: Baby's First Hydrofluoric Acid

Episode Date: November 20, 2020

John McMaster (@johndmcmaster) told us about the process of opening up chips to see how the processors are structured and what the firmware says.  See John’s website for information on getting started (as well as digging much deeper). John has given some interesting Hardwear.io talks including Capturing Mask ROMs and Taming Hydrofluoric Acid to Extract Firmware. His talks and many others are available on the Hardwear.io archive. Or sign up for the Hardwear.io Online Hardware Security Training, Berlin Jan 2021. As mentioned in the show: John wrote a blog post about his top lab accidents and explosions. Paper: Reverse engineering Flash EEPROM memories using Scanning Electron Microscopy by Franck Courbon, Sergei Skorobogatov, and Christopher Woods Rompar and bitract are the two programs mentioned as helpful for getting from an image to binary code.

Welcome to Embedded. I am Alicia White alongside Christopher White. Let's talk about the inside of chips and what they look like with John McMaster. Hi, John. Thanks for being on the show. Hi there. Thank you for inviting me. Could you tell us about yourself as if we met at the Hardware.io conference when it was in person? Sure. I would say in the community, say the embedded community in Twitter, I am mostly known for my work on computer chips. What I like to do is I open them up, that is I decap them, and I look to try to figure out how do the circuits inside those chips work. You know, I take microscope images, I analyze, you know, some firmware maybe

I find inside them, and then I post that on Twitter and various publications online to kind of share with people what I found inside these chips. And I want to talk to you about the process of that and why the chips look the way they do and all sorts of things. But before we do that, we want to do lightning round where we ask you short questions. And we want short answers. And if we're behaving ourselves, we won't ask how and why. And could you give us all the details? Are you ready?

I am ready. Give it to me. Favorite chemical? Oh, geez. Hydrofluoric acid. Least favorite chemical? Oh, geez. Hydrofluoric acid. Least favorite chemical? Least favorite chemical. I'm worried if that was the answer.

I'm going to go with cinnamon aldehyde. All right. I think I need to know what that one does. Mercury-ish? No, it's actually very random. It's pure cinnamon extract. And the reason why it's my least favorite is because a bottle of it broke in my lab and everything smells like cinnamon now.

And that was years ago. And I don't think it's going away by now. Okay. Okay. Back. We have to behave. No. Does a civilian-owned tank require a driver's license?

Oh, dear. Someone did some digging. Oh, that is a whole can of worms. I don't know if that many people know about that. Short answer, no. McMaster car or digi-key? McMaster car, of course. Favorite processor? Favorite processor? mcmaster car of course favorite processor favorite processor um i guess that i would go with 68k is that also the prettiest processor or is there a prettier one it's the prettiest because i did

an involved project with it and i fond memories of it. Complete one project or start a dozen? Complete one project. That is the new marching order. Do you have a tip everyone should know? Take things slow. Always get a perspective when you're having a difficult time. Okay. According to your LinkedIn bio, you have a semiconductor failure analysis lab in your garage, including a high power optical microscope, a lapping machine and scanning electron microscope. So are you building Frankenstein? What are you doing here? Funny you mentioned that because my original career choice was actually genetic engineering. But I looked into that and it was going to be a lot of schooling.

And so I went with 10 years, maybe. And I went with computer science instead because it was a little bit more approachable, but also something that I deeply enjoyed with robots and whatnot. And so not so much a Frankenstein machine in a biological sense, but sure, AI is kind of thinking and, you know, playing with a circuit. So it's kind of a Frankenstein machine of sorts. So you've got a CS degree, but you do mostly hardware now, right? I would say that one of the things that's been really tough as kind of an identity crisis

in Silicon Valley is I do have a computer science and computer engineering degree. But if I say that I do computer science, people think of me as kind of a big data engineer. And that's really not my skill set. You know, it's more about, you know, real-time embedded operating systems, you know, maybe Verilog, you know, I2C drivers, you know, that sort of of area so i don't know if you would necessarily call that a computer science field or not you know it's kind of in between that and electrical engineering in the embedded space and depending on who you talk to i would say you get different reactions about that well i think for me it would be yeah that's what i do too but i don't have a scanning electron microscope well we could we do have nothing

stopping us we do have a good optical microscope although i don't know that i'd call it high power that's an okay optical and i honestly don't even know what a lapping machine is it's a dog yeah the dishwasher um well okay we could we could go through those. So lapping machine, that is one method to take apart a computer chip. You may be aware that in the old days when they were making chips, that they would maybe just kind of sputter some metals onto a chip. And if it looked about right, maybe they'd etch some off and kind of call it done. But as the number of layers increased on a chip, they found that if they didn't smooth out the layers between them using a process now that's called chemical mechanical polishing or CMP,

that planarizes the different layers of a computer chip. So when they add the next layer, it goes on straight and doesn't get the crooked artifacts from the layer below. From the failure analysis perspective, which you could vaguely call the work that I do, I basically do the same thing but in reverse. I would like to take one of these

smooth layers on a chip and I will polish them off. And that's an alternative to using something like hydrofluoric acid. And so, yeah, different ways to basically get to the layers of a computer chip. So basically a very large sanding thing, very small sanding thing, very fine sanding thing? Precise. Precise, yes. Very precise sanding machine, yes. But this sort of failure analysis is different than writing I2C drivers. Is one your hobby and one your career, or are they kind of overlapped? If I had to say, I would say that embedded development, so let's call it writing to I2C drivers, that's more my career. And this kind of chip stuff is a little bit closer to my hobby. And there's kind of a whole long thing about why that's the case. But the short answer,

I would say now, as I'm a consultant, maybe two-thirds of it is still embedded development, and maybe one-third of it is chip-level stuff, this kind of low-level analysis. So how did you get into the chip-level stuff? Oh, yes. What happened was, when I was going to college, I started learning to analyze firmware. So, you know, maybe get a binary on a Windows computer, wanted to learn a little bit more about it. And I learned a little bit of how to go to an assembly language and whatnot. But I also had this interest in robotics. And I started learning a little bit about the processor on these embedded systems. And I wanted to do that same analysis, but for those chips in that system. And I quickly learned that

unlike desktop systems, where you just had all the code more or less, these embedded processors had protections, and you couldn't actually get the firmware to look at. And that led me down this route of learning, okay, I know that you can't get the firmware out, according to all the official sources, but how can you? It has to be on there somewhere. And this led to kind of this rabbit hole of understanding, you know, you know, they talked about transistors and classes, but I didn't really know what a transistor was, you know, how do we get this, this code out, just trying to really truly understand how do these embedded systems work and how can I learn more about them? That was kind of the genesis of it all. When was uh at a company in around 2004 2005 we had a chip that had some sort of cryptographic

thing in it and as we were doing kind of okay how could people reverse engineer this to get our key or whatever um one of the people brought up well people could decap this chip and look at it and probably read out you know read out whatever's in the EEPROM or I don't remember the exact mechanism for storage, but it was protected. And back then it was, well, yeah, but nobody's going to do that. It would cost a million dollars. Was that wrong in 2004 or has stuff become so much easier in the last 15 years? You know, it really depends on a lot of factors. Maybe I could give you some examples. Even in 2004, you know, if you're talking about a security processor,

those tend to use older, less expensive processes. So, you know, just because, I don't know what the state of the art was in 2004. Let's just say 65 nanometer. I don't know if that's completely off or whatnot. Sounds in the ballpark. You know, you might find that the security processors, you know, might use, gosh, I don't know what it would be. Let's call it 300 nanometers or 250 or something. You know, something a lot larger because they're not making the state-of-the-art Intel CPUs. You know, they're

making more cost-constrained devices. And because of that, they're using the old technology that's no longer leading edge. And once you start thinking about that, and maybe another data point is even my optical microscope, without going into the scanning electron microscope, that has a resolving power of roughly 150 nanometers. And so you start thinking about, okay, that's the starting point. And then, you know, depending on the features you want to resolve, maybe the metal layers are actually a lot bigger than transistors. It becomes very plausible to start looking at these circuitry. Aside from that, when you start thinking about, you know, these million dollar attacks that you hear about in, I don't know, I want to call Hollywood, but wherever you, whenever you hear about that, I don't know if there's a movie about decapping chips. Most of the time, you should do this sort of very invasive fib work, you're rewriting circuitry,

as a very last resort. There is so much you can do just by voltage glitching a chip without opening it up. Electromagnetic glitching, EM glitching, is becoming a much more popular and very powerful attack, and those don't require decapping at all. Well, I think the first thing you should do is probably look for the serial port the engineers use for debugging. Oh, sure, sure. Yes, yes, yes. That's fair. That's fair. I'm sorry. I should have added that. Yes. You should first look to see, hey, maybe this isn't locked. Maybe there's a JTAG is open. Yeah, sure, sure. Fair enough.

First check if the door is unlocked before smashing through the windows. Exactly. Yes, yes. There should be a progression. I remember working on masked ROMs, and that makes sense to me. It makes sense that you can see the code in that way because it's truly different things happening. But if it's a flash on a modern chip, does it look any different when it's programmed? Can you really get the firmware out that way? And why does it look different? Technically, you can. There are some papers.

I would say his name, but I think I would butcher his name, so I won't do him the dishonor. But Othello has done some very good work showing how to directly extract the flash. As I understand it, it is possible to do that, but it is very, very difficult. So what you would probably see instead is you would understand the architecture of the chip a little bit, and you would trick it into reading out the flash rather than doing this sort of direct, you know, microscope readout of the flash. Ah, yes. Be careful with your bootloaders. Yes. And there's a lot you can do besides that. So let's say even your bootloader was 100% secure. If I was able to glitch the program counter on the chip, you know, to go somewhere else in your code,

maybe that would also unlock your chip. A number of chips are vulnerable to that because they, just the way that the bootloader unlocks the chip. So Chris was saying that at the time he was working on something where decapping was a possibility, it was very, very expensive. At least somebody was saying it was. I mean, I seem to remember it was expensive. But is it cheap now? I mean, if I had a chip,

how expensive would it be to... Set up a lab to do this? Not to set up a lab, but to go to somebody else's lab and ask them to do it. Well, I want to make a very clear distinction here. If you're talking about just decapping a chip, imaging out some optically visible wrong, that's a pretty straightforward process that, more or less, I can get done within an hour or two, if we're talking about something older. If you're talking about something that requires very high security and might require fib work,

I don't know about a million dollars if it's something relatively off the shelf. But it's not certainly out of the reach if you know what you're doing. The bigger problem is that the sort of people that know how to do this work, you know, it's not certainly out of the reach if you know what you're doing. The bigger problem is that the sort of people that know how to do this work, it's kind of a supply and demand thing. So there's a lot of personal relationships, I would say, in this field, you know, knowing who's good with what microcontrollers and kind of trying to figure out how to slot things in. At least that's kind of my impression of the industry. What is FIB? FIB is focused ion beam. What it's primarily intended for is if you've made your new shiny computer chip and you power it on after, you know, months of fabrication and all this stuff,

and it doesn't work and you, and you say, Oh no. And you want to know, what can we do without doing this whole process again of sending it out for wafer fabrication and getting the chips back and all that? And the solution to that is often that you use a very specialized instrument called a focused ion beam, and that can do two things. It can take material away from a chip very, very precisely. And it also can deposit material, such as new traces on the chip or new insulating material. So just like you might have bodge wires on a circuit board, this is the tool that will make bodge wires on your silicon. I had no idea that was even possible. Wow, that's really cool. How often does that happen? I mean, is this for really expensive chips or is this happening?

I mean, all chips are pretty expensive when you have to do them yourself. Sure. I would say that, I mean, I guess as a point of reference, I know at least two people that own Fibs in their garage. So this sort of tool is becoming relatively accessible even to hobbyists now. In terms of how often people make mistakes, I would say EDA industry, the electronic design automation that designs these computer chips, these days go through great lengths to try to prevent having a dead chip when you get it back. That said, clock and reset circuitry can be really tricky to get right, and I've heard lots of stories about people messing chips up and needing to do this sort of rework. So yeah, maybe not every chip that you get back, but often enough that I definitely hear about it. Okay, you mentioned the lapping machine, which was the layering. And that's different from decapping, which is not decapitation,

but decapsulation. That's correct, yes. Is that just taking the outer shell off, or is there more? I would say when I think of decapping a chip, there are one of two processes that people usually mean. The first, and I would say the most common, is taking the outer layers off to basically remove the epoxy packaging. You know, people call it like a P-dip, a plastic dip. In this case, plastic is actually epoxy and kind of a glass resin. Removing that, the lead frame, maybe the bond wires, and just ending up with a bare silicon die with a little bit of circuitry on top. I would say that's the primary thing that people

mean when they say decap. A related secondary thing, which I call live decap to distinguish it from this, is where maybe you don't take it out of the package entirely, but you remove just enough packaging to see the circuitry on the chip, And this would allow you to still use the chip, say on a circuit board or something like that, if you had to probe it maybe to test something out. What does it look like when it's decapped? Is it just a shiny metal coin, square coin? Yeah, I would say probably the most interesting thing is when you see a lot of pictures of chips online you see these sort of shiny iridescent you know images of a lot of these rainbow colors one of the things that i didn't realize until i had done this for a bit was those sort of images

are usually under relatively specialized lighting so if you're having maybe an older aluminum chip, yeah, it looks just kind of like a shiny, maybe silvery color, and then maybe a little bit of a black background on there. That's roughly how I would describe it. And then you put it under a microscope. Correct, yes. A high power microscope is enough to see some things, everything? Well, the analogy I would use is maybe the microscope that a lot of people are familiar with in the embedded world are, say, soldering microscopes. Which I don't like using time-zoom as a benchmark, but let's call those, I don't know, 50 times zoom, maybe something like that, 30 times. Once you start getting to these metallurgical microscopes that are able to look at things on a much finer detail, you start getting 200 times, 500 times zoom. So the amount

of detail that you see is considerably higher with those type of microscopes. And I would say those type of metallurgical microscopes are fairly capable of looking at chips up until, I don't know, maybe around year 2000, maybe mid-90s, something like that, depending on how leading edge of a chip you're looking at. Have you put other things in your microscope to look at them? Oh, for sure. I would say that I got a request recently, and I need to follow up. Someone gave me some image intensifier tubes, you know, which have these kind of very intricate fiber bundles.

And so maybe you're going to take a look at some stuff like that under there. I've looked at maybe insect parts. Minerals are a real fun thing to look at under microscopes, especially since I have a lot of polarization optics and stuff where you get some really fun effects. But primarily chips, you know, just because that's my interest, but certainly other things as well. The polarization, is that how you get different colors? I mean, when you have a chip and it's decapped and it's sort of silvery with a little bit of black, but then the pictures that I see, they're like red and green. Sure. Is that from the polarization or is that some other coloring method? There's two ways I could answer this. I would say the primary effect you are seeing is

thin film interference. If you've ever seen, you know, like bubbles, for example, right? You know, you get those kind of rainbow-y colors. It's that same effect, but under a microscope where you have this, they call passivation layer or field oxide, which are these thin layers of silicon dioxide between the metal layers. And depending on how exactly the chip was manufactured, the thickness between layers, that can cause different, you know, sort of beautiful rainbow colors depending on how exactly you illuminate the chip.

And that's primarily what you're seeing. See, I thought, I mean, most of the time, you know, like the ROM over here is sort of the same color. No, it's not color-coded, no. No, okay. Oh, well, if you've got a CAD program, I mean, here's kind of a funny bit of history there. You may see that polysilicon is represented in red.

And I believe the reason why that was was because in early chips, it tended to show red in microscope images. I don't believe that polysilicon is actually red in color. I think just somehow the way the manufacturing tended to work out with that layer height, it just happened to interfere in kind of a red color. Why do different parts of the chip look different? You mean like a kind of an overview image, like maybe they have some sort of regular

structure here versus there? Yes, exactly. you image like maybe they have some sort of regular structure here versus there yes exactly i would say one question i get a lot is how am i able to maybe look at something and say oh that's the rom versus the ram uh one way to think about that type of stuff is something like a ram tends to be say a six transistor arrangement and these older chips and that tends to be, say, a six-transistor arrangement in these older chips, and that tends to lead to these maybe called hourglass, you know, kind of where intertwined circuits tend to go in this relatively complex but regular pattern, versus when you look at something like these ROMs, where they're essentially isolated one-bit memory cells, tend to have a much simpler regular arrangement. So one way to think

about this is thinking about kind of what's the design entropy, you know, how complex is this design? And that at kind of a macro level can often ever gotten to see the CAD, the plan for the chip and then the actual chip and gotten to compare how the circuits look different? I have actually a little funny story about that. There was a chip I was working on, and I looked under the microscope, and I discovered that the text was backwards on the chip. And I found this really entertaining just because the way that they displayed it on their screen, it didn't quite translate to the way that they thought it did on the chip, but because mirroring a physical system basically got you the same

circuit, it didn't really matter at the end of the day. So yeah, I've seen this a little bit. I wouldn't call myself a chip designer. I've done a very, very small amount of it, more on the Verilog FPGA side than the computer chip, the ASIC side. So I'm not super involved with that, but yes, I've seen that a little bit. Why are chips so pretty? The rainbow interference certainly helps a lot. You know, personally, I also like symmetry. I think that, you know, symmetry can be very beautiful. A lot of chips have a lot of regularity, and to me, there's just kind of this beauty in engineering there where you've got all of these intricate designs you know it does something but at the same time

it's all very tidy and very lined up and symmetrical at least to me that's how i kind of see it it always kind of harkens back to tron to me citysca you know, that are weird and futuristic. Racing your motorcycle. They look like aerial photos of cities sometimes. They do often look like aerial photos of cities. Okay, so once I have my bottle of hydrofluoric acid, what do I do next? Don't drink it. Let's start there. So many things. Don't drink it. Don't do it.

It's dangerous. This should be done by trained professionals. But okay. Yes. And I would say that if you're getting your baby's first hydrofluoric acid, there are low concentrations you can start with. So let's assume you're starting with something relatively benign that you even can buy over the counter, here in the U.S. at least. So starting with something like this, what I would do for a typical project is, this is assuming I've already imaged a chip and want to get the basic high-level information, I would put it into a little beaker, typically made of something non-reactive like Teflon or maybe polypropylene, both high-quality plastics. And I would let that sit cold for, gosh, I don't know, if I'm using low concentration, maybe 30 minutes, something like that, maybe 15 minutes. And then I would wash that chip off with water and then clean it with IPA, so isopropyl alcohol, and blow that dry,

and then inspect it under a microscope. And at that point, I get a little bit of feedback about how quickly the chip is etching. Maybe at that point, for example, metal is just starting to get exposed. It no longer has that protective layer. And so as a next processing step, maybe if it's not exposed yet, I need to put in more acid. But if it is exposed, then I may use an etchant, like oversimplifying a bit, hydrochloric acid, and etch away all of the metal from the chip. And then that allows me to get another microscope image after that that maybe has the metal removed, but now I can see the polysilicon and the transistors below. And so by kind of repeating this process and taking a series of images, I can reconstruct all the layers of the chip. How many layers do chips have? Oh, geez. Even for older chips,

you know, kind of the first generation ones, You know, you think about maybe, you know, a couple different dopant masks, and then you've got, well, I guess related to that, the polysilicon. You've got contacts potentially between polysilicon and the diffusion layers. You have contacts between the metal layers. You have the metal layers themselves. You have the cutouts for the bond pads. So even on older chips, you might have, you know, I don't know, 10, 12 layers, I think maybe by the time you count it up. And certainly when you start looking at modern chips, because, you know, they do a lot, lot more of those. Gosh, I don't know the layer count on the chip that I worked on, which was maybe a 65 nanometer

chip, but I think it was in the ballpark of like 40, maybe, if I had to make a quick guess. So certainly if you get into higher performance chips, you know, it can get really up there. And most of these layers are planes, right? I mean, they have some things that go from plane to plane, kind of like vi is in a circuit board. Yes. But for the most part, you're doing 2D logic, and you're not trying to do 3D. Well, when you say you, I guess there's a few things. One. Well, I guess I can maybe answer this in one way, is my personal interest, I personally don't deal a lot with reconstructing the full circuitry on the chip. My personal interest tends to be more in extracting the firmware of the chip. Going back to our earlier conversation about jiggling the door before breaking down the wall

or knocking on the door, the most return on investment for this kind of chip decapping tends to be extracting, say, bootloader firmware or, you know, mask ROMs on chips. And so for the vast majority of my serious projects, that is the only layer that I care about, is spending time to figure out what layer is that on, and everything is 100% tuned to just target that one layer. And so in that case, it's only a very small 2D area that I'm really focused on. Okay. So you take a picture of the firmware and you found the right layer and you take a picture. And then what? How you go from that to ones and zeros, first of all, this image into more of an abstract computer representation.

Say going from a JPEG to something where you've got kind of a 2D matrix of all the bits that you saw in that image. And then once you've got that, which you can do either computer vision or you can just kind of manually type out like, oh, I saw a bright spot here. I saw a dark spot there. Maybe that's a one versus a zero. But once you have this 2D representation, like this matrix of bits, it's not really, like you said, it's not really an object file. So there's a little bit of an art then from going from that bit matrix, let's call it, into a usable like.bin or.elf or whatever you're looking for. And I would say my favorite strategy for doing that is I know a number of common memory layout techniques, and I typically have some idea of what the architecture is that I'm looking for. So say, for example, if it was

an 8051, maybe it's very likely that there is a interrupt jump table at the start of the firmware, and the very first byte is probably 02 for a long jump, or possibly 01 for maybe a short jump. And I will then look for that pattern in this kind of matrix, kind of thinking about what I know are common memory layouts. If that doesn't work, then maybe I'll start looking at some very, very minor parts of the circuitry on the chip related to the address decoders to give some hints. And typically that information is enough to kind of turn that into an object file. Typing out ones and zeros. Sure, yeah. That seems like a terrible waste of an afternoon.

Right. And because of that, there are several programs out there to do that automatically for you. There is RomPar by Adam Laurie, which is the tool that I primarily use, and I guess I'm also the maintainer for these days. And there is also BitTracked by Chris Gerlinski. And I would say that you should definitely start by one of those. And if you have very clean microscope images, you should be able to do that automatically in short order. However, a lot of times, you know, there's maybe dust on a microscope image or something like that, and it tends to mess up these computer vision algorithms. And I would say because of that, typically there's some amount of post-processing involved. But if it's a very small ROM, like maybe you just need 256 bits, sometimes there's very small ones like that, it may be quicker just to kind of sit there for literally two minutes and just go and type it out.

256 bits, I could do that, yeah. Yeah. Is this hacking in a bad way? I mean, you said some companies, they need it. I get that. But when you're doing it in your garage just to look for fun, is it wrong? I would say that a lot of the projects that I post are purely for educational, nostalgic purposes. If you look, you'll notice that I actually mostly post information about older chips. And one of the reasons why that is, is I feel that posting chips that are 20 years old or more, there's really not as much invested in them, or irrelevancy, I should say. And certainly from

a legal perspective, if you look at mask right in the U.S., you know, just like we have copyright, we also have mask right. That expires at the 10-year mark. So we're well beyond the legal, you know, kind of high level. I should say I'm not a lawyer, but this is my rough interpretation. You know, there's obviously still a lot of patents and stuff, but in any case, just from the educational perspective of just kind of looking, trying to understand your favorite computer from your childhood, how did the 6502 work and that or something like that, I really haven't seen a lot of friction against hobbyists studying these sort of projects. Maybe if you posted some information about how does the latest security processor work, you know, in some, you know, console, I think you're going to get a little bit more flack for that. So I tend to stay pretty clear away from those. But at least for the projects that I've worked on so far, I haven't had any problems. It's funny that people think it'd be

bad to look at something. It's breaking my brain at the moment because, yeah, anyway, I'm very surprised that the copyright is only 10 years. So it's mask right. And part of the reason why is let's say, I just want to say I'm not a lawyer, so don't take any of this too seriously. Let's say that you had mask right on a chip for 10 years, but you also had a patent on, I don't know, so let's say the floating point methodology on that chip. Even though I could theoretically copy your mask in 10 years, that would still violate a patent that you held. Gotcha. So it still wouldn't be commercially viable. I think that's part of the reason for that is there's still a lot of core IP that's being protected by other legal mechanisms. How do you decide what project to do next? Sure. I would say there are several mechanisms for that. Certainly, you know, personal interests

drive things. But a lot of it is I would like to experiment with some new technique. For example, one of my side projects right now is trying to get a plasma etcher up and running. If a project came in which I thought would be a good match for that plasma etcher, I might select that project just because it would be more interesting than for me to just decap an image, another chip, which I've done, gosh, I don't know, a thousand times at this point, and you know, it's just not as exciting. So a lot of the selection is based on,'t know, a thousand times at this point. And it's just not as exciting. So a lot of the selection is based on what's going to challenge me a bit and get me some new technique to try. What does a plasma etcher do? A plasma etcher is a more modern way to basically create ICs. And the way that they do that is by basically taking, say, fluorine atoms

and launching them at an IC. And the really nice property this has versus using hydrofluoric acid to etch a chip, which is what people did traditionally, oversimplifying a little bit, is that this is directional. And there's these words, they're like anisotropic or something like that, but I always pronounce them wrong, so I'm not going to use them. The idea being that if you use hydrofluoric acid, for example, it may under-etch a circuit that you're trying to save, say like a polysilicon gate that you want to look at maybe under a microscope. If you used hydrofluoric acid, it would go underneath the polysilicon and the polysilicon would eventually float off. But if you use a plasma etcher, it shoots fluorine atoms at the polysilicon. The polysilicon blocks those

fluorine atoms. It doesn't really react too much with the polysilicon, but all of the silicon dioxide around the polysilicon gets etched away. And so you get left with this very clean, sharp polysilicon, which gives you great transistor images, where otherwise you have to be very careful doing that with traditional chemical methods. I'm lost in thought at Christopher telling me that when you have fluoride in your toothpaste or mouthwash, what it actually does is replace some ion in your mouth with fluoride ions. It changes a mineral from one kind that your body produces to something else that's stronger. Oh, interesting. Sorry.

But, I mean, that kind of, I don't know, it made me think of that. So what kind of project would require a plasma etcher? And would it still be on these older chips? Yeah, there's a lot of reasons why I might do it. One example was a traditional problem for me is I would like to very clearly image a contact ROM on an old chip. There are many ways that you can encode data into a chip. Maybe you do it by either creating transistors or not creating transistors. Another way to do it is if you have metal layers, you can choose to put vias essentially between the layers, and that encodes whether something's a 1 or a 0. Those vias tend

to be very large, so in theory you could use an optical microscope to see them quite easily. The problem is the surface of these chips, that field oxide, the silicon dioxide, can sometimes be very uneven. And because it's optically clear, it also can serve as a lens and actually distort the image of the contacts below. One of the ways to correct that is to use a plasma etcher to actually remove that silicon dioxide. And in theory, those ions will etch the silicon dioxide a lot quicker than they will the metal, and that could give me a very clean contact image, which otherwise would be hard to get with my traditional microscope setup. So those are kind of the sort of projects I'd be looking for,

but the main property is it's just a lot more even than a lot of this more traditional acid etching, so it should allow me to get more modern chips that are a little bit out of what I can currently process. When you do process a chip now with the hydrofluoric acid method, do you need more than one of the chips or do you usually get it on the first try? I would say if it's an older chip that's maybe one to two layers, the current strategy is I will take a very high resolution image of what I can see. And that typically will show you the first two metal layers just due to the way these chips were manufactured.

And by the time I strip away the metal and I'm left with just the transistor layer below, that's typically enough information that if you wanted to get the whole chip information, you could. The one bit that has traditionally been very challenging is, for a time, a lot of chips used something called an implant ROM. And the important thing to note about this is, these bits were not visible under a microscope image without doing special processing. I went through a lot of work to try to understand how to successfully extract those bits the first time out of a chip. I would say that's something I'm a lot better at than I used to be, but it's still a little bit of a tricky process. If a chip doesn't have that special implant layer, I can typically get it out. If it does have that implant layer, it's still a little bit hit or miss. Okay. I'm going to switch gears a

little bit because I have listener questions I want to get to, but first I want to talk about conferences. You gave a talk recently at Hardware.io called Taming Hydrofluoric Acid to Extract Firmware. I assume that's pretty much what's in the tin? Oh yeah, so the talk was about the sort of process where I do to de-layer the chips involves a lot of chemicals, especially on more modern chips. And this machine basically helps to apply just the right chemicals at the right times to get a higher quality images as I'm taking apart a computer chip with a lot less effort. Okay. You gave a talk last year at Hydro Hardware as well. The previous talk was about post-processing the microscope images into usable firmware, whereas the recent talk was about how to generate high-quality microscope images. It's another way to think about it. Do you have any conferences you're planning on going to soon?

I don't currently have any on the docket. I've been pretty busy with work, but it's probably a good time to start thinking about that for the future. How did you start going to the hardware ones? Well, hardware IO specifically happens to be nearby me. So that made it just very accessible. You know, I have this group Mountain View Reverse Engineering. I try to foster kind of a local hardware and reverse engineering community. That was a very easy sell, you know, hearing that there was a hardware reverse engineering conference nearby. I was happy to try to do what I can to support that. That's what kind of started me speaking a little bit more at conferences. Aside from that, I've kind of

presented some things at Maker Faire, although those were a little crazier projects. I wouldn't say that traditionally I have spoken a lot at conferences, in part because a lot of my work is somewhat sensitive, and I have to be a little bit careful about what I say. As I've started to do a little more freelancing, it's been beneficial for me to kind of network with people more. And that's kind of given me a little bit more incentive to be more active in the conference community. And that's kind of what has changed that recently. That makes sense. When I first started consulting, I did a lot more conference stuff. So I totally, totally understand that um the since since i did connect with you from the hardware io folks uh i feel like i should say that they have an online training in january uh january 27th to 30th of 2021 um sooner than that there's the Open Source Firmware Conference, December 1st through the 3rd, and the IoT Online Conference put on by what looks like UBM, but I don't think it's them, but it seems like most of the same people.

That's December 8th and 9th, and I'll put all those in the show notes, as well as the Hardware.io and the archives to the Hardware.io, which had a ton of talks, including yours. Oh, thank you. Well, I'll have to check those out. Okay, so now some listener questions. First, I think I have to go back to the lightning round. Civilian tanks? Where do you get a civilian tank? And why would you want to drive it on the roads that would require you to have a driver's license? So, first of all, you must have done some real digging to find that. I think I posted a picture or something, I don't know, a long time ago on my Twitter.

Oh man, this could be a whole podcast episode in itself. But I'll give you the plug for it, and I'll probably get some questions about this. I basically joined a kind of a startup incubator, hacker house sort of thing that was on the property of the former, part of the property of the former Military Vehicle Technology Foundation, which people called kind of the Tank Museum in Palo Alto. And as part of that, there was a military vehicle that got more or less abandoned on the property that the landlord had. And so we kind of drive around. So one of the perks of living there with other people is sometimes we would drive that around.

And I would say one of my funniest memories from that is I think I got a noise complaint for driving a tank late at night, which was kind of funny. Do your neighbors know what's in your garage? You know, I would say that of all the places I've lived, no one has ever cared. It's kind of one of those funny things where people always think, you know, just because you have all these weird things in your garage that people are going to be really nosy. I used to live in Troy, New York with Andrew Zonenberg. And I remember even, the sketch is so sketchy. We would be on the sidewalk, so we didn't really have a proper lab

at the time. Just cooking chips and lab coats on the sidewalk with a hot plate and lights out there, work lights. People would walk by, police cars would go by. No one ever asked us any questions, despite how strange that was of a thing to do. I would say that's just been my experience. When I was in Mountain View, the landlord would come into the garage and he would complain about the cardboard on the side of the house. I guess, I don't know why people, maybe you would think that people would ask these questions, but no one ever seems to. It just has never been the issue that you might perceive it would be.

Maybe you need more bakers that are filled with weirdly colored liquids that light up. Yeah, it doesn't look Hollywood enough. Okay, I'll get on that. Maybe a Jacob's Ladder. Plasma ball. That kind of stuff. Yeah. Have you ever had a lab accident in your house?

I would say the most popular article that I've ever written was an article titled Top Lab Accidents and Explosions. And where I go through some of those. That's a podcast. Yes. and explosions. And where I go through some of those, I would say, yes, and there are more since that came out. I've only ever had one that I would say had serious consequences. You know, I certainly have tons of scars on my hands. I usually wear a fair bit of protective gear, especially since I got any reasonable budget, you know budget to do these type of things. And although I've been caught in a number of explosions, a good example was I was making lead bricks to do gamma spectroscopy. And one of the things you have to do if you've ever done lead casting, say people do this for bullets a lot, you see a lot of information on this online, is you have to be very careful never ever to get water in your old lead as you're throwing it into

the pot. And because I was cooling down these bricks, you know, to kind of keep the molds going quicker, something happened where some water got in some lead. And I was wearing very heavy protective gear, but there was this 20-pound pot of molten lead. And I remember I threw some lead into this pot and there was this massive lead explosion. And even some of these gray boxes I have to store materials these days still have lead embedded in the side of them from this explosion. Now, I happen to have been wearing extremely heavy gloves and jacket and all this stuff. So I got sprayed with basically molten lead. But because I was wearing so much protective gear, I didn't get any injuries at all. And I think that's a good lesson for people. It's a life philosophy of mine. I guess you could say,

if you're going to do something that might be a little bit dangerous, you know, just kind of quantify the risk. It doesn't mean you can't do it. Just be very careful and make sure you have a backup plan in case you make a mistake. Always have an exit plan. Yeah. So your lab is basically cinnamon and lead. Cinnamon and lead. Well, I don't do as much of the radiation stuff as I used to,

but certainly I have a little bit of that. But, you know, solder, you know, whatnot. I would say lead is not too unfamiliar to a lot of people in the embedded space. Certainly a number of chemicals. I would also say I have a lot of robotic stuff. I haven't talked about it a lot, but I think some people are aware that I got some bomb disposal robots,

and that's been kind of one of my recent hobby projects is kind of driving those around. Similarly, I find it really funny being in Silicon Valley where we have so many robots. I've driven those around a little bit, and no one has asked any questions about them, which I thought that some people were going to give it a weird look or something,

but I guess not here. Maybe you should have the bomb disposal robots do the lead pouring. I was just thinking that. I've thought about that. Definitely needs more exploration. Some of those questions are for Rick, but now I have some from Azmita, who recently saw your post about the Nintendo S-PPU1SNES picture processing unit. Yes, okay. What is that, and can you tell us about it? Basically there was some community interest to get some very high resolution pictures of this a while back. And someone very generously collected some

funds and said, Hey, John, you know, if we give you this money, because you know, these are very large chips, uh, and we need a lot of images, you know-layer them. Would you be willing to put in the time to collect these images and post them so that people can start looking through these Nintendo, they're basically graphics cards, is kind of a way to think about it. This is the graphics engine of the SNES and the Super Nintendo Entertainment System.

And so a while back, I was allocated some funds. I used a good portion of those funds to buy basically a very high-power optic. It's called an oil immersion lens, and this produces very high-quality microscope images. And that's kind of a partially completed project now, where I have taken the top metal image, that is almost the chip as designed, which shows the kind of like the circuit board traces of the chip. And I then posted some follow-up images where I used some hydrofluoric acid, took off a little bit of the chip, took a high-resolution picture of the chip, and then repeated that process a few times. And by doing that, you know, kind of got a

layer stack up of the chip showing all the different parts. And now with all that image data out there, some people are now actively working to try to understand the inner workings of this chip. And fortunately, there have been some related projects using similar designs that they're able to leverage. And I think the community is already moving to understand some things out of that shell. Okay. That was a new microscope. How many microscopes do you have? Well, that wasn't a new microscope per se. It was a new optic on an existing microscope. Okay. And yet the question stands, how many microscopes do you have? Well, okay, if I had to, let's count them off. Now, mind you, there's only two microscopes that I use heavily. The two microscopes that I use very heavily are my main metallurgical microscope

and my soldering inspection microscope. So those are definitely the two favorites. Aside from that, I also have a laser probe station. I also have another metallurgical microscope, which was a Craigslist impulse buy. It was like $300, and for that caliber microscope, I was like, okay, I can't resist $300 microscope. Another one is I have kind of an infrared microscope. I also have a scanning electron microscope and i have a confocal microscope i think that would be the list what's a confocal that would be super cool yeah confocal microscope the high level idea is you eliminate out-of-focus artifacts in the background of an image. The idea being that instead of looking at the out-of-focus parts of an image,

just get the very crisp in focus. And extrapolating this a bit, what you can even do is you can get it so that different focal planes of an image are encoded in different colors. And so the end result is you tend to get these very high contrast images that show layers in different colors and at very high resolution. And they're very useful, for example, if you wanted high contrast optical images to reverse engineer a chip. It allows you to do that much easier than a conventional metallurgical microscope

would let you do. Is that like that camera that Phil and Rob worked on? Not really. What's the name of the camera? Lytro. Oh, Lytro. Oh, yeah. Lytro. The way that mine works, maybe this will give you a little bit more idea. Mine is called a, I think they call it a Nipov disk or something like that. The basic idea is it's almost like call it a Nipov disk or something like that. The basic idea is it's almost like you have a pinhole where, you know, if you had some light coming out of that pinhole, it focuses on an object. And if it is in focus, it will come back through the pinhole. But if it's out of focus, it'll miss that pinhole. And basically you have one of these pinholes for every pixel. And the way that they do that is by putting a bunch of pinholes on a disc and then spinning that disc very quickly.

Yeah, we had one of those at Avenger and we used it for looking at like biological samples and stuff to look at various layers and tissue because it almost was like a thing where you could scan through and look at different layers, especially something that's translucent. And then yours would even, if I had to guess, the biological ones tend to be laser-based rather than disk-based. Yeah. Similar concept, different implementation. Okay. That covers the listener question. So I want to go back to one other thing. You have a huge wiki-based website that tells people how to do all this. Why? I mean, why did you do that? It's kind of the Linus Torvalds approach to something. I would say that what the wiki really is, is it's me working on a project and then posting my notes of what I did so that when someone asks, how did you do that? I just kind of share the notes of what I learned from the last time I did on that.

And I use that then to answer emails, you know, by pointing people to a page. And I also encourage collaboration from others that if they're working on similar things, they can also share their experiences on there so that it will save me time next time I need to work on something. It seemed like a good portion of that site got turned into a college course. Yeah. And this goes back to, I mentioned Andrew Zonenberg, who I was cooking chips on the sidewalk with at RPI. He stayed at RPI a bit longer than me. And we collaborated a lot, especially at that time on our projects, you know, because we had a kind of a shared lab space up in New York by Albany. And he eventually got permission to teach a course, basically, you know, alongside a professor.

And that was kind of a core interest of his. So yeah, he took a lot of our shared experiences, you know, projects we had worked on, and used that to create a course over there. I was really impressed by both the website and the course because it just laid things out so beautifully and it was all there. It wasn't like I needed to watch videos or anything. It was the slides were nice. I'm so bad at watching videos. It's funny you mentioned that because a comment that I've repeatedly got is, why do you write things as text? Why don't you make more videos? Because text is searchable. There's a lot of reasons. But at the end of the

day, yeah, my preferred medium for communicating technical information today is text and pictures. Well, that's what I prefer. Although I know people prefer podcasts and i know people prefer videos podcasts are terrible everybody should not listen to them no i mean podcasts though is i don't know if you would you would go to a podcast maybe you know to learn about you know some car you're very passionate i don't know if i would go to a podcast to learn how to like change the oil pan on that car you know i feel like it's a different mindset some visual things that are required yeah uh so i had one question um before we wrap up uh and you talked about the processes used to to examine these chips

and that some of them are quite quite a bit easier with larger feature sizes and older older parts where do you see this going in like a decade when the older parts are now 14 nanometer and 10 nanometer and things like that are you going to be able to step up your techniques to to be able to probe those or is there some wall eventually? Oh, for sure. As I mentioned, you know, the trend is the failure analysis equipment has to keep up so that, you know, when Intel makes these new parts that they can, you know, actually debug them when they have problems. And then over time, that failure analysis equipment trickles down, you know,tier fabs, and then to corporations, and then finally to the hobbyist market. I know two people today that have these focused ion beams

basically in their garage, these very high-quality instruments. And I expect to just see higher quality microscopes in people's garages, better equipment. I think that it will require, you know, a little more involvement maybe than we have today, but it's not going to be out of reach because more and more of this equipment is going to filter down. You started your career with computer science, computer engineering, and you've gone long embedded in hardware and deep into the chips. I mean, I've done a lot of embedded and I've never gone deep into the chips like this. If somebody wanted to do similar things with their career, do you have any advice? Well, my biggest piece of advice is always, you know, follow your passions. And at least for me, the way that I've structured everything is finding these passion projects and with kind of a goal in mind, you know, pursue that.

I would say, you know, if you wanted to learn, for example, about chip security and there was a current chip that you wanted to learn about, maybe instead of decapping it, maybe start with something like fault injection because that's going to apply to kind of your passion. It's going to teach you a lot about how these chips work and it's going to be a lot more approachable. But that sounds like very good advice. Do you have any thoughts you'd like to leave us with? I wouldn't say any thoughts at this time, but you know, thank you very much for having me on the podcast. Our guest has been John McMaster, Embedded Engineer and President of McMaster Consulting. Thanks, John. This has been really interesting. All right. Thank you. Good chatting. Thank you to Christopher for producing and co-hosting.

Thank you to Sparsh from Hardware.io for pointing me in the direction of John. Thank you to Rick and Asmita for questions and to our Patreon supporters for his mic, which arrived DOA, but that's not part of it. It's not your fault, Patrice. It's not your fault, Patrice. No. I will try to do a postmortem on it. I'll let you know. You can always contact us at show at embedded.fm or hit the contact link on embedded.fm.

And now a thought to leave you with. You look like you don't have one. I don't actually have one. Okay, bye everyone. Always buckle your seatbelt.