Embedded - 332: There Were Fires

Episode Date: May 29, 2020

Doug Harriman of Simplexity (@SimplexityPD) spoke with us about motors, controllers, and designing mechatronic systems. Simplexity (or if you want to contact them) Doug recommends Control Systems Engineering by Norman S. Nise. Elecia recommends Notes on Diffy Qs by Jiří Lebl from American Institute of Mathematics list of free and approved math textbooks. They both like the 3 Brown 1 Blue YouTube channel. If you liked the part about how to choose a motor, you might want to watch Doug’s Webinar on DC Motors & Motion Control Systems (you’ll have to give your info to see it).

Welcome to Embedded. I am Alicia White. I'm here with Christopher White, and our guest this week is Doug Harriman of Simplexity. We're going to talk about mechatronics. Hi, Doug. I'm interested to hear what mechatronics is. Morning. Can you tell us about yourself? Sure. I am currently the CTO of Simplexity Product Development, where I lead our EE firmware, software, and systems engineering teams. My background is in mechanical engineering and mechatronics. I spent 14 years at HP, right out of grad school. Started out doing mechanical design, but then spent a

dozen years doing motion system development on inkjet printers. After that, I spent a couple years at a wind turbine startup working on a 10 kilowatt wind turbine, and then moved over to Simplexity product development, where the bulk of my time is split between managing those groups, supporting sales and marketing, and doing systems, mostly motion systems development. All right. I know you listen to the show, so you're familiar with lightning rounds. Are you ready? I guess I am. Favorite fictional robot? I'm going to say, I don't know if it's actually the robot, but the AT-AT walker from Empire just captivated me as a kid. Favorite non-fictional robot?

I think some of the intuitive surgical robots are pretty darn amazing. How many individual robots combine to make Voltron? I have no idea. It depends which Voltron. Oh, see, I should't have written that question. I don't even know the answer now. Favorite Fourier transform. Favorite Fourier transform.

Hmm, I'm not sure. Doesn't everybody have favorites? I don't understand the question. I don't either. Is there just one? The fast? Fast or continuous? Like the Dirac pulse.

Oh, favorite function to Fourier transform. I mean, that one's totally the best. Although sine and cosine is pretty, so. This question has gone wrong. I'm sorry. It went south quick. Favorite Lego component to use in mechatronic design? It's got to be the motors.

Oh. But which ones? I mean, there are different brands. Brands? Yeah. I'm going to go with the Lego brands. I only have two, a small one and a large one. I like the large one better.

The gray ones with the black shafts and the, yeah. All right. And do you have a tip everyone should know uh i think it is write it down i've learned a lot of my job is communications and writing it down is so critical okay so what does mechatronics mean that's a that's a tough question i get that a lot so i think about it in terms of the system a mechatronic system And it's a system that has elements of mechanical elements, electrical, a control system, and that control system is almost always implemented in some sort of software or firmware. So it's really got to have all of those elements to qualify as a mechatronic system.

How is it different from robotics? So I think a robot is a mechatronic system. It's it different from robotics? So I think a robot is a mechatronic system. It's one of those things that are necessary and sufficient. All robots are mechatronic systems, but not all mechatronic systems are robots. So I worked on a project last year that had a little door controlled by a motor and a sensor to make sure it was shut. I don't think anyone would call that a robot, but it's clearly a mechatronic system. Oh, okay. So by this definition, I've worked on mechatronic systems. How exciting. Me too. Some of the toys I worked on were mechatronics. Absolutely. Toys are great mechatronic systems.

So you've seen a lot of different motion systems. What are the common pitfalls? I think the most common pitfall is people get really focused in on the area where they've got the most expertise. So I've seen systems really heavily over-designed because they were led by the EE, and the EE really focused on the motor driver. Or the mechanical engineer was really focused in on their piece of the problem and it's uh the failure to kind of step back and look at look at the fact that all of the different disciplines play in and have to interplay with each other well i'm i'm laughing here because i i have such a great example of this i worked on a system the system i'm thinking of that was mechatronic which was a laser delivery system for medical, it was a dermatology kind of thing. But it was totally driven by the optical engineers who had this idea of how to focus lasers and change the focus.

And so the optics engineers were so excited about how to deliver the laser pulse energy in a particular way that everything beyond that had to be this horribly complicated mechanical and software system to have to design this little tiny telescope to focus. And it's exactly what you were saying. It was like, well, we got to do this and it's great and it's exciting. And everything that followed on was just a disaster because it was so hard to do. Yeah, it's really common to see that. And it's usually driven by either kind of who's the most senior engineer on the program with the most clout or maybe within a company, what group has the most clout. So I have some people that worked at a laser company. Yeah, the physicists kind of ran the show. And so

everything revolved around how the physicists wanted to do things. You come from a mechanical engineering background. I do. Yeah, it's kind of odd. I've got my bachelor's and master's are both in mechanical engineering. My master's was in control systems within mechanical engineering. So, of course, it makes sense that I lead the EE firmware software group at my company. But basically what happened was control systems these days are only implemented in code. Way back when, they would be implemented with electrical components, but I've never seen that done personally in practice.

So I had to get up to speed in firmware to be able to implement my design. So I was lucky enough while I was at HP that they paid for me to take some remote CS classes and computer engineering classes through Stanford so that I could get up to speed. That kind of is what led me into firmware and embedded systems. If you could do it all over again, would you do software first or would you, is coming at it from a mechanical background, the most useful for you? I think mechanical or electrical is really critical. You really have to understand kind of the engineering and the physical components behind that.

I've seen people that are pure CS have a hard time kind of understanding the variability of the real world, of the mechanical systems. In fact, when I was getting ready to go into college, as I was touring different colleges and talking to different professors, I really wanted to be in robotics. And I'm kind of surprised, but they all said, well, you should go into mechanical engineering as the background. So I think that was good. I do wish I would have taken a lot more programming when I was in school. You mentioned control systems. And this, I think, is part of the CS versus systems engineering. Could you define what a control system is and how they're commonly implemented?

So, specifically around control systems, like what I studied in grad school, it was all about the feedback control laws. And so the math behind both how to model the system you want to control so that you get it into a form that you can then do the development of the control system on. But it's really the equations about how you read some sensor data and you determine a proper input back into the system to control it, to get it to do what you want to do. So typically the way that looks today is you're reading some sort of sensor, maybe through an A to D, but usually most sensors are digital these days. So you're reading over a spy bus or I squared C bus. And then you do some math

looking at the sensor value read versus what you want it to be and then you you make an adjustment to a control output which is could be a DAC but is again typically a PWM signal these days and so basically you're reading a sensor comparing it to what what you want that sensor reading to be and then applying a control output okay and so we've heard about PID loops. That's one way to do it. But there are a whole bunch. I mean, I could just read the sensor and subtract and just send what I want. I don't need it to be formal. It can just be, oh, I'm not to the position I want to go push on for a little while longer. Yeah, yeah, absolutely.

So most people have heard about PID because that's the one class that tends to be taught in undergrad. So double E's and M's will get and chemical engineers, too. They'll all get some introduction to control. And PID is the topic that's talked about. And PID is nice because it's fairly intuitive. You can get an idea about how to tune it. And there's some good rules of thumb out there, like Zeiger-Nichols tuning. So they can give you some basic steps to go through to get a system tuned. So if I had to guess, I'd say roughly 80% of the control loops implemented in the world are probably PID or some variant thereof.

PD. We're going for PD here. That's just PID with the I equals zero, right? I know. It's just PID or some variant thereof. PD. We're going for PD here. That's just PID with the I equals zero, right? I know. It's just the stability. So, kind of the example you were getting to earlier was maybe it's just a proportional. So, you look at the error and you multiply it by something. And sometimes that's just a unit conversion from whatever your sensor units are to kind of whatever your output units are. So, you know, a millivolt in and you

want volts out kind of thing. That would just be a proportional gain. But so, it's clearly, yeah, P, I, and D, any of which can be zero, and that leaves you with fewer letters. That said, I didn't spend two years in grad school talking about PID. It was actually not mentioned at all. So, there are more advanced control algorithms. There's a large field of linear control systems starting with state space control, and that's all based off of a system model. If you remember your controls class, you probably talked about root locus and how you change the gains and the poles of the system would follow some specific trajectories in space. And the really cool thing about state space is you get to put the poles wherever you want them.

So it's really easy to find a control that gets things stable. And then the field kind of delves into how do you properly pick those to optimize for different things or to minimize the effects of noise on your system or maybe some disturbances that come into your system. And then past linear control, there's a whole large field of nonlinear control that can deal with some really nasty problems. Highly nonlinear systems, hard nonlinears like friction and backlash and ways to address that, those sorts of things. These are very, very math heavy.

I think of them as Fourier because we did so many Bode plots. But what kinds of math are critical? So, I mean, it's all based on dynamics and dynamics, the study of change, which gets you straight into differential equations. So everything in controls is going to start with some sort of differential equation. So linear control systems are going to be based on linear differential equations and nonlinear is going to have nonlinear differential equations. So that's the core fundamentals. And everything can be really broken down into first and second order differential equations. So that's the core fundamentals. And everything can be really broken down into first and second order differential equations. So if you can develop a gut feel for those, you're in good shape for at least understanding what

the system is doing and maybe what it should be doing. Beyond that, you get into systems of differential equations, and that's where linear algebra comes in big. Okay, so we're talking differential equations, linear algebra, the whole subset of Fourier stuff, discrete and continuous. I just want to make a robot. Do I need to know all this stuff? You know, it's one of those things where the more you know, the more maybe you realize you don't know. So, no, you don't have to know everything to get basic performance.

I think it's like any area of engineering. The more you want to dial things in, the more you want to reduce cost and increase performance and push things to their limits. That's where you need to know more and more. So, it's definitely helpful. It gives you some intuition around where things can go. But there's lots of other ways to address things. And, you know, your gut feel on things like just let's turn the motor on and when I get close, let's just turn it off. That's a very simple control system. People will call that bang-bang control. It has been used for real problems. That's a technique that we discussed in grad school for satellite attitude controls.

They would actually just turn a rocket motor on and then turn it back off quickly to adjust satellites. So it is tried and true and it's something that you can just play with really easily. How do I figure out that that's not enough? I mean, so I try it, and I can't get it to work. What do I do next to figure out what scale of problem I have? I mean, do I just stick in a PID loop or do I then start studying differential equations? So I'd say a PID loop is probably a good place to start. And you can get a lot out of a PID loop. So like I said, there's some good techniques around how to tune those,

but there's also some tips and tricks for things that you might want to do. So one of the things I like to do is throw in a feed-forward control. So feedback is really good for dealing with uncertainty in the system. Parameters are never exactly what you think they are. You may get some disturbances that come in. Someone may bump your robot, bump your system, and it needs to compensate for that. But feed forward really lets you take care of the things that you do know pretty simply and can save you some time in your tuning. So with feed forward, you just say, well, I know something about the system. I know there's this much friction in it. I know that the inertia of this component is whatever it is, and I'm trying to accelerate. So I know I have to apply at least

that much force or torque to get it to move. And doing that, you can get some really basic motion profiles working pretty well. You'll quickly see that there is variation in the system, and it's not doing exactly what you want it to do. And that's where you can turn on the feedback, and you'll get some, you don't have to be as aggressive, or maybe you could be more aggressive with your feedback control, because you're no longer relying on the error response to drive everything. You can actually just look at what you wanted to do and use that as an input to get things going kind of how you want. Another thing to be really careful about is the inputs you give the system. So in school, they talk a lot about step response. And so a lot of people just try – if I want to go from A to B, I'm just going to give it a step input.

I want to – give me position B, and then the system doesn't really perform like they want it to. But you can kind of think of a control system now, the controller plus the system. That kind of looks like a big filter from input to output. And so if you shape your input appropriately you can kind of get the filter to do what you want so shaping your input is actually really important and for a lot of systems a lot of systems are position controllers you're again trying to go from a to b think about what you would like the velocity profile to look like really typical it's kind of a constant acceleration a constant velocity phase and then a constant deceleration to get

you into place. And by not whacking the system too hard, but giving it a little more of a gentle input, you can usually get a lot better response. So I want to go back to the terms feedforward and feedback, because that's probably something that people are like, what? To feedforward, let's say we were building a toy caterpillar and we wanted to go one foot. Now, we should know in the lab, on our desk, that to go one foot should require us to do a PWM at 10%, 20%, 30%, 40%, ramping up until we're on for 12 seconds and then ramping down until we're off. We don't have to look at our encoder. We don't have to look at any position information.

That should get us about a foot. Now, if you want to actually go a foot, you need feedback. You need something to tell you how far you've actually gone. And that's often an encoder that will tell you how many times the motor has rotated or something along those lines. But it could be, you know, GPS. Not for a foot. GPS, not Fairfoot. So feedback means you're using the sensor to tell you if you successfully did your feed forward. And I think a lot of people forget that you can do feed forward and you can get good results. And then your feedback is only affecting your feed forward plan. It's not controlling the whole system. It's augmenting something you thought would work to start with.

That's what I was trying to say. Yeah, exactly. And in fact, in the lab, it's really easy to do the first experiments you talked about, right? Turn the system on at different PWM levels. See what it does. That gives you a really quick measurement of the friction in your system. And in fact, that's a really good characterization technique to look at unit-to-unit variation. So that's something that's very common that you'll do. And again, PWM is great because it doesn't measure just the friction, right? Your power supply has a voltage variation. There's variation

in your motors. They are not perfect sources of torque. They have different resistances and different torque constants. And so you will start to measure your overall system variation. So doing that experiment, A, like you said, gives you some good idea what the feedforward needs to be. And B shows you what kind of variation you have in your system that you're going to have to deal with in feedback. There's kind of halfway solutions too, where you might, and I've heard the term open loop to describe things that aren't, don't have feedback to, um, where you have a system that's in an unknown state when you started up, but you, you have a procedure that maybe homes a motor. So, you know, where its initial position is. And then from then on, you can operate without

feedback with some confidence, but there has to be some step to say, oh, the caterpillar's on the starting line. Yeah, that's very common with stepper motor systems. So, stepper motors, they're great because you give them a known input and they should have a known output. And I say should because the stepper motor itself will have that response. You give it a step command, it's going to step some number of degrees. But that's the motor itself. A motor itself is pretty useless. You have to attach it to something, and now you have a system.

And if the system has too much load torque and you apply that step, it may not do anything. So a step remoter system has to be designed carefully so that you know you have enough torque to cause the motion. But then, yes, you can operate in a fully open loop system that way. But your point earlier about figuring out where things are at homing, we often call it dirty power up. That's the term we use at HP. So if somebody yanks the plug in the middle of an operation, you don't know where you are. And it's really critical in any of these mechatronic systems to have the ability to figure out the state of the system and get things where you want to kind of reset everything and then start over. Okay. So you were just saying that it matters, uh, it matters about the motor and

all of that. So how do I pick a motor? Do I just figure out what GPIOs I have and the voltage rating for the motor? And, and by the way, how do I stop blowing up FETs? same questions it's all one question yeah um so this is where we really get into the topic of systems engineering so there's different types of motors um the the types of motors that i deal with most in in product development and uh tooling systems measurement systems or testers that sort of thing typically three types of motors, stepper motors, brushed DC motors, and brushless DC motors. So that's kind of your first big decision is which class of motor or technology of motor you want to use. And I was talking about stepper motors a few minutes ago. Stepper motors are thought of as

being really easy, and they are easy to control from a position standpoint in that you give them a step command, they move a fixed distance, and then they stop. Wait, wait, wait. That's only if they have a controller. If they don't have a controller, you have to do weird things with moving the... Help me here. Help me here.

Well, so you're applying current to a couple of coils and they'll take out... The coils. If you have a six-coil stepper motor, you can just cry. That's the only option if you're making your own control for that. Why would you do that? Well, it involved Darth Vader. Anyway, the stepper motor, mostly they do have some sort of control system in front of them that lets you say, take three steps forward or three steps back. Oh, they do. I mean, you can definitely buy off-the-shelf controllers that will do that for you. Fundamentally, in the simplest stepper motors, you have A phase and B phase that you're controlling. You apply a current to A phase, and it will step and align along that. And then you switch and you apply to B, and it will just step through these different positions. Very simple drive, but it will do what you need it to do in a very basic manner.

And so people like stepper motors because conceptually they're very easy to control. They do have some downsides. One I talked about earlier a bit where once you attach them to a load, if you don't have enough current, enough torque constant in your motor, and you apply that step, it may not step. It may not get where you want it to go. And without any sort of feedback sensor, you have no idea that it's gotten there. So in practice, when stepper motors are used, they typically are well oversized, at least 2x, to make sure you have some sort of torque margin in operation and you don't miss steps because you are relying on the fact it's going to get where you want it to go when you say go.

They can also be really noisy. Anybody that's got a low-cost 3D printer typically has stepper motors and they sing. You hear them at the step rate. You basically are getting a bunch of harmonics at the step rate and it'll sing to you. So they're quite noisy. And because you oversize them, they tend to be expensive. The main reason I don't like to use them in a lot of product work is you don't have any way to know how much torque margin you have without building an elaborate test setup that has some sort of torque transducer to measure the actual load in the system. Whereas with a feedback system, you can always measure the response from the sensor and the input you're giving it to assess how much margin you have in design.

So a stepper motor, that's not a PWM motor. That's a go now, step one, step two. You don't PWM that and do the acceleration and velocity profiles? You can. So you can do your acceleration and velocity profiles by how many steps per second you're commanding. So you can command step rates. So definitely you do that sort of thing. And that helps, right? Because the acceleration load, the inertial load during acceleration is critical, especially if you've got a large inertia you're moving. So to keep from running out of torque margin, you may want to accelerate at a low step rate so that you don't overload the system and miss steps. I wish you'd told me that 16 years ago.

Yeah, so many things. Okay, anything else about stepper motors that we should know? I think that's a good overview. They are nice. They are simple. To me, the big deal is, since I do a lot of stuff in product and products at higher volume, stepper motors tend to be not the best cost performance choice. So I like using them a lot when the NRE, the engineering time versus part cost, sways towards, hey, it's much better

to get something done quickly because you can get them in and working rather quickly. And if you have to put in a big motor and your time's more valuable than the motor because you're only building one, it's a good solution. But past that, the next choice for a really high-performance motor is a brushless DC motor. So those are kind of like a stepper motor, but they are typically just a three-phase motor. Now, there may be multiple poles and slots in the motor, so that three phases may be

repeated multiple times around the motor. But electrically, you only have three leads that you're driving. Those can be really high-performance motors. So, you see them a lot in drones for running the rotors because they can operate at very, very high speeds and they can be very easy to commutate in that application. But they can also be really good for really high torque applications and you can get really, really smooth torque out of them depending on how you drive them. The complicated thing with brushed STC motors is there's a lot of different drive algorithms out there. So you really want to think about how you're driving the motor depending on the needs of your system. You said the word commutate, also commutation, which is where you change the pole. Let's see.

You're moving the motor, and so you have to keep moving the poles around. It's kind of like there are a couple of magnets and you're energizing which magnet you want the motor to go towards. But you have to stay ahead of the motor because you want it to be moving and you don't want to chunk, chunk, chunk. And so that commutation is usually there's like sign commutation that helps you hand off the motor from magnet to magnet. What else is there? Right. So, the three technologies I mentioned, stepper, brushless, and brush DC motors, basically the trick there is how you commutate each of these. They have different commutation methods. And what you were saying, right, Typically, these motors have a permanent magnet that's attached to the rotor or the stator. Different things can move. But the trick is you're using

a coil of wire to create an electromagnetic field, and that's interacting with a permanent magnetic field to generate a force, which, because it's in a rotational system, is a torque. And so, the whole trick is how do you keep the electromagnetic field in the correct relative position to the permanent magnetic field to generate the force and keep things moving because they want to align and that's a stable position and you stop. In a stepper motor, that's what a step is, is you turn on this coil, it creates a field, the permanent magnet or the electromagnet sets up that field, and the permanent magnet moves, and then it stops in that position. So that's what you're doing when you do a step. With a brushless DC motor, you can actually run it in kind of a stepped application. application it will align but the commutation techniques are all about how you how you deal

with figuring out where you want your electromagnetic field to be with the vector so that you you keep the torque moving and kind of the shape of that so there's there's really simple ones they use hall effect sensors so you typically have three hall effect sensors and it provides you're able to create a very rough um sinusoid and for your your um drive pattern and because of the placement of the magnets on the the rotor of the motor and the placement of the sensors it's set up so that the the electromagnetic magnetic field is always leading the permanent field and that's what keeps the system running. Then you get into more complicated techniques where you might use an encoder to sense the rotor position and then you can start shaping your magnetic field. So, a sinusoidal drive uses, as you would expect, a sinusoidal current field to try and generate a sinusoidal magnetic field

that's running ahead of the permanent magnets. And that will generate a smoother torque, right? So as the electromagnetic field is less and less a sine wave, the rougher the torque is going to be, there's going to be a torque ripple to the system. So there's a lot of techniques around doing that. There's sensorless control. And what that does is it looks at the back EMF of the motor. So, these coils, you're applying a voltage, getting a current. That current creates a magnetic field, which is then going to generate torque. But it's also working in reverse at the same time.

As that coil moves through a magnetic field, it generates a voltage, and that's the back EMF. And so that's a voltage working against you as you're applying a voltage trying to create a current. It's generating a voltage that's countering you and trying to reduce the current. But you can use the difference in kind of expected current when you're applying a voltage and the current you're actually generating, that difference should be due to the back EMF, and you can use that to estimate how fast a motor is going. And in BLDC, they do a centralist control where they use that to let them estimate what the field should be and update the commutation.

And this is just the motor. I mean, we're talking about controlling the motor. We're not talking about controlling the system right now. This is the motor. I mean, we're talking about controlling the motor. We're not talking about controlling the system right now. This is the motor controller, and we're probably going to be putting something on top of this in order to control the arm or the caterpillar or whatever it is we wanted to control. Absolutely. So that's why I didn't say it yet, but brushless DC motors are typically used in higher performance applications because you do have this whole other lower layer underneath your system control, just trying to control the torque in the motor. Brushless DC

motors, if you use field-oriented control, which is kind of one of the nicest ways for driving them, gives you the best torque it has three independent current loops uh i think they're typically pi loops just trying to control the current in each of those phases to get you that so they get pretty complicated so what you see a lot in consumer products i'm sure everyone here has dealt with um are just brushed dc motors they're a lot simpler you apply a voltage and they spin and that's because the commutation is handled mechanically. There's literally two contacts inside the motor that are brushes and they touch a commutator ring, which is just a slotted ring that basically connects the voltage to different coils in the system. And as you move that motor a little bit, the brush will contact a different section of the ring, and then you'd basically turn off one coil and turn on the next. And so, they're a mechanically commutated motor, which makes them extremely simple to drive,

but they just spin. It's really kind of a speed control type thing. And the brushes wear out, right? And the brushes wear out. Yep. So nice thing about stepper and brushless DC motors is neither one of them have a mechanical brush. So the main wear components in those is just the bearings that support the shaft. Whereas a brushless, a brushed DC motor has the bearing issue plus these brushes can wear out. And so the really expensive, really fancy brush DC motors use gold contacts and do a lot of stuff to try and increase brush life. So with your standard brushed DC system, you're applying a voltage or typically a PWM.

And then you don't have to worry about how the motor is being driven. It's taking care of that itself. And you can focus more on the high-level control for your system. Okay. So, high-level control for our system. Let's talk about that sensor we're going to measure. And since this is mechatronics and not chemistry or whatever, we're probably... So dismissive. If he was a chemist, I would be more excited about chemistry, okay? But we're probably talking about encoders. Yeah, that is definitely the most common sensor in use.

It's a good position measuring system. Typically, they're relative. They can be absolute, but they're typically a relative position. Absolute ones are expensive. Tell me this show is going to be the greatest hits of my nightmares from 10 years ago. Yes, encoders. Encoders, sorry. Black and white little stripes. Yep. There are different types of encoders. We're talking about quadrature encoders typically, which means you've got the common system is optical, it's clear and black,

and you're shining a light through it. And usually there's two receptors and they're 90 degrees out of phase. And so there's a trick there where, because the signals are in quadrature, if you look at the state of, say, channel A, when channel B transitions from low to high or high to low, you can then figure out which direction things are moving. So that's why you use a quadrature encoder. But yeah, they're low cost, they're optical. Downside being dust and grime can get in there and mess your signal up. But that's the most common type of encoder. We have used some at work that are inductive to get really, really high accuracy

or really high resolution encoding systems, but most people will deal with an optical encoder. I had one that was a leaf spring and a series of small divots. That was for a toy. It was very cheap. It was pretty hilarious that I would have to count the divots.

Okay, so encoders, you mentioned quadrature encoding. That's one of the main ways that we have to read the encoders. But for most of us, that means finding the right part in the microcontroller and saying, here, use these lines and tell me where things are. Are there reasons I should look inside my KEP decoder? Inside the quadrature decoder block? Yeah. I mean, if I just hook up the GPIOs right, can I just believe what the software gives me

or do I have to understand this? Most of them are, I haven't had too much of an issue with believing the values. I think it's important if you can to use a quadrature decoding block. So a lot of microcontrollers support that. And so they deal with these two signals that are 90 degrees out of phase and will just give you a raw count. So they deal with counting up, counting down, and that helps a lot.

If you don't have that, then you're probably in a situation where you need to be interrupting on every transition and looking at the other channel, and that can get... So bad. Yeah, it can be a really high processor load. Especially, I mean, the thing is at some point, you're going to stop and you're going to stop on a transition.

And the thing may sit there and toggle up and down at really, really high rates. And if you miss transitions, you start losing your position. So it's really nice to have that all taken care of in digital hardware. Okay. Don't make your own quadrature encoder software. Not if you can avoid it. Absolutely. But we've talked about the quadrature encoder and the black and whites, but you also said relative versus absolute. This is almost always relative. And in order to get to an absolute position,

you have to do some sort of homing mechanism, such as the time where I homed it by putting as much current as I could until the back current almost lit on fire, except for that one day. And then once you know it's like stalled, you say, okay, this is the end of my travel, and now I have to travel the other direction. And I know I have 5,000 counts to go the other direction, and I should be done there. Absolutely, yeah. So if your mechanism has a hard stop, that's a really common low-cost way of doing that is to drive to a hard stop. And basically, you can do it a couple of ways. You can look at the current going into of ways. You can look at the current

going into the motor. You can look at the error signal in your controller. So your PID error, it's going to spike when you stop moving. That's another trigger you can use. The important thing there is to make sure that you have enough margin in your system. You want that when you go push on something, you want to be able to push on it significantly harder than you have to push on your system otherwise in operation. Otherwise, you might false home. That's a real problem. You think you're in your home position and you might be in the middle of your travel. And then you want to make sure that the mechanism is strong enough, the drive system is strong enough to withstand the torque that you're going to apply.

Another common way of homing is to put a separate home sensor somewhere in your system. You can use a mechanical switch. It's really common to use a really simple opto-interrupter and have something in your mechanism trigger that as you go home. So that's nice because you don't have to whack anything and you've got a very sure signal. If you have a rotary system that doesn't have a hard stop, you can get a lot of these. These encoders will provide an index pulse. So it's just a third channel that you'll see as things go around to let you know at least one place in your system that's at a known angle and then you can work relative off of that. Those are nice because it is hard on your motor to slam into the home position when it isn't expecting it. Absolutely.

And we talked about brush wear. If it turns out that your brushes always end up in the same place and you apply a lot of current, you may really wear your brushes in one particular place on your motor and lead to early failure. So failure. A lot of these mechatronic things, toys are one thing. If they fail, nobody really cares. But you talked about printers. Okay, maybe nobody cares that much, although it's money. Turbines, though, you don't want those to fail. They're really hard to install. How do you deal with redundancy in hardware and software?

Do you worry about those things or you just do the math right and have the confidence and do your failure analysis and be done with it? So I think, you know, you kind of hit on something. First is how much do you care and how big a deal is it if you fail? So that's going to, you know, weigh how much cost you want to put into failure redundant systems or redundant systems to help you deal with failure, time and effort, that sort of thing. Yeah, like you said, with printers, it was a low-cost consumer device. We didn't want anything to fail because the game there is all about selling ink and you lose money on the printer. So you don't want people buying multiple printers.

You want them buying a lot of ink. But at the end of the day, if something broke, it wasn't the end of the world, whereas there's a lot of systems that are safety critical. So for me, a lot of it is looking at how you detect errors in the system and how you deal with errors in the system. No magic bullets there, but you really have to do some analysis and think about the situations you're putting your user into and what could go wrong. And we've talked about some of these things, the homing and stall detection. If you're going to do homing or anything with current, you might as well have your stall detection on all the time so you know if there's, you know, somebody's holding the caterpillar and it can't move. Absolutely. So I think a classic trap in

any sort of engineering development system is you get your requirements from marketing or some users, and they talk about how this thing should work. And everybody talks about the happy path. If everything is going great, what's going to happen? But looking at failure modes and how you're going to deal with failure modes is critical. So things I like to do in all systems, have that home sensor, verify you can get where you need to be. Leave your stall check on at all times. You may set your stall limits differently depending on the mode of operation. So when homing, you may set them very low, so they're very sensitive, so you're sure you get that stall quickly. In an operation, you may increase those. Another thing that's

really important to do is to use timeout. So any move that you expect to complete in a certain amount of time, if you don't get that move complete signal from your controller, so you've got some trajectory and, hey, you've reached the end of your trajectory, I should be getting close to where I want to go. If that doesn't respond in time, throw a timeout and shut things down. Those kind of simple sanity checks can really save your sanity in the long run when you're debugging your system. When things are running, you leave something running overnight in the lab to test it out, and you come in the next morning and it's just stopped. If you don't know where or why it stopped, you're in for a long hunt. So any sort of basic sanity checking is really helpful.

Another thing we did at HP that actually turned out really great is while we had a quadrature block handling the counting for us, we were able to access some additional registers. And we were lucky because it was a custom digital ASIC. But we looked at timing registers for each quadrature pulse. And we were actually able to detect if during the assembly process, when they were greasing the carriage rod, if they accidentally got some grease on the encoder strip, because we could look at that timing block and see issues, see huge variation in timing of certain pulses. If you didn't have that visibility, that would have slipped through and you lose counts over time and ends up the print would shift down the page and looks really bad. But just by adding a little additional information, we're able to pick out another signal and do a whole other error check on the system. So any sort of little tricks like

that you can do in your system where you can identify things operating kind of out of whack can be super helpful. You mentioned manufacturing, which is another area that mechatronics is sometimes more difficult than non-mechatronics. Plain old electronics. But motors are resistors of a sort, and resistors have variants. And actually, all resistors have variants, all capacitors too. And so even if I design the perfect system that works on my desk, reproducing it, well, reproducing it is hard because of these variances all total up. I've always thought this was the real reason we do control systems. Absolutely.

I mean, absolutely. One of my professors in grad school liked to say that, you know, the reason we do feedback control is because it linearizes the system. It takes a lot of variation out. It takes some of the nonlinear behaviors out. It really seeks to let the system adapt and take care of itself when that variation crops up. So that's a really good reason for it. And manufacturing is definitely a challenge, especially in the consumer realm. you what you called uh regular electronics kind of some sort of pca in a box basically uh but but once things start moving there's a whole host of other issues that crop up and that goes back to the mechatronics where it's mechanical electrical code and control systems all interacting with each

other so um it's important to make sure you're you're watching what's going on in manufacturing things i like to do a lot or are exactly what talked about earlier, like maybe apply a fixed PWM or do a move at a known speed and monitor the PWM applied or the error that you're getting in production. So you could see if maybe a grease step was missed because this unit all of a sudden requires a much higher PWM than nominal to execute a given move. That can be a great signal to say, hey, there's an issue in manufacturing. But it is a different set of testing than a lot of CMs are used to doing. In one project I worked on, because our boards seem to be on some weird electrical edge, we ended up doing a burn-in where we made it run the system. We made each board run a golden system for, I don't know, 10 hours or something ridiculous.

And many of the boards failed. Like they couldn't run it or they couldn't, or there were fires. It's the story of my career. Christopher's right. Is that common or is that rare? I mean, I've definitely seen systems where people wanted to do run-in. I think it can be a great error check. It definitely just addresses the fact that there's low margins somewhere in the system. So, the big companies that have all those companies typically don't have the budget to do

multiple rounds of prototypes. And so you are, are forced to do more kind of ad hoc things and really try and work around issues as opposed to having the time to go in and engineer things correctly. With respect to motor drivers and overcurrent and blowing fats and that sort of thing. That's where doing a full system design is really, really helpful. Fundamentally, these systems are big power conversion systems. You've got some power supply you're plugging into the wall, and you're going to try and convert that electrical power into mechanical power to move some load where you want it to go when you want it to go there.

It's like a dam in reverse. Yeah, kind of so, right? And that was actually what was tricky for me when I worked at the wind turbine company is that system wasn't at all about driving the wind turbine. It was around getting it to not run away on you, to control the speed. So it was running a control system in reverse to me. But yeah, it's an energy conversion system. And if you mismatch components, if you have a problem with your impedance matching from these different realms,

from the electrical realm into the mechanical realm, you can cause real problems. And that's why we see a lot of issues with systems where the electrical system and the mechanical system are designed independently of each other. For a mechatronic system, I think it's really important to have someone who can be a mechatronic lead and talk to all the disciplines and understand what's going on in each of them, at least at a high level. So the way I like to design these systems is to look at the load. Don't let the ME tell you what the gear chain is going to be, what the gear ratio

is going to be. What is the load? What is the inertia you have to move? What is the friction? And what sort of motion profile does that need to go through? What speeds is it running at at the load? And what kind of accelerations are needed because that's going to fundamentally define what the output mechanical power is. Then you can look at different gear ratios and look at how the shaft speed that's driving that load, whether it's a rotary load or a linear load, you're going to go through some sort of belt or something like that to convert it to a linear thing and you're going to end up with a shaft speed you're driving that gear ratio is kind of a an impedance tuning tool right it's going to change uh reduce torque as you increase the gear ratio but increase speed um at what's required from the motor and so that

gives you a tool to look at um what sort of torque and speed your motor runs at which then are going to define the current and voltage you need. And between the gear ratio and the motor winding itself, which is basically how many turns of copper you have inside that motor, you have two tools at your disposal to alter the ratio between speed and torque and thus between current and voltage. And so those two knobs allow you to change that impedance match problem. So if you're blowing FETs, you probably have too low of a gear ratio or you have too low of a winding resistance. So if you increase your gear ratio, you're going to reduce the torque, which is going to reduce the current needed,

or increasing your winding resistance, which means putting in more amp turns, but also so you get more torque per unit current. It's going to increase your resistance, but it lowers your current, and that will keep your FETs from blowing up. The trick there is you still have to have enough voltage to drive the current you need to get the torque. So doing that sort of multidisciplinary optimization to balance voltage and current and torque and speed, that's the real trick to keeping from blowing up your electronics. Okay. So do I need to get a master's degree for this? Or is there a book? Or how do I learn enough? Is there a pamphlet? Yeah. Do you have a cheat sheet?

Three simple steps. It's actually not that hard. So while we talked a lot about differential equations earlier, this sort of analysis is really kind of algebraic. You just, if you think about a motion profile,

it's a standard trapezoidal one. It's a constant acceleration, a constant velocity, and then a constant deceleration. So you can, if you want to do a one-point analysis, you could say, well, if I'm at the top of my velocity profile, but I'm still accelerating, I'm kind of at my maximum velocity and maximum acceleration, that's the highest load on the system, right? You're going to have the most drag, especially if there's any kind of hydrodynamic drag, and your full inertial load.

That's a single operating point. And then you can start playing with different gear ratios and just back calculate how much torque is going to be needed out of the motor and at what speed. And then you can just go look up on motor curves. Hey, if I'm operating, I've got a given input voltage. What does that motor torque speed curve look like? And am I going to be able to drive this load point? And if it doesn't work, you can look at changing motors or you can look

at changing gear ratios. So it's actually not that difficult to do. It's really, you know, it's first principles mechanics, first principles electronics. But doing both together is the real trick. You got to do both together to really figure things out. Going back to budget, if I have a robot that's working okay, not great, but okay, and I want to improve it, do I spend the money on actuators or sensors or powerful processors? Yeah. Unfortunately, there's never a single answer there. So as kind of the theme here. It depends. It depends.

But also, again, it's back to the – it's a mechatronic system. It's mechanics and electronics and control systems. So your problem could be anywhere, quite honestly. But there's some fundamental things first of all the best control system is it's never going to be easy to fix crappy mechanics so are you sure because i was going to use ai to do it you know wondering how long can't we just control the motor with deep learning yeah i'm sorry doug i shouldn't interrupt it Can't we just control the motor with deep learning?

Sorry, Doug. I shouldn't have interrupted you. You want to have good pieces to work with, right? Don't make the problem bad for one guy. It sounds like your physicists that you described earlier, they made their life, right? But they caused problems for everyone else. So good mechanics are important. Make sure you try and reduce friction as much as possible. In my experience, you can't control friction to some given level, but you can do

your best to minimize it. So, that is important. These systems tend to have a lot of problems if there's backlash in the system. So, if you have to reverse direction, you don't want to be slamming through backlash. That's going to wreak havoc in your system. So if you have to do that anti-backlash gearing or some other mechanism that may not have backlash, a lot of robots just use direct drive to just eliminate backlash completely. Those things are important and compliance. The stiffer something is, the less you're going to build up energy in your system between the motor and the load that's going to come out and bite you later. So, cleaning up your mechanics as much as possible is a great place to start. Motors are not ideal torque sources, right?

So, especially the low-cost brushed DC motors, they have a lot of torque ripple to them. And so if you have an issue and you can trace it back to there's a ripple and it's coming out of the motor, you might want to just try and buy a better motor. So if you have the opportunity moving to a brushless motor with a sinusoidal or field-oriented control, that can give you a much smoother torque output and can clean up a host of issues. Around the sensors, you've got to make sure you've got enough resolution. So a lot of people, they focus only on resolution, so they put the encoder on the shaft of the motor where it gets some multiplying factor as it goes out through the gear train to the load, which is great. You've got a lot of resolution, but you also have a lot of open loop between your sensor and your load. So fundamentally, these feedback control systems,

they control the position of the sensor or whatever the sensor happens to be reading. So if you are reading the motor shaft position with an encoder and you're running your feedback control on that, you're controlling the motor shaft. You're not controlling the load. That's a byproduct. But any errors in your gear train, any errors in your mechanics, any backlash, that all has to be compensated in another manner. So maybe you need to just move your sensor. That might give you better control. And also, fundamentally, if you need a higher resolution for, say, stopping accuracy than your sensor's fundamentally able to measure,

you're going to have problems. So, spending money there certainly may help you out. Personally, I have not run into computation as being a limit. So, faster processor hasn't been a problem. In practice, the fastest control loops I've dealt with were around three kilohertz. They can often be run just with integer math, and so they're not that heavy a load on today's modern processors. So I don't think that doesn't tend to be an issue unless you get into very advanced controls where there's a lot of math going on. But that said, things like having that quadrature decoder, that also helps a lot.

If you had to do that with interrupts, you're driving your processor load up considerably. Okay, but we started this conversation, you said that whatever the background of the person designing the system tends to affect how the system is designed. And you've given us all these things we should do with motors and sensors, but you're a mechanical engineer. Yeah, so I just apply a wrench and it always works.

So I think, you know, fundamentally, these things, mechatronics, which the term has been around for a while, but I think it's only been in the last five or ten years that academically the engineering schools have started to get behind it and realize that there is this fundamental cross-disciplinary education necessary for mechatronic systems and to do a good system. So, you know, if you're working in a company today, often companies have either a technical lead who happens to have trained themselves in a cross-disciplinary manner, or a lot of companies have what they call their motors guru. Those are people that have taken the time to just learn what they needed to learn from the other disciplines. So, you know, I know a little bit about electrical engineering enough to at least speak the language around motor drive and encoder, a little bit of signal conditioning, that sort of thing. And I spent some time in school getting some additional CS training so that I could be dangerous in the code and implement things myself.

So, in business today, you're probably going to have to go out and kind of train yourself and partner a lot with people that are in the other disciplines that are required for a mechatronics system. In school, if your school has a mechatronics program and this is the kind of thing you're interested in, that's a great place to go because they fundamentally understand

you're going to need to learn stuff from different engineering schools. And if your school doesn't, then it's a great opportunity to take some of those technical electives in other engineering disciplines. Do you have a book recommendation for control systems? There is one. I think I sent you a link, but it's by a guy named Nice, N-I-S-E, that I found to be kind of the most approachable. Unfortunately, it wasn't the one I used in school, but I found it from another friend who had it. So I thought for a Controls 101, the undergrad controls class, it was one of the most approachable ones that I'd seen. Yeah, they have it.

It's textbooks priced, though. Ouch. That it is, yeah. So maybe a used copy. The math hasn't changed, so older editions are probably a good way to go on that. Yeah. And do you have a differential equation book you'd recommend? No. I think most of the math I learned in college was through the engineering classes. So, I didn't really like any of my math classes very much. I like the AIM, American Institute of Mathematics or Mathematicians or MICE or something.

It's not MICE. It has a list of approved textbooks and two of them are differential equation ones, and Notes on Diffie Qs is something I've read without doing the problems. So it's not like it's stuck a lot, but it's readable. Awesome. They have all kinds of books, including Calculus, which I know is an area people kind of get frustrated with. But calculus in context was really good. Cool to check that out. I've, the YouTube channel 3Brown1Blue, he does a really good job.

He does a really good job. Yeah. Explaining things that I... That kind of math to English conversion. Yeah. And cartoons. I'm always in it for the cartoon i've seen some of your cartoons that those are pretty cool um okay let's see we have a few listener questions actually the truth is the listener patreon wrote most of the outline for me this week

but these are specific listener questions gareth asks how to deal with cabling? How do you get a signal across an axis that moves or rotates? Yeah, I've seen two real techniques for that. One is a large service loop, right? So you have a cable, you give it some slack, make sure you don't bend it too much to where it breaks. That's probably the most common. The other one I've seen is slip rings. So in the wind turbine too, because a wind turbine has to spin 360 degrees to face the wind, and it may go around and around and around and never come back, you use slip rings, which are kind of like commutator brushes in motors. The downside of those is they tend to be very noisy.

So for high-speed data, probably not such a good idea. But you could likely use that to transmit power and then maybe use wireless across that interface. But no wonderful solutions there. I was really hoping for a magic solution. This seems like a problem that's begging to be solved. That it does. It is a nasty one. Do you think Asimov's three laws of robotics are relevant to mechatronics design? I think they're relevant. I don't know that they're relevant to mechatronics design, at least not as I see it. I see the mechatronics stuff as a little bit more low-level, whereas they apply more to the higher level decision making.

So all the AI people out there, I think that's the level of the control that those rules are going to apply to. Andrea and Tom worked together to make this question. In the next 10 years, are we more likely to be hurt by an offensive robot or by tripping over a Roomba? I really hope it's nothing done on purpose. I'm most likely, I think, to step on a Roomba and go like a cartoon and have my feet fly out from underneath me. My biggest concern is kind of around the more industrial malpractice type things. So, classic ones in mechatronic systems these days are the Toyota unintended acceleration problem and the 737 MAX issues, right? Those are clearly mechatronic systems. And through one engineering failing or another, they took a safety critical system and didn't do what they needed to do and caused people to be injured and killed.

Well, that's a far more depressing answer than I was hoping for. I'm sorry. Okay, that's allowed. Christopher, do you have any questions? We covered a lot. I know. I want to go back and like ask all of the questions again now that we've done it. Oh, you know what?

You kept mentioning backlash and I wanted to make sure we defined what that was, because some folks may not know. Sure. In a gear train. So you have gears, two gears, they mesh together. There's teeth that come together. In order to allow them to come together and then come apart cleanly, you have to have a little bit of a gap, and that gap allows the two shafts to rotate just slightly with respect to each other so um if you think about gears spinning one way and then switching direction uh the gear teeth kind of go from contacting one set of faces to the opposite set of faces and that little bit of of gap there is called backlash and the better the higher quality your gear train the lower that the less backlash you'll get. Um, but, but lower cost gear trains can have quite a lot. And if you're trying to, you know,

very carefully control position, and let's say you get right to where you want to step, and then you realize you want to reverse just a little bit and you have to go through that backlash. It's really a really nasty problem to deal with. So, you know, you want to try and avoid that. There are fancy mechanical designs that actually spring load the gears. Yes, yes, yes. My nightmares are coming back. Exactly. You don't want to do that. So, typically what you want to do is you want to come in, you want to come in slow from one side, from the side you're driving from, and push your way into position. And if you overshoot a little, call it good and stop. You're not going to get there.

The only other thing you're going to do is back way up. So you go through the backlash and then kind of retry the whole move. I had a system where I was stuck designing. It was a service station. So the thing that cleans the printheads in a printer. And we had a stopping accuracy requirement way tighter than we were really able to do. And we had to do a series of moves to kind of find our way in. And we would kind of both repeatedly go through these moves, long moves to go through the backlash, and then also do a little adaptive algorithm. So kind of learning how far we needed to go until we could dial the thing in. So it would sit there and retry and retry and retry until it learned its way in.

So it can be pretty painful to deal with. Unit-to-unit variation. That sounds extremely familiar. You sent me a list of megatronics you've worked on, and we've talked about inkjet printers and wind turbines. But I have some questions about some of the others. Sure. A pizza vending machine. That was a cool one.

That was a project I worked on last year. I affectionately called it the Robo Toaster. So basically what it was, it's a machine. It hasn't hit the market quite yet, but will be soon. Where your local pizza shop will cook up pizza, cut up the slices, put them in aluminum trays, and then this machine has a big refrigerator with cartridges that hold a bunch of these aluminum trays. And then when you order it, the trays get dropped onto a pusher and through a series of conveyors. They get run through

essentially a toaster oven, kind of like a lot of the sandwich shops have, that just drives straight through the oven to cook it. And then it moves it to some different output chutes where you can retrieve it. So that one was fun. It was something like seven different mechanisms that are used to move the tray through the system. And we can operate with, I think, three different trays in flight at the same time. So there was a lot of cool work around a bunch of state machines to control things properly and coordinate the motion and keep everything synchronized. That was really fun. A mechanical baby crib.

Is it a mechanical crib or a mechanical baby? Just for clarification. I think there should be a mechanical baby for testing. But the crib is, yeah, it basically is an electromechanical system to rock a baby to sleep. So it just oscillates back and forth. Yeah, when our robot overlords come, they're going to be like, really? There's nothing that could go wrong with that. I guess they had the wind-up ones back in the day.

Yeah. Absolutely. I think they've been very successful for a high-end smart baby crib you do it fast enough the baby's contained by centrifugal force uh a gene sequencing machine which i think deserves a fist bump because i have worked on similar things so yeah so uh yeah at simplexy one of our big clients is in that space and so we've done a lot of work with them out of our San Diego office. I've only gotten to do a little bit with them, but we do a lot of work with them.

I specifically worked on some vibration isolation, which we didn't really talk about, but is a systems dynamics thing. It's very close to control system. Just get the optimal rubber feet that would isolate this very sensitive optical process taking pictures of DNA. Yes, optimal feet. That was the thing you lit on fire, right? You. That was the product you lit on fire, right? The DNA scanner. Yeah.

I mean, twice. One's just the board, the other time just the voice coil. Never the whole system. There you go. Share the fun. And you have worked on a VR system. Yeah, again, I was a little on the periphery with that, but we actually did some work with Valve on their HTC Vive system, specifically around some of the speed control for the laser scanning system that's used for positioning. Lighthouses.

Lighthouse, exactly. Worked with Alan Yates a little bit on that. He's been a guest. He's a great guest. Smart guy. Well, Doug, do you have any thoughts you would like to leave us with? You probably noticed a theme during my descriptions earlier of some of the problems, but really it's all connected, right? I think that's the main thing around mechatronics system is you got to realize that the electronics mechanics, the control, the firmware, they all interplay with each other and they really need to be thought of together as a system,

not as individual pieces that can be designed in a vacuum. Our guest has been Doug Harriman, CTO of Simplexity Product Development. not as individual pieces that can be designed in a vacuum. Our guest has been Doug Harriman, CTO of Simplexity Product Development. Thanks, Doug. Thanks, guys. It's been great being on. Thank you to Christopher for producing and co-hosting. Thank you to our Patreon listeners Slack group for their many questions, both here and answering each other, which is really neat.

And thank you for listening. You can always contact us at show at embedded.fm or hit the contact link on Embedded FM. Now I have a quote to leave you with that is only going to make sense to like 10 of you, but I'm hoping you really enjoy it. This is from Robotech. Dana, I don't get it. How can Rick get married?

I thought he already had a wife. Lisa Hayes. Dana, I think you're confusing him with someone else. Dana. But mom said he's married to his job. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them.

At this time, our sponsors are Logical Elegance and listeners like you.