Embedded - 227: Half of Everything Is Wrong

Episode Date: December 21, 2017

Anthony Navarro (@avnavarro42) of Udacity (@udacity) spoke with us about learning. We talked about the Dreyfus model of skill acquisition (an education-oriented technical readiness level) and a little about on trunk skills vs. leaf skills. Elecia took Udacity’s term 1 of Self-Driving Car Nanodegree and is planning to take the free AI for Robotics course next. Anthony is enjoying soldering lessons via Boldport (hear #171: Perfectly Good Being Square and Green). Anthony noted there is a free Embedded course on Udacity.

Welcome to Embedded. I am Eliseo White, here with Christopher White. A month ago, I mentioned taking a Udacity class. This week, we are joined in studio by Anthony Navarro of Udacity. Hey, Anthony. Welcome. Hi, how's it going? Could you tell us about yourself? Sure. I have a really diverse background, I guess. My education is in computer engineering. I have a master's there, but I also spent some time in the army, both as an infantry soldier and then as an aviation officer before I went on to Lockheed and worked in some of their space stuff

and then transitioned over to their autonomous systems division before I came over to Udacity. And what do you do for Udacity now? So I'm the product lead on their robotics and self-driving car programs. Cool. Well, I know that you listen to the show, so we aren't even going to tell you the rules for lightning round. We're just going to go right to it. Great.

What was your least favorite college class least favorite college class oh that's a tough one uh there's so many to choose from i know well there were some of the you know some of the technical ones that i were hard and i didn't like at the time but they were really relevant in the future. But probably some of them were more of the, I guess, soft or non-technical classes that I had to take. But I don't remember. I tried to forget them, so I guess I don't have one to specifically point out. I enjoy learning most things. Hardware or software?

A little bit of both. Complete one project or start a dozen? I have a history of starting many, many, so. I don't think we've had any answers that haven't been that yet. What's a tip you think everyone should know? Always learn more. Continuously learn and try doing more. If you could learn one subject instantly, matrix what would it be i think it'd be

language just being able to understand different languages any particular really like to learn japanese all right favorite class in high school uh favorite class in high school was probably an a plus plus or a plus class for computer repair really turned me on to just actually working with computers, which is what I thought computer engineering would be when I got into it. It's like, oh, it's hardware. Cool, it's taking computers apart. I like that.

It wasn't what that was, but it turned out that I liked computer engineering as well. Okay, let's get on to the meat of the show. And that is, I guess we should start with what is Udacity? Yeah, so Udacity is an online education website that you can go to and learn things like AI, front-end development, self-driving cars, robotics, Android, basically a lot of different technical topics with some business topics as well, like digital marketing. And you go on, you have a classroom. It's kind of like the term was a MOOC from a while

ago. I think the term's kind of gotten away from that a little bit, but that's basically what it is. Anybody can join, and from anywhere in the world, you can learn how to learn a new skill. And at what level are we talking? Like technical trade school or advanced university or? It's a little bit of everything. So we have an introduction to programming course where you can go from zero to being able to write hello world and do some Python programming. And then we have programs like our robotics and self-driving car courses that are advanced and you need to be able to solve, you know, solve programming problems, abstract, like given a word problem, it into you know programming

and then also understand the calculus matrix linear algebra and everything that goes with that so it can be pretty advanced and we do see ourselves as a job ready training so going through our courses the goal is you know you're either trying to up your skills or get a new job. And you said MOOC. Those are the massive online courses. I remember there were some at Stanford like 10, 15 years ago. Yeah. More like 10. And there were 10,000 people signed up and it still got graded, but it was mostly that it was very robo-graded.

Yeah. And that's kind of where, you know, I guess Udacity spun out of. Sebastian Thrun, our founder, he was a professor or he is a professor at Stanford and he decided to release one of his courses online for free. And when he realized that the top student in the course was number 400th in everybody who was taking it, he realized that there are so many people out there that are really smart that may not have access to the Stanford education, but could benefit from what they were teaching. And so that's part of why he founded Udacity. I've always been very fascinated by the democratization of information and how information seems to want to pass itself along. Yeah. But Udacity also does have a pay-for aspect to it.

We do. So we started off doing free education, free courses. And that's great. But what you see even with the original class is that you have, let's say, 100,000 people sign up. Very few of those actually complete the class. And when you introduce this option to pay, well, or you require the payment, people actually complete the class a lot more because now they're tied to it. So you may have fewer people enroll, but you have a lot more people complete, which is kind of what our goal is, is to get you through the material, not just get you to sign up and watch a few videos, actually get you to learn a new topic and to you. And so where you said maybe

10,000 people would sign up initially and some small percentage would finish, say, 100. That's probably a bigger fall off rate than you see. But if you get 5,000 people paying for it, as long as it's not too expensive, then you end up with closer to 3,000 people completing. Is that what you're going for? Exactly. Yeah. I mean, it's great that people, you know, want to sign up and we have, you know, we have millions of people that have signed up for our free courses.

And for our paid courses, we have, you know, tens of thousands. But it's, I think, you know, at Udacity, we see it more valuable for somebody going through and actually gaining these skills, bettering themselves, moving on, versus just saying, yeah, we have numbers and more people have signed up for our free classes. But it's really the completion and the betterment of those skills because it's great to have the information out there, but if you just sign up and don't actually go through it, it doesn't really propagate. And what does the paying for give me? Yeah. So when you pay for a course at Udacity, not only do you get access to the material that we've created, but we really focus on the services.

So mentorship, personalized reviews. So you have a mentor in the classroom that you can reach out to ask questions, kind of check in with weekly to make sure that you know, you're kind of motivated to keep going. And then when you submit one of your projects, because at Udacity, we're very project focused, we have quizzes in the classroom and everything to kind of help you learn concepts incrementally. But then the actual coursework, or the actual piece to pass the course is projects. And so when you submit a project, you actually get a human grader on the other side reviewing your project and giving you feedback on that. And so it can be as much as, you know, you did a great job, you checked all the boxes, but you missed some of these that could have made your code more efficient. And so it's not just you passed it.

It's also personalized feedback to better your experience. I did a project and I submitted it, and the reviewer noted that in my project documentation, I had said I went to the forums because I was having trouble and I found this reviewer's suggestions to be very helpful. And the reviewer on my project noted that was their suggestion. I feel like I got extra points for buttering them up without even knowing. Yeah, that's awesome.

Yeah, the forums are also another thing that we have in some of our programs. We don't in others. We're trying to constantly figure out what is the best way to service our students. So we had introduced something for a while called Live Help, where it was almost like a little box in the bottom where when you needed help right away, you could message and you'd go into a queue to get something back. We tried that out on some programs. The forums are something that we've, you know, removed from other programs and then added back to certain other ones depending on the level of skill um when you have a topic that's more advanced having a

forums that you can search and see more technical is uh more technical details is really nice and you have slack channels too i joined the slack channel and now i i don't log in because i i don't want to talk to people live. I want to search forums. That's fair. Yeah, I think it depends on the personality. And that's why we have so many different options. Some people like, you know, they're going through it and they want help now.

And some people like to kind of just peruse forums and find their own answer. And I think there's benefits to a few of those. Even when you look at the walkthrough videos that we do, we try to kind of give you this boost to help you get through the projects. But we don't want to give too much away because then you're really just waiting for these walkthrough videos to come out. So the challenge is, you know, how do we help people get through the course and get through it successfully, learn the skills, but not, you know, hold their hand too much. It's a very tough balance because you're trying to teach something. And knowing computer engineering is not the same as having the skills to teach it. Correct.

The whole education sub-genre. I mean, I'm a nerd about this. I minored in education, so I care about it a lot. How does Udacity choose the pedagogy? How do they choose what they're teaching and how do they find the research to make sure that it isn't the same as watching YouTube videos and not participating? Yeah, I mean, that's obviously a very important piece. If we're just YouTube, then why not just go to YouTube? Which is a great way to learn many things.

Yeah. How to change out the Prius's battery, for example, was fantastic on YouTube. Yeah. And I think what you see with YouTube videos are really good for are, I need this now and I need to do it and I don't need to replicate it in the future. I just need to like do it now and learn it now. And if I lead, you know, has the technical expertise in that area to be able to select the projects we should go for, the outline of the course. But they also have the understanding of how people should be taught so that when the content developers create a module, they can look it over and make sure, yes, this is all the information needed for this project or it's not. And so we kind of develop these courses with a project focus. And then once we've decided the project, we build backwards on all the content that's needed for that. And really focusing on that

content piece to make the project successful, hopefully, will get you the skills, you know, to make you successful in the nanodegree. So let's take some concrete examples and self-driving car, because that was the one, the term one is the one that I finished. One of the projects was the German traffic signs. Yeah. You're given a database of German traffic signs and a list of, well, not a database, you're using a lot. And you're given a list of, well, not a database, you're using a lot and you're giving a list of what's what. And you're supposed to create a machine learning algorithm to identify these signs. And in the course you go through and you learn about machine learning concepts and

TensorFlow and how to write a little bit of that code. And there are quizzes as you go on. But how do you choose what is going to drive that? I mean, you were saying the projects drive it, but how do you choose which quizzes and how do you just decide which modules go where? Sure. So when you take that one, you look at, you know, what do you need to do to be successful in that project? So part of it is learning TensorFlow, which is a framework for deep learning. Well, we don't expect the students have that deep learning understanding. And so we need to go through teaching deep learning or teaching deep learning, understanding like convolutional neural networks, and then also understanding what TensorFlow even is. So there's a lab that is called like mini flow that we put in there that kind of lets you write this packages that you'll be using in TensorFlow before you actually just get to use TensorFlow. So you understand a little bit what's going on behind the scenes. And then understanding, well, what are features?

What is this thing looking at? How is the convolutional neural network working? And how does deep learning even work? What is deep learning? And so to solve this, you have to apply these deep learning algorithms. You have to write a TensorFlow program to go through and identify the signs. And everything that you need for that, we go back and we teach you. Okay.

So you go back and you teach it. And you have little quizzes along teach you. Okay. So you go back and you teach it and you have little quizzes along the way. Yep. And to a large extent, my final project was going back or my project there was going back and taking each one of those quizzes and cutting and pasting and putting them in my project. Sure. And then tweaking. Yep. That's pretty common. it depends on the project but it's meant to give you these stepping stones for you know here's an aspect you learn here's an aspect you learn it depends on the content developer and how they're developing the course um but it is i mean with some of these advanced topics we do have to do a little bit more, you know, I guess, easier steps, stepping stones to

get there. So if you've solved this piece and you've solved this piece, then hopefully you can take these and put it together. And what you may have noticed in the first term, we also offer like challenges. And so I know for like the lane finding one, we offered additional videos that you could look through and try to apply your algorithm. So great, you did this and you kind of, you met the minimum requirement for the project, but here's how you could go through and try to apply your algorithm. So great, you did this and you kind of, you met the minimum requirement for the project.

But here's how you could go above and beyond. Now try your algorithm on this video, which has much more windy roads and shadows and basically watch it completely fail, which is how mine happened the first time I did that program and everything. And it's interesting until you get to the later lane following,

you learn better algorithms to use to actually identify your lanes and especially on windy roads. That was one of the useful things was that it started out with one lane finding algorithm and then you learn more and then later you get to do a much better, more in-depth landfinding system. And there was a lot of, like, I didn't necessarily need to complete the previous project to do the next one. But it sure helped if I understood all the little things like how to read things in and how to change their color spaces. And then I would get to dig in more on what was presented, but also those other little bits. It was a little hard because sometimes I hated that. I felt like, well, why did you teach me the dumb way to do this? Why did you make me go through mini flow?

All I really needed to know was that graphs are a thing. But then if I think about how little I know about graph theory, okay, that was kind of useful. But at the time, that's always a problem with learning. At the time, it feels difficult. It is, you're struggling. That's part of learning. How do you get people to get past the struggle?

That's, I mean, that's a hard question. I mean, to your point, even in college, I look at like my signals class. Signals are stuff. And it wasn't, I didn't see the relevance of it at the time. And I was like, well, why am I taking this? I don't want to do, you know, things with, you know, electromagnetic fields and things, all this other stuff. But then I got into computer vision and I realized, oh, all of that stuff that we talked about in signals actually applies here. I wish I would have paid a little bit more attention now.

And yeah, the programs, they're hard. And actually, our first program, you said like you did the lane finding one and it was, it was easy and you didn't really need to know everything there, but it was, you got through and it was kind of fun and you did learn some stuff. We do those first projects kind of in the way that make them kind of simple so that you start out the program seeing that like you can complete something, you can do this, you know, kind of help build up that motivation. And then it does scale up and go, you know, from there. But that first program and that first project, that's kind of how we aim to do that. But let's say two, three projects in, you're not continuing on. You know, part of that we have mentors there for, but that is a constant area that we're looking into.

I mean, we don't want people to just sign up and we get your money and great, we've got your money and we ignore you. We really want people to complete the courses. So we take the feedback that we get from our students very seriously and we really encourage them to give us feedback, either through Slack, through the feedback forms as you're going through the course, or if you find errors, we have a waffle board that students can report. And we have teams that go through all of that to really incorporate those. Now, in the university, you have a course that's out there and it may not change for years. We are constantly iterating on ours. And so even the self-driving car course, we're finding areas where students may get hung up and we're trying to figure out how to still meet the learning objectives, but ease their time going through it.

We recently got an update in Apple podcast where we can see when people fall off the show. And for us. In one show. In one show. In one show. And so if I started a screed on something everybody loved, but I hated, maybe people would just turn it off and we'd see a little cliff. You see that with the classes, don't you? You see where people just stop. We do. We can see how far they progress into the course.

And especially, I mean, one of the key points is projects. You know, where are they falling off on the projects and everything? And if they're completing a few of these projects, why are they stopping? And as they get further along, you look at self-driving car and it's, you know, three terms, about nine months long. And people who go through all three terms and fall off before the very last project, it's kind of, I mean, some of those are kind of, it is. And so it's like, we try to figure out why is that happening? And life does get in the way. We understand that. But if you make it that far, you know, what is, what is keeping you from moving on? Because there's gotta be, it can't be just the course because you've made it that far. You've put nine months in and you're like, you know, or eight months in and you're a month away and you've paid for the course.

And so, and you've been committed this whole time. So what is that, you know, thing? Well, one person falling off there, that's life. But you get start seeing two or 3% little dips in your data. And you can start building a class that helps with the retention and you can start finding out what's going wrong oh look everybody falls off after this module huh i wonder what's wrong there yep and we do that and we also look at the like the lesson feedback uh so every lesson has like a frowny face, neutral face and happy face. And, you know,

it's really helpful when people actually add comments to there too, because then we can see they didn't like this lesson and here's why. Yeah, but those are limited by number of characters. I know this. True. Which is maybe a good thing, maybe a bad thing. I think it's probably a good thing for us if we're trying to go through a lot thing. I think it's probably a good thing for us if we're trying to go through a lot of it, but it also might lose some of that information if you can't capture everything you were dissatisfied with in that one module.

So traditional universities solve part of this problem up front by having admissions, going through application processes, trying to make sure that the students they admit are actually going to succeed, or at least some great proportion of them. Do you do similar things where, okay, you say you're tailoring a course based on what you see happening, but is there upfront work to say, well, you need to have some level of expertise before you even attempt this? Yeah, and actually, Self-Driving car was our first program that uh put in emissions

admissions um and so we have our ai our self-driving car and our robotics program since they're considered the advanced programs that we have they all have some level of admissions and so we go through we ask you some questions we look at your background and based on you know how you answer all this kind of stuff that's how we admit people. So we do want to see people that have relevant skills that will be successful in it because it doesn't do you or us any justice to let you in if it doesn't look like

you're going to be successful. Now, that being said, we've seen people get admitted that are in high school all the way to have a PhD. So there's a wide spectrum of people that are in high school all the way to have a PhD. So there's a wide spectrum of people that are successful in these programs. And a lot of it, you know, comes from what your background is and your level of commitment. Have you not admitted PhDs? I'm kidding. Yes. No, we have. Yeah. I have turned away. I mean, cause you can have a PhD and that's great, but if it's not in a relevant area and you've never really coded, it just, you know, you're

probably smart, but you should probably start somewhere else before you just dive in headfirst. Because it is, if you don't have that experience, we say about 10 to 15 hours a week, but you can expect a lot more if you don't have that relevant experience coming in. And that's one of the questions with the admissions is how long do you think you'll have for this? And I bet anybody putting two to three hours a week is like, yeah, you don't want to take this. It's too much work. Yeah. If you only have two to three hours a week, I don't even know if you can make it through the videos. You can play them at a faster speed, which I really appreciate it.

You and me both. I love watching videos at like at least 1.25 or maybe up to 1.5, depending on the speaker and the topic. So Chris was saying about craftsmen and where do you fall? And you said job ready. And as I was trying to think about how do we talk about how familiar we are with skills, I bemoaned on Twitter that I needed a technical readiness level for education. And it was a guest, that I needed a technical readiness level for education. And it was a former guest, Jonathan Berry,

who sent me to this thing called the Dreyfus Model of Skill Acquisition. Were you familiar with that before I sent it over? I was not, but I thought it was actually really interesting. One of the models that I've been not a fan of was the way the army liked to teach certain things, which was rote memorization first and just, just memorize it, just do this. And I thought that was useless. I mean, I could, I can sit there and memorize all the pieces in a book that told me, you know, my limits and everything for helicopter. But the fact that I understood engineering and could understand where all these pieces connected to, I could tell you where all the components in the

helicopter were, but that's an understanding of how things work and why things are connected and why there are certain limits. So I learned better that way. I wrote memorization doesn't work for me, but the model was very, very interesting. Actually, when you sent that to me, I was like, I'm going to look into that more because that is a very interesting model. You learn depth. And I know a lot of engineers have trouble learning breadth and then depth because we want to go to first principles. And learning how to type in something like a monkey is very difficult because it's like, oh, what does this do underneath? How do I change this to be what I really want? I don't just want to type what you told me.

And that's a really tough problem because on one hand, you can do a lot of powerful things in TensorFlow or Keras and get things done, or you can remember how to do back propagation and build it all from scratch. How do you, do you like to learn first principles? I actually like to learn, I guess it depends on the topic. I, deep learning is an interesting topic that you've actually seen a lot of researchers and people say, we need to go back to academia for some of this because it is so easy to apply that we're not doing it justice for, you know, the problems we're solving. Everybody's just kind of throwing these crude models in different ways at different problems and they're solving them,

but maybe not in the most elegant way or ways that are really going to push the field forward. I like to learn things because I like to learn probably a million different things and jump in a different bunch of different ways. I do like to learn how to apply them and I like to understand why they work, but I don't need to know them maybe at the mathematical level. And I'm okay with that. I haven't gone for a PhD and I probably won't just because I don't think that I'm the depth person. I really like to have a really broad area of knowledge. And so, but understanding why something works is very important to me.

So even if I'm just, it's a broad area, it has to be a little bit deep because otherwise just that surface area knowledge doesn't mean enough to me. I always need to cantilever it off of something I already know. I can't just learn in a vacuum. But let's get this dry for smart on. Sorry. Clearly I need to cantilever it off of something i already know i can't just learn in a vacuum but let's let's get this to try for smart on sorry clearly i need to explain this so uh the first level is novice where you have a rigid adherence to rules and as they're taught this is the memorization stage this is the i don't understand. I don't have any judgment about it. I can follow these rules. And then there's the advanced beginner, where you start seeing things around you.

If you think about driving a car, the novice knows where the pedal is and the steering wheel and knows what those things do. But putting them on the freeway is a terrible idea. Because there's just too much extraneous information. The advanced beginner starts to separate these things, but everything is treated with equal importance. So in the car idea, as you drive, the car coming at you is treated with the same importance as the scroll running in front of you because you're just parsing too much information and you're only an advanced beginner. And then there's the next stage, which is the competent. And that's,

what is this? Coping with crowdedness, multiple activities, accumulation of information, and ideas of how actions relate to goals. Now, a lot of people stop here, right? I mean, we don't have to go beyond competence. You understand the rules. You can formulate some ideas for other people. You can do planning. Then there's the next step, four, proficiency, which is the holistic view of the situation.

I feel like that's kind of where you're saying you like to be. Because you can prioritize the aspects, you can understand what's different from normal, and you can adapt to the situation. And then there's the final level, which is expert, which is you don't have rules. You don't have guidelines. You just know what's going on. It's more intuitive, but you can also explain it entirely. And you can use analytical approaches like mathematics in order to address new situations and problems. Yeah. finish term one of self-driving car as a machine learning and computer vision person? Where am I on there? So I think most of ours actually fall into the advanced beginner phase. And I think that that's the same with university, as far as at least a bachelor's degree. You've learned the

tools you need, you've learned how to apply them to solve certain problems. But you've only seen a limited number of problems. Given something outside of what you've seen before, you have a hard time maybe applying it to a new situation. And I think most jobs out of college and everything, and out of our nano degree programs as well, really set you up for that competency. Like you're going to get into the job, you're going to learn how to solve different problems, solve these companies problems, and you're really starting to build that.

So that's where I think most of our programs fall for the job readiness is you've been exposed to this stuff, you understand it, and you know the nuances, but you haven't applied it to a lot of different problems yet to really be more competent in that. That makes sense. And that works with what I felt. To some extent, I was sort of considering a graduate degree. And then I ended up with the audacity class. And it wasn't very graduate degree-ish. Because a graduate degree is trying to teach you to be an expert.

And it's trying to teach you to be an expert the long way. For Udacity, I felt like I could become an advanced beginner or even competent in a number of small subjects. But I wasn't necessarily going to be an expert in any of these subjects, which breadth is a great thing. That means you can put a lot of things together. You might not build things from scratch, but given where machine learning is right now, putting them together is pretty cool. Yeah.

And I don't think that our goal is to teach you to be at that expert level. Those expert levels are few and far between, I would say, in people. And usually it's from years and years of experience dealing with so many different problems. And I'd love if we could teach that on an online course. I don't know how, you know, realistic that would be. Um, but a lot of our audience is looking to grow their skills in their current career or they're looking to transition careers. And so especially for those people looking to transition into a new career, like say self-driving cars, I think that

advanced beginner to competent areas where they're looking to be, because they're going to try to get that entry level position into self-driving cars. And that's what your college graduate in computer science with like maybe computer vision background would be looking for as well. Masters, you kind of take another step towards that, you know, competency and a little bit past that. And then PhD, obviously, you're learning to be an expert. But as you go down, all of these things become more narrow. So with the self-driving car program, it's very, very broad.

Our robotics program is the same way. It covers the breadth of the field and not like one vertical. So computer vision in self-driving cars and perception is an entire vertical in self-driving cars. And you can just do that without doing path planning, controls, common filters, all of that. But that's a more of a specialization. And so we are looking at how we can maybe go deeper into some areas. But for now, you know, it's good to provide that breath for people to really find out if you want, if you like self-driving cars, what do you like about it?

Yeah, the survey is very important to figuring out if you want to go vertical, where do you want to go vertical? There are a lot of different interesting problems. And it's easy to get stuck early. And then you realize halfway through your PhD program, wow, this wasn't what I meant to do. So that's, yeah, it is important to try for at least some breadth. Yeah. I mean, one of my favorite stories to tell is, I mean, when I took computer engineering or when I chose computer engineering in college, I chose it because it was more of the hardware of computers. And I thought that was going to be taking computers apart, quickly realized it's not what it was. And luckily I love the topic, but, um, nanotechnology seemed like a

really cool idea to me. And so we had a really state-of-the-art nanotechnology lab and the professor offered to show me around and we went, I saw everything, saw this like gold evaporator. And it's like, this is all really cool stuff. I don't want to do anything of it. Yeah. I've had that experience. Yeah. It's like, I love that. I was like, this is really, this is going to enable so much stuff. And I really like what's going on here, but it's not for me. And I luckily came across robotics. And as soon as I started playing with robots and delving in more with robots, it was like, yep, this is what I want to do. And so like looking back now, it's like, why wasn't I always

playing with robots? But you know, it's not, when I was in high school, it wasn't like it is now, where there were so many programs promoting all of that stuff. I didn't even have a programming class in high school. And so really stumbling upon that and finding what I loved really helped me attach to that and move forward with it. But finding out, you know, this big field of computer engineering, all the different areas and finding out what I didn't want to do and what I actually like to do is actually a very important piece for me. Are you familiar with the idea of trunk skills versus leaf skills? I can assume what those mean. as me well so i'll i'll monologue for a second here um a trunk skill is is something you're you have a lot of breadth and a lot of depth and breadth and leaf is more um specifics uh going back to driving a car accelerator uh and how the steering wheel works would be a trunk. You know how that works.

Trying to find where the windshield wiper is in this particular model of this particular car is definitely leaf skill. Yeah. So given that, can you come up with some leaf and trunk skills just for chats? Sure. I mean, just in general, as an engineer, I think you learn a lot

of leaf skills in school and some trunk skills, but what you really come out of with the trunk skill is how to problem solve. I think that's the best thing you get out of an engineering school, because whenever you go to a company, they're going to teach you how to do things their way, teach you how to work on their product, teach you how to use their software or something. But you've one thing you've done throughout your whole college experience, or even as you go through our programs, is you're solving problems, and you're learning how to solve these problems. And, you know, some of them are difficult. And that's fine, because that makes you work through them. And to me, that's like the core trunk skill of an engineer is problem solving, being able

to think through something in a systematic way and break it down and really figure out, you know, what to do with it to really start. I guess if there's a branch skill on here to really start doing the more of the branch pieces and building up more of the core in your domain. So, um, I spent last night playing around with SolidWorks, and I'm not a mechanical engineer, but I understand enough about how it works, and I've been around engineering enough

to kind of get my hands dirty with a little bit of direction. I can do tutorials, or have a friend tell me how to do a few things, and I can go a long way. And so that's, I think, what you're trying to aim for through those trunk skills, is now when you get a leaf out here, how, how can quickly can you make that skill, you know, relevant and use that skill or learn that skill? So in the case of your car, how quickly can you find the windshield wiper?

Well, you've been in so many cars, you know how to drive cars, you've done other windshield wipers. So based on the manufacturers of cars, you've been at, you know, it's probably here, it's over here. And, you know, that should be, that's, I think what the goal is with that kind of, with that kind of model, at least for me. Yeah. And learning how to learn is definitely a trunk skill, the persistence and the resilience you need to make that work. I think that can't be understated. Remembering how to do a special git rebase command is maybe a leaf skill. Sure. I tend to put a lot of mathematics into trunk skills

because they tend to do things across different areas. As you mentioned with signals, I remember I took signals and I fell in love. Like I just, like they said Fourier and I'm like, oh, that's how the world really works. And it was completely gobsmacked by this idea. And I applied it to everything. I mean, I think there was one day I was in the cafeteria applying it to like peas, because it was such a concept that resonated with me. And so now I like look for that all the time. And yet I don't use it very often in my actual job. It's kind of sad. Yeah. and now it's moved out to leaf. And I sometimes have to look up what I want to find, how to use something. You mentioned that earlier about looking up things

and how YouTube can be used as an external memory. I'm going to look this up, I'm going to use it, and when I need it again, I'm going to look it up again. I find myself doing that with some of my audacity uh uh code that I don't quite remember how to do this and so I want to remember and I I go back and I look it up yep how do we move past that how do we move these look it up skills to use it every day? Well, I think you can move, obviously you can move it in that way by doing it more often, using it more often, solving problems with it more.

I actually signed up to make sure my soldering skills stayed sharp. The Bold Port Club and everything. I think you had Sarah on your show a while ago. And when I heard that, I went and checked it out. And since then, I've purchased every one of his projects. Some of them are still in the box because I bought all the backlog of them. I have a small collection that I haven't started, yes. Nice.

But I wonder if it's also a generational thing. If you look at older engineers, older engineers really know the depth of what they had learned at one point. And when you try to teach them all these new different things, they don't, they're not as good as applying the new skills and learning like 20 different skills, but they really know the core basics of, you know, some discipline in engineering or something of whatever was in their background. Whereas newer engineers tend to go less deep like that. And they tend to learn a lot of different things because we can look things up. We've learned, we've become in a generation that has had the internet.

We didn't have to peruse through books. We didn't have to read the books to just find it. We could just Google, you know, control F. And I don't know if there's one reason or one of those that's better than the other. I actually think that it's really a combination of those that really help out companies. When you have somebody who understands all the small nuances through years and years of experiences and weeding through the details, and you have somebody that can understand and apply those details to new concepts. So I don't know if it's, hopefully we can still keep learning that deep, but I have a feeling the way our society is in the way that people are raised now with Google this, Control F this, information at your fingertips, there's less of

a need to have to memorize all those certain formulas maybe. I know teachers always in the past, like, well, you won't always have a calculator with you. Well, now you kind of do, and now you don't even have, you can probably just say something. I think I asked one of my home devices what an integral of something was, and it spit back out the answer, and I was like, man, this is crazy. I know where I fall in that divide right now, but I probably am wrong. But I do feel like we're losing something with the latter. And it sets up a source of conflict. And I've seen it within companies where, yeah, you have younger engineers and they'll come up with some idea that's maybe smart, but not super deep.

And so they haven't figured out the implications of what it means. And then there'll be a conflict with a more senior engineer who has seen things before and uh and so it's like okay how do i both impart and you can tell which one i am in the story but really how do i impart some wisdom while also while also uh giving consideration to a new idea or what seems like a new idea. Yeah. And I completely agree with you. I think, like I said, there's value in both of these crazy new ideas, this ability to learn so many things and grow this breadth. And there's value in that deep knowledge. And I think it's great that we're in a society right now where we have some of that mixture and it does cause conflict, but I think it's great that we're in a society right now where we have some of that mixture and it does cause conflict.

But I think it's great that we have that. As we go on, if people continue with just the breadth, we're going to lose that depth. And I think that is, you know, a detriment. I don't know. I mean, I see what you're saying, but is this just youth versus age? Could be. I mean, I remember being young and stupid. I remember thinking I knew everything and putting things together in ways I thought were genius and having the old engineer just sitting there kind of chuckling.

But you didn't have YouTube. But I didn't have YouTube. And I, you know there there there's that um and i feel like i have learned so much that they will too but i think eventually yeah well i think even you look at your college classes how students do work now you get the digital book and when you're looking for the answers, you control F versus, you know, back when there was none of that, you just, you had the book, you had to read through the book. You weren't assigned to read the book maybe, but you, you could maybe

like peruse a little bit, but you'd at least find the area and you'd start reading through these paragraphs to find your answer. And just that simple skill of like, you know, having to do that over and over, read this, maybe you read that section before, but you're looking for a new answer. That makes you, I think, that's going to make you deeper and understand it better. And you're going to see it more often. Seeing the surrounding information. Yeah. Yeah. It's one of the things I have lost with Amazon instead of going more to the library is the books on the shelf next to the book I was looking for. It's amazing how important those other books can be. And it's the same thing.

You're looking for the integral of Sinex and, oh, look, there are a whole bunch of these. Maybe it wasn't Sinex and, Oh, look, there are a whole bunch of these. Maybe it wasn't Sinex. I need it. Uh, yeah. On the other hand, on the side of maybe we don't need to always know all of this stuff. Information has a half-life.

Oh, it does. Yeah. Um, and it's like, not math. Yeah. Yeah. Yeah.

It does. It does. Math does. No, no, no. But most facts, of all the things you know, 50% of them will be wrong in 20 years. Not math. It depends on the math. I mean, you know, was Fermat's last theorem unsolvable or not? That's not a fact that changed.

That was a question that was unanswered. True. Okay. Fine. Not math. Some math might be, though. Some math might be, but it's not an area where, like computer science, where you might throw out an entire programming language and well, we're not using that anymore. Um,

sure. There are corners of math that are less applicable, but anyway, I'm just being annoying. I wasn't trying to start a conversation. I agree with that. This information, I mean,

computer science has a half-life. Biology has a half-life. Yeah. I'm not even going to touch math because I think there's probably some, some things we could argue about there, but that's not what this show is supposed to be about. All about math. Where was I headed? Why would you learn anything if everything's gone wrong?

If half of everything is wrong? How do you convince people to learn things that are ephemeral? I mean, these machine learning concepts, they're not going to be the same in five years. No, not at all. And actually, one of our most popular programs is deep learning. And deep learning is very, very much in its infancy. And it's changing. I mean, one of the biggest breakthroughs in deep learning is something called GANs,

Generative Adversarial Networks. And it's only been around, Ian Goodfellow created it, but I think it's only been maybe, I don't even know if it's been five years. It's been very short period of time. And it's one of the biggest breakthroughs and there's so much happening around with it. But, you know, that just upset, you know, a whole nother area that, you know, other areas of deep learning that were being used. And who says that tomorrow, somebody is not going to write a

new paper and completely change it out. But that's not changing, stopping people from getting into this field and just wanting to learn more about, well, how do I use deep learning in my project? And so even though deep learning is something that is definitely going to change, we see a huge response of people just wanting to learn it. And so I think when you find something that just people are interested in, it's less about is it going to change versus this is exciting for me to learn now. And then, yeah, later when it changes or something or I want to apply it to robotics or self-driving cars, then I'll take that one. Or, you know, we'll solve new problems. And that's partially why, because information changes so much, that's what we see at Udacity. That's why our programs are like living and breathing.

And they're constantly being iterated on. Because things change. Even robotics. Robotics, I applied, I guess, deep learning and everything in robotics is very new, but you look at like reinforcement learning and it has huge powers and capabilities to do so much of this stuff. And we decided to teach that in our robotics term two course, because it's such a popular topic and it's just, you know, scratching the surface of its applications, but it's exciting. It's new. And companies really want to see some of those new skills because they have people that

have done other things. They don't have somebody that can do that necessarily. Maybe they'll learn it, but they're also looking to hire people with those skills that maybe they don't have in their own, you know, company right now. And it's always easier to learn the diff. Yeah. Even if half of all of the facts I know are wrong, half of them aren't. And so, it's faster for me to learn the half that are wrong than to learn all of the new ones, which, again, won't be wrong soon. Going back to books, I like books. It might come as a huge shock that I like books. You don't have books with your classes. We don't.

One of the things that we did on one of Sebastian's oldest courses, at least the first course that I took, which was, it's now called AI for Robotics. It's one of our free ones, but it started off as like intro to programming a car. We had notes. And so everything that was done in the course basically got copied into a nice outline. So you had these notes.

And back when I was in college and I was taking this course, I actually printed it out. And I highlighted it and I wrote over it as I went through it. And I think that's really, I really like that. We've since gotten away from that. I don't know if there was a conscious decision as to why or if maybe they just didn't do it. But it is an interesting area to think about having something physical that you can either refer to later. And we always have the classroom. If you graduate from our classrooms, you keep access to the content and everything.

So you can go back into the classroom, but it's not the same as being able to write on something, you know, dog your certain pages, you know, sticky note to really flip back as quick. I guess that's part of the problem with an online course and doing it worldwide is then we have to ship books worldwide. Oh, I'm okay with digital books. Okay. I still highlight those and okay and can note on

them and and i still will page through to figure out what i wanted maybe i'll search for it but i do tend to read around what i wanted um but that has been hard for me because i i go back and i don't know where i'm going back to and i end up having to shuffle through many, many lessons to figure out what was that thing I meant to go back for. Yeah. And it's a good point. Okay. Let's see. I feel like I've asked all these questions, but I have totally gotten offline on my outline.

AI for robotics. That one sounded really cool. But I have taken the first term of self-driving car and about half the second term. Some of your modules are reused into different formulations. Yeah. Should I take AI for robotics or not? So I'm assuming, we changed the name on our robotics program recently

and now it's a robotic software engineer to kind of line up with what we're teaching in the course. But you're right, there's controls, there's localization, there's some perception stuff that all gets covered in self-driving cars and is also in robotics. And I'm okay with our courses like talking over the same topics. What I've been very adamant with the programs that I'm in charge of is that the projects are different. And so to me, the value is great. You've, you've taken, you know, the first term of self-driving car, you've learned some computer vision and deep learning, and you've solved problems around them. That's awesome. But you're not an expert as we were talking about, you know, the different levels,

you're not even at this like, you know, competent level of, you know, learning yet. So doing more projects, hearing it explained in a different way, I think only benefits you, especially if you're early on, you're really trying to learn these topics, or you want to learn more about them in a different way. And so that's really where we focus that even if you've done it in self-driving car, great. If you do it in robotics, you'll be solving a different problem and using similar techniques, but not exactly the same techniques. If someone wants to try Udacity, but isn't ready to plunk down the money, what classes would you suggest?

It really depends on your interest. Well, let's assume they're interested in listening to a show about embedded software. That's fair. Actually, I think we do have a free embedded engineering one. I haven't gone through it, but it's one of our older free catalog. But we also have some courses or what we're starting to do is the free previews. So if you look at like our intro to self-driving car robotics as well, and self-driving car, we haven't put it on our main page yet, but we do have a free preview of

it. We're trying to offer these free previews so you can just see if you like it. We do have many other courses. The AI for Robotics, which is, again, the renamed course that Sebastian did many years ago. That's still a really good course. It's in the old image form quizzes where you see Sebastian's handwriting and there's bubble pop-ups at the end of the video and you select the answer. So it's not the same type of projects and the same kind of quality you're going to see in the classroom because it is much older,

but it gives you an idea of like learning online and seeing the topics that you would learn and how things are explained and seeing if you like that. And he is a fantastic instructor. Sebastian is great. You and I have talked a little bit that I don't like some of the other instructors. But he always makes you feel like you understand it. And you go away and try to implement it.

That isn't always true. But the concepts, the intuitive concepts, he does a really great job with. So I agree. AI for robotics is on my list, even though i've taken parts of the self-driving car just because it's mostly him yeah uh are there any other classes you would recommend for people to what i mean the previews um is that i get access for a little while or i get access to the first term or how does that work it works a little bit differently in different ones we're kind of trying to come up with a standard way to offer the free previews.

But the car one is up to the first project you get access to. The robotics one is kind of the same, where you get access up to the first project. The intro to self-driving car, we tried to do something where you just get a little bit of a preview of each of the sections. So you get to see a little bit of different content, different instructors, and get kind of a good overview of it. All right. That makes sense. Let's see. What class would you want to take next from Udacity? Personally? So I actually just signed up for a design sprint nanodegree, which is a non-technical one. But it's because as a product lead, there are certain skills coming from the engineering world and moving into the product world. There's skills that I personally think

that I lack. And so I'm trying to gain more knowledge on those. So that one's a short one. It teaches you how to do this, basically a four-day long sprint of the design of a product or the design of something. And I found myself doing it at work really quickly, having to move to the digital form because it's, you write a bunch of sticky notes and you're sticking them everywhere. And I covered the girl's desk next to me one night at work. And then I realized this was the wrong nanodegree to take over Christmas travel because I'm not going to have all my sticky notes. So I moved to the digital version where I'm drawing things and making notes that way. And so I can still do the same thing. But since I am working, it's on my own, the design sprint, you do work on a team usually,

but on the nano degree, they teach you, they give you kind of, here's what your team said, and then you get your input and everything. So that's what I'm doing. Um, beyond that, technically I do, I started off as a self-driving car student when it first opened up, actually, before I was with Udacity. At one point, I should complete that. Where are you? Probably right about where you are, actually, about middle of term two. What happened with me was I participated in one of the self-driving car races that Udacity hosted.

That took up a bit of my time. And then Udacity hired me, and I had to move to California. And that took up a lot more time. And then the job's been busy. Life got in the way. Life got in the way. But then the robotics one as well. And then we have a new flying car one coming out.

So I'm very interested in that as well. That's bonkers. Yeah. A lot of companies will pay for students, for employees to take your classes, won't they? It depends on the employer. So I worked at Lockheed before I came over to Udacity. And since I was in the autonomous systems division, I brought up like, hey, there's this course. I think it would really help out.

Will you pay for it? And Lockheed's policy is degree-seeking education. However, since my management did see the value in it, they said, yeah, we'll pay it out of our budget instead of the normal Lockheed education budget. So, you know, even at a big company like that, that, you know, my team was willing to pay for that type of education. And so it really depends on the company, but the skills you get are relevant to those areas. And a lot of companies, if you go in and you say, I want this, give them a couple months to say yes, and they might. They might find the budget for it, whether it comes out of going to

a conference or whether it comes out of your boss's discretionary funds. Wanting to learn is pretty important. It is. And I think, you know, when you look at this compared to other professional training, it's extremely cheap. I mean, when I went to Lockheed and I said, yeah, it's going to be $800 a term. It takes about nine months. And so it's a total of $2,400. They're used to hearing, you know, $3,000 to $5,000 for a week of professional training. And so for them, it was like, yeah, that's a no brainer. You want to sign up for that? That's fine. And I have a friend in Germany who the only reason he knew about Udacity and was in their self-driving car program is because when he got hired, his employer said, you should take

this. We're going to pay for you. We want you to take this. And so he did and he loves it. And if it's a struggle and you really want it, you can always offer to pay for half. That shows a huge amount of commitment on your part and still means you don't have to pay for all of it. So, yeah. Yeah. shows a huge amount of commitment on your part and still means you don't have to pay for all of it so yeah yeah and you have a few uh scholarships not often but for a few of your programs we do yeah

so we reach out to some of our partners sometimes to offer different scholarships so with self-driving cars uh specifically the intro to self-driving car program lyft partnered up with us to offer you know relationship or sorry to uh, scholarships to people going into the intro to self-driving car program. So they, you know, offered 400 of them. We've awarded a hundred of them. They're kind of on a rolling basis, um, that we're going to offer them at different times, but yeah, there's a, and then the Google just offered like 50,000 for, uh, Android. So if you want to be an Android developer, there's scholarships for that. So we are trying to get more into that.

And what you see there is we really want employers who value the skills. And so we work really closely with employers to hire our students. But if they're also throwing money to get the students to take it, there's obviously some incentive for them. And they believe in that program.

And then when they have those skills, hopefully you'll go work for them. Maybe intro to self-driving car. That's the one I helped out on. Yes. Yeah. Okay. Go take that one. But when you get to the part where I talk, just close your eyes.

It'll be way more natural that way. Do the employers get my information? Yes. As long as you're, you want that. Um, we do have ways to opt out of that because if you're working already with one of our partners and maybe you don't want to find a job, then we don't want to send your name to a list of prospective employers and your employer going to be like, are you looking for a job?

Um, so we do have that And we do offer also career services. And so resume reviews, LinkedIn reviews, GitHub reviews, that's all part of the package when you come to Udacity that you get as not just the technical skills, but we offer all these services as well. If you sign up for one of the nanodegrees. Yes. This is not just everybody gets your resume checked.

Correct. I'm sure there are services for that, but they need to make enough money to pay you. Sure. Yep. So the teaching part, we talked some about the quizzes, which are formulaic enough that they're computer graded. And we've talked about the projects where the humans grade them. How do you learn how to teach? How do you learn how to build these projects

and how to decide what's going to be on the quiz? That's a great question. Personally, I don't build the quizzes. I look to our curriculum leads and to our content developers for that. When I look to hire them and when I look to put them on the team, it's great. You have the technical skills, Cool. I can check that box off really quick.

But how do you feel about education? How do you feel about teaching somebody something? Have you ever taught somebody something? And so a lot of times in interviews, we ask, you know, well, teach me something. And that's a good one. Yeah yeah so we sit there and we don't care what the topic is it can be technical non-technical but we want to hear how you break down you know a topic and actually teach it to me on this on the fly and so to me you can learn through some education experience um and some people just are naturally good at explaining something. Sebastian, you're saying, you know, he's a great teacher and he is. And he's also a professor at Stanford, probably for a reason, not just being extremely smart, but he's really good at explaining, you know, the topics that he teaches.

And then do you see this survival of fitness thing happening with your education where I noticed on one of the classes that I took, you had a pointer to a YouTube video. And then on another one, there was a pointer to a Khan Academy lesson set. And so it isn't just you teaching things, it's you finding the best ways to, that things are taught. Yeah. So we obviously can't teach everything. We have limited hours in the day, but there are certain core materials that if we're going to, you know, get you ready for self-driving car and you're like, you're not ready or intro to self-driving car is a good way because, you know, maybe you don't have the skills even to get it, like make you successful in intro to self-driving car. How do we get you ready for that? And if we don't have the material in house, we're going to point

you in a direction to really help you out with that. Um, we see ourselves as, you know, one of the best, highest quality online educations around. I mean, there's a lot of other players in the field, but the way we do it and the way we kind of cater to our students and the services we offer, we see having a little bit more of that personal touch and really helping you get through it and be successful at it. And hopefully teach it in a really good, relevant way as well. That makes sense. I mean, you do have to keep it relevant. I am stuck on term two of self-driving car. I kind of dropped it because I wanted

the information and then I said, I don't like how this is being taught. It was entirely about how it was being taught. Because I care about that, it annoys me more than not only my not getting the information I want. These people are doing it wrong. And then I went off to a book and I couldn't penetrate the book. And that was very hard for me. How are you going to get that feeling?

Talk me into finishing this. Yeah. So you're not alone. That is actually feedback around the portion of the course that you're in that we've heard quite a bit. And it's something that, you know, when you sent me an email with some of your feedback on that, it went around, I've probably got about four or five responses to that now. And I've made it very clear, like, this is something we need to fix because it's not, you know, it's not just you, whose opinion I do value quite a bit, but it's also other students. And, you know, when I go and I talk to students, we do a lot of meetups. We invite students to come out. We have different

events and we like to hear, you know, what are your thoughts on it? At least when I go into it, it's great to like socialize with them, but I want to get their feedback. And I don't want to hear that it's great. I want to hear what's wrong with it so we can go back and fix it. And so that's part of what we do. Now we did build self-driving car very, very fast. And as we were, you know, getting it ready, as students were in there, we were, you know, keeping ahead of the students. Just keep a week ahead.

That's all you have to do. And so because of that, sometimes, you know, certain concepts get lost or certain things don't get taught maybe in the best way. And we have gone back. So I was in the first term of self-driving car or the first cohort. And that meant everything had just been, you know, maybe beta testers and then us. And so it was very, very rough at certain points. And so even the experience that you had in term one was very different from the experience that I had. And that's good that we're going back and doing that.

And so we do have a goal to kind of like slow down a little bit and focus on that quality before it goes out the door so that our first students get a good feeling out of it, get a good experience. But there's going to be pain points, and there's going to be areas that we identify where students drop off. And that's where we need to go back, where we need to focus and we need to improve. And so taking the comments you had and actually going back and doing it now, getting past where you're at, um, it gets better. And then term three, the first project's a little long. It takes about two months to kind of go through the material and build up to that first project. But then you're

just a couple steps away from actually putting your code on a car. And since you're here in California, you can actually come up with us and you'll get to see your car, the car run code in person if you want. Yeah, that was a little crazy. Getting to the end of term three, you actually get to put code on Udacity's car, which I'm sure that will work fine. We've actually had it happening. We've had hundreds of students graduate already, and many of them run code on the car and learn lessons from it. If you give it a one for acceleration in the rear world, that means you just gave it full throttle. And then this little parking lot, it's spinning its tires and going. And so it's, it's been exciting to watch some of that. Um, but it's really cool for the students to, you know,

see their car, get the video of their car running and everything. Um, and yeah, I mean, there's nowhere else in the world. Can you even really put code on a self-driving car unless you have one. All right. I'm about out of questions and I think we're about out of time. Chris, do you have anything you want to add before we go? No, I don't think so. All right. Anthony, you knew this was coming.

Do you have any thoughts you'd like to leave us with? I think I'm going to leave you with the same thought almost that I left with the tip, like is be a lifelong learner. Um, things change, uh, skills change. And if you just continuously learn, it's only going to better you.

Uh, and it doesn't have to all be technical, but just constantly learning something new and approaching life with that curiosity is a huge, huge advantage. I think to a lot of people. Cool. Our guest has been Anthony Navarro, Udacity product lead for robotics and self-driving cars.

Thank you for being with us. Thank you. Thank you to Christopher for producing and co-hosting. And, of course, thank you for listening. For our Patreon supporters, thank you for bearing with us as Patreon continues to move the bar. And I do appreciate your support there. And if you have any questions, feel free to email show at embedded.fm or hit the contact link on embedded.fm. I have a quote to leave you with from Robert Frost. Education is the ability to listen to almost anything

without losing your temper or your self-confidence. Embedded is an independently produced radio show that focuses on the many aspects of engineering. It is a production of Logical Elegance, an embedded software consulting company in California. If there are advertisements in the show, we did not put them there and do not receive money from them. At this time, our sponsors are Logical Elegance and listeners like you.