Lenny's Podcast: Product | Career | Growth - AI and product management | Marily Nika (Meta, Google)

Starting point is 00:00:00 There is something called the shiny object trap, and I'm always telling people, hey, don't do AI for the state of doing AI. Make sure there is a problem there. Make sure there is a pain point that needs to be solved in a smart way. Once you have identified what that problem is and what that's very, very high-level solution is, then reach out and try to figure out how to actually implement it. Welcome to Lenny's podcast where I interview world-class product leaders and growth experts to learn from their hard-win experiences building and scaling today's most successful companies. Today, my guest is Merrily Nika. Merrily teaches the most popular course on Maven on AI and product management. She's currently product lead at Meta, focusing on Metaverse, avatars, and identity.

Starting point is 00:00:49 Prior to Meta, she was at Google for over eight years, working on Google Glass, computer vision, and machine learning around speech recognition. In our conversation, we touch on what KEMs should be paying attention to when it comes to what's happening in AI. We talk about a bunch of resources that'll help you get started in the world of AI, how AI tools available today can already help you do your job better as a PM. We also get relatively technical into what exactly is a model, how our models train, all kinds of fun stuff like that.

Starting point is 00:01:19 Enjoy this conversation with Marilee Nika after a short word from our wonderful sponsors. This episode is brought to you by Amplitude. If you're setting up your analytics stack but not using Amplitude, what are you doing? Anyone can sell you analytics while Amplitude unlocks the power of your product and guides you every step of the way. Get the right data, ask the right questions, get the right answers, and make growth happen. To get started with Amplitude for free, visit Amplitude.com.

Starting point is 00:01:49 Amplitude, power to your products. This episode is brought to you by Epo. Epo is a next generation AB testing platform built by Airbnb alumni for modern growth teams. Companies like Netlify, contentful, and cameo rely on Epo to power their experiments. Wherever you work, running experiments is increasingly essential, but there are no commercial tools that integrate with a modern growth team stack.

Starting point is 00:02:15 This leads to wasted time building internal tools or trying to run your experiments through a plunky marketing tool. When I was at Airbnb, one of the things that I loved about our experimentation platform was being able to easily slice results by device, by country, and by user stage. Epo does all that and more, delivering results quickly, avoiding annoying prolonged analytic cycles, and helping you easily get to the root cause of any issue you discover. Epo lets you go beyond basic click-through metrics, and instead use your North Star metrics, like activation, retention, subscriptions, and payments.

Starting point is 00:02:49 and Epo supports tests on the front end, the back end, email marketing, and even machine learning clients. Check out Epo at getepo.com, get EPPO.com, and 10X your experiment velocity. Marily, welcome to the podcast. Thank you. Hello. Thank you for having me. It's very much my pleasure. We've interacted a little bit on Twitter. We've never actually talked before just right now. I've seen your course just kind of all over the place. on AI and PM. And so I just thought it'd be really fun to have you on and help us all understand what the hell is happening in AI and especially AI on product. So thanks again for being here.

Starting point is 00:03:33 Yes, thank you. I'm really excited. I would love your help as a former full-time PM slash everyone listening that is a current PM to help us understand what is going on with AI and product. Tech in general and tools in general move really fast. You know, if you're trying to pay attention to like what's happening. It's really hard to stay up to date on where things are going. And it feels especially hard in AI. It feels like there's just something coming out every day. And so I have a bunch of questions along these lines. The first is just like what media do you pay attention to to stand top of what's happening and what's new and what's interesting in the world of AI and machine learning. As you know very well, subscribing to newsletters is something

Starting point is 00:04:14 that's really, really impactful. And of course, I subscribe to your newsletter. But I'm a big, big, big fan of the download by MIT Technology Review or TLDR. And they're not necessarily AI-centric, but what I'm advocating for and what I'm telling people is that in the future, everything will be AI by default. So even if you have something that's technology focused, you will see a lot of AI starting to get sprinkled in there. I want to follow up in what you just said there, but maybe we'll save it a little bit. Maybe going in a different direction first, what do you think is overhyped in the space of

Starting point is 00:04:48 AI right now? what do you think is under-hyped and under-valued? I would like to discuss Chatsubit, which is both under-hite and over-hyped at the same time. I was reading this article this morning where there are writers complaining and they're very, very fearful, and they think, oh, writing online is going to die. Everything we're studying for is going to be in place. They're going to take our jobs and so on. And I'm just like, no, no, no, no.

Starting point is 00:05:14 Chargipity and technology is enhancing our work. It's enhancing us. It does not spill from us. So that's what comes across right now. And there are other things that are under hype. Like, obviously, Chaudibati is amazing. I'm using it day to day. But there are other things AI can do in an amazing manner.

Starting point is 00:05:33 Like, I was reading a research article the other day, and said that AI can now detect lights. So lie detection, whether it is for security reasons or at work or anything like that, is now possible. So I encourage people to go to these newsletes. and go through these online blogs, 10 cranes, and so on, and just read what's happening.

Starting point is 00:05:54 It's not all about chaty-D-D. There's more. There's more about the AI, but you should read about. You mentioned that you use chat GPT in your work life. Talk about that. What are you actually using it for? Even when I'm at work,

Starting point is 00:06:05 and I'm trying to come up with a nice mission statement. Right? When we're PMs, you come up with mission statements, it's just crucial part. And it's where the core begins. You want to get people excited. You want to get people excited. inspired. There is nothing I can write that's going to be as willing as what ChatsyPD would

Starting point is 00:06:23 write. So what I do is I literally go to Chatsy B2 and I say, rewrite this mission statement from me. And it just, even first try I produced something which is fantastic. So that number two, it helps me create user segments in a fantastic way. It will think of user segments that your mind wouldn't even go there. Like it's just within color and it will provide the motivations. they will provide the pinpoints, and you just come up with ideas as you read it. And then the last thing that it does is it provides ideas for you that are AI enhanced. So I just use a beta day, even pros my day-to-day workflow,

Starting point is 00:07:02 but I'm not making it do my job for me. I'm asking it after I have already had a mission in my head and what it is I want to do. So the way you're approaching it is you just put in, come up with a better mission statement then, and then you give it your version of the mission statement. Exactly. Interesting. And you're saying that that comes up with a better mission statement than the one you had.

Starting point is 00:07:24 It's better because the mission statement is going to be read by all disciplines. It's not just going to be read by PNs that already have a lot of context and understand. It's going to be read by leadership, by junior people, by stakeholders, by other departments, by competitors. And he needed to be on print and in the words that are meant to be understood by everyone. you're going to keep going to understand it. And they would get inspired by it as well. And then you also said to use it for personas. How do you actually frame that prompt with JetGPT?

Starting point is 00:07:55 Let's say you're working for a specific product area and you know you want to create some fitness band. So you would say something like, who would be interested in a fitness band that doesn't have a screen? And it will provide a bulleted list of people like, hey, young professionals that they're interested in but don't have enough time, people that do not want to charge their wearables, everything. Then the list goes on.

Starting point is 00:08:21 It's just fantastic. You were talking about how you think the future of AI is it's the default and is what you mean there that it's basically baked into every product we use and it helps the user do better things. It helps the product work better. Is that what you mean or is it something else? I believe that all product managers will be AI product managers in the future. And this is because we see all products needing to have a personalized experience, a recommender system that is actually good.

Starting point is 00:08:52 I mean, you cannot watch Netflix. You cannot even watch a movie without needing that. After you watch White Lotus or like Stranger Things, you will want something similar to watch. You're not going to want like a romantic thing to be suggested or recommended to you, right? Also, automation is another thing. We need to keep improving on society. We need to keep making technological and things. past friends, you're not going to be able to do that if you don't have an AI-centric view in every

Starting point is 00:09:17 sector that you're working on. When you say that every PM will be an AI PM, is you're thinking that you'll be using AI tools in your job as a PM, or that you'll be building AI into everything you're building with. How do you think about that? I think it's that you will need to get comfortable with having a partner that's a research scientist. And you don't know. need to understand that these people can produce a smart model they'll be able to do some automation, some personalization, some recommendations, some on. In a lot of people, really uncomfortable with that. A lot of people don't know how to approach the researchers. A lot of people don't like the uncertainty that research has. A lot of PMS are very, very used to,

Starting point is 00:10:05 okay, I'm going to do this, I'm on lunch, I'm going to do this, I'm on lunch. Whereas when you're working with research, it's more like, we're going to try this. And then you need year, if it doesn't work out, we're going to shut everything down and people have completed food so I feel that if people get more used to uncertainty and research things are going to be good in the end for them. I thought you were comparing Chad GBT as like a researcher you're working with but you're actually saying people will have PhD researchers on their teams helping them build models into their product to make their product better is that is that what you're saying?

Starting point is 00:10:43 Correct. This is exactly me about it. And from a product perspective, like in imagining like three bubbles in my head. So you want to find the intersection of something that's desirable by users, something that is going to be a viable business, and something that is going to be feasible from a research scientist and technical perspective. And then when you have that, it's just going to be a fantastic product for a domestic lunch that you can run with. So yeah, whenever I say researcher, I mean, research scientists, that can produce an AI emotional learning model.

Starting point is 00:11:15 Wow. Didn't think about how every cross-functional team might end up with a research scientist. Interesting. Interesting. For PMs who are curious about learning how to do this stuff, what are a couple things that PMs today, who have no experience with AI, what can they do to start learning, how to build AI tooling into their products, understand what hell is happening in the space of AI? This is a good question. And I guess the message don't want to pass. is you shouldn't be overwhelmed by these technologies if you don't have a technical background because you can learn these things. And as a PN, you'll never need to actually train or code.

Starting point is 00:11:54 Also, even if you want to trade, there are no code approaches for training models. But to answer the question, if you're working on any product, you can always sprinkle in a smarter feature. So you can make it more secure. You can personal lines it. you can enhance it with fraud detection. You can make it more ethical. If it's healthcare, you can make it faster. You can make it more accurate. If it's shopping, you can create better recommendations.

Starting point is 00:12:22 Basically, anything where you can get data behind the behavior of the users can be improved with AI. So I guess it's all about changing the mindset of PMs, taking its step back and just thinking about, okay, I have all these data that's just lying and sitting around. what is it that they can do with it? I've been meeting PMs that said, oh, we don't have any, we're not collecting any data, we don't have any dashboards. So even that is a huge first step towards AI. And then just start thinking about it,

Starting point is 00:12:51 what you could do, just hire and get a business science intern and just see what they are going to. There's just so much people can do. So say you want to start investing in some sort of model, some sort of AI within your team, you're saying maybe hire data scientists who can help you start to build something that you can start integrating. Is that your advice on the first step of once you start,

Starting point is 00:13:15 you want to start getting serious about building some sort of AI component? There is something called the shiny object trap, and I'm always telling people, hey, don't do AI for the sake of doing AI. Make sure there is a problem there. Make sure there is a pinpoint that needs to be solved in a smart way. Once you have identified what that problem is and what that's very, very, very, high-level solution is, then reach out and try to figure out how to actually implement it. And there's a definition I like giving.

Starting point is 00:13:48 I usually say that a generalist PM helps their team and their company build and ship the right product. But the AIPM helps their team of company solve the right problem. So if you want to get into AIPM, figure out what the problem is that you will get the data science to create a moment for solving. But there needs to be a problem. there needs to be audience, there needs to be a user and a thing going for it. What are signs that AI may not be a good approach to solving a problem?

Starting point is 00:14:17 You said that, you know, and this happened on a lot of my teams. Oh, we're going to build a really cool model. It's going to do something really smart in this case. And it often ended up being a very low ROI investment and took like six months to a year before you even knew what the hell was happening. Do you have any thoughts on signs that maybe this isn't a place you should be putting a lot of time into AI? versus like this is definitely an opportunity. Yes, we should invest a lot of time into this.

Starting point is 00:14:43 Don't do it for your MVP. It makes zero sense. Do not waste time of data scientists that can train models with using powerful machines that are going to take weeks to train. This is because if you have an MVP and you just want to get buy-in for an ADOAWR feature that may use AI in the future, fake it. create a little Figma prototype and just show it some users and just bait what the AI is going to be doing.

Starting point is 00:15:13 So I have a lot of young early stage entrepreneurs who are talking to me and they say, oh, how we should we train this model to do this and that because we want to prove that there is a market. No, do not use AI. You should use AI where you think you already have some data or data from an adjacent product that you feel you can leverage for your own product to create something that's meaningful, recommendation, automation what we talk about. But not for an MVP.

Starting point is 00:15:43 Please, people. This is my life. How much data do you think you need for AI ML to have a chance to contribute? You have a heuristic of, if you have anything less than this, it's not going to work at all? This is a good question,

Starting point is 00:16:00 and it honestly depends. I'm not trying to do. If you're trying to classify, If the photo is a cat or a dog, obviously, even if you have, I don't know, like 15, 20, labeled photos, that's going to work. But if you want to create voice recognizers or complicated NLP applications, you're going to need thousands of thousands of data. And this is what's making this not be easy, right?

Starting point is 00:16:22 AI systems are not easy to delve. There is a lifecycle of a machine learning project. And after scoping, you need to figure out, oh, God, how much data do we need? Where do I find this data as well? right? How much data? Sometimes they've seen people synthesizing their own fake data just so that they can have something to train with and test their models. But the exact amount is hard to be on the quite, especially from the end. Like, I'm sure data scientists have a different opinion. Yeah, my guess is most startups are going to have nowhere near enough data to build their old

Starting point is 00:16:57 model and make it something really interesting. So do you have a thought on when it makes sense to try to build your own model, try to train your own GBT type thing versus use something that's already out there, like say GPT or the journey or all those guys. If you are a big tech company and you're offering a service that is going to do like speech recognition or that it's going to have like their own tragedy, you want to use more data and more diverse data to train and retain and train. Because if you don't, then your quality is going to be the same as every other companies. There are agencies that are selling data, packages of data that are ready so that you can get them and train your models.

Starting point is 00:17:47 But the question is, if everyone takes that exact data set, then the quality that every single company is producing is going to be the exact same. So you do want to diversify, you do want to collect your own data. And I guess a good question from my pream perspective. is when is the quality of your product good enough to launch? And that is like a really interesting point because it's totally your responsibility as a PM to decide, okay, the recognition of whether this photo is a cat or a job is good enough for the users. It's like 70% accurate, 80% accurate.

Starting point is 00:18:23 Where is the bar? Where do we launch? And that's why the AIPM rule is so cool because you have problems like that to solve that no one else has kind of tackled before. So it's all on you. We've thrown out these words model and we talk about training models. Do you have a good succinct kind of explanation for what a model is for folks that haven't, you know, that aren't that technical? And then just the general idea of training a model. Like what is a simple way to think of heroes what a model is.

Starting point is 00:18:52 So I have a three-year-old girl and I'm teaching here about life and everything. So I was recently teaching here about the animals. And, you know, you explained things to her once. or twice, like what the mammoth is or a rhino and so on, but you will end up training your kid's brain by repeating the same information again. So you will say, hey, here's what your rhino looks like. Here's what the rhino looks like. Here's what the nautophone looks like. And once you've done this enough times, then your kid will see an animal on the street and they'll be able to rub them eggs and say, oh, yeah, that's like the rhino we're talking about.

Starting point is 00:19:31 this is exactly what a model is. The model is like a kid's brain. It has the ability to take an input, which means it has the ability to take an image and say, oh, I recognize what this is. That looks like a rhino, but I'm 70% sure about this. So it will output the probability of this certainty.

Starting point is 00:19:50 And you said image, but it could be text for, say, in chat, GPT in the future. I imagine video. There's also a voice like whisper. That's an awesome explanation. Basically, it's trying to recreate the human. brain is a nice way of thinking about it. And then training a model. Can you talk about what that it means? The process of training the model, for example, is providing a lot of images that are

Starting point is 00:20:13 legal and say, hey, here's what it looks like. And we're talking about thousands and thousands of data data sets for this. And once you do this, there's a process where the model is just processing this information and it's learning. It's finding patterns through it. And the patterns are not in the form of, oh, if this is gray, then this means this. No, it just learns in a smart way how to identify specific things that we don't even understand. And then it's able to output the probability of whether a photo is going to contain the cat or dog. Just conceptually, what is the output of the training? Is it code that is auto-generated with these decision trees and weights and things like that? Is it a database of weight? Like,

Starting point is 00:20:59 just conceptually, what is the output of a training? meaning that becomes a model. What's the simplest way to think about that? So let's imagine speech. Speech is a great example. For example, I'm talking to a device, which is like a home system, and I say, hey, what is the weather like today? This is going to take my voice and audio and it's going to process it.

Starting point is 00:21:17 And the output is going to be a transcription. So it's literally going to be text that corresponds to what I said to it. Thinking about the stuff you've worked on at Google, at meta, anywhere else you've worked on site projects, even. What are some of the cooler application? of AI, machine learning that you've worked on, contributed to, or even seen that you can talk about. I imagine there's a lot of sensitive stuff too going on. One thing I want to talk about is the team I used to work for Google, which was the ARVR team.

Starting point is 00:21:48 And they were working on an airglass. And actually, they had a video on last year's Google I.O. they were able to have the Google Glass on someone that spoke one language, and then this other person was sound in front of them who spoke a different language. And the glass would take as an input, the audio that came on that other person, and they would transcribe it, it would translate it, and show it on the screen for that person in their language. So we're talking about the ability for this devices to unlock.

Starting point is 00:22:25 the borders of communication. And that is not science fiction. This is what amazing and mind-blowing, there's no science fiction anymore. These things are real, but technology is here. It's just a matter of connecting the pieces to the puzzle in order to see them coming to life.

Starting point is 00:22:42 So I think that one was the most, one of the most impactful things I've ever seen. I remember that demo, and it was pretty incredible. Okay, so thinking a little more broadly, do you think chat GPT, or just say GPT, or GPT5, GPT6. Do you think at some point this will replace product managers, something I see on Twitter.

Starting point is 00:23:02 A lot of people are like, oh, my God, product management's dead. This thing made my product requirements document for me. Or you talked about how makes your mission statement better. Do you think there's a place where PMs aren't necessary anymore? Oh, absolutely not. As I said, like, it makes everything better. If anything, it's going to free out time for me to do other things than are less tedious. For example, I am running so many projects and they all need their P.RD.

Starting point is 00:23:27 And the PIRD have all these areas that are common across, across all of them. If I had a system that can actually write the Tdew stuff for me so that I can focus on the more strategic side of things, that would be incredible. It will make us smarter, if anything, you will unlock new areas of product monuments that we haven't realized that are there. Are there areas, do you think, with your kind of vision of all PMs, will be? be AI PMs, are there areas that you think PMs should invest more skill-wise or areas they should less focus on invest because, say, some machine learning model is going to do that for them? I'd like to see people being less overwhelmed, less intimidated, less afraid to start learning how to code, how to train a little model on their own. This is because even if, you know,

Starting point is 00:24:18 TadGBD or these no-code applications may be able to do this for. us, it gives you a different approach, a different mindset, a different, if you want, confidence to know how things work. And here's a silly example. I was learning how to play the piano when I was young. And when my teacher came in, I was like, oh, I want to learn how to play this cool song. There were some songs that I read. And she said, no, you need to start with classical music. And I just hated it at the time. And I said, why do I have to do this? because she said if you learn the fundamentals and how, you know, where things started and the beginning of music, it's going to help you along the way to create music on your own if you want to. And she went right. Like I just loved it. So it's the same with coding. I encourage people to just take an online course, understand more, get your hands dirty, pair up with someone else that's in the same boat as you, because this is going to give you the skill set to understand how that tool that's going to help you in your day to day was even creating.

Starting point is 00:25:17 in the first place. Instead of blindfoldly, just trust it to do your job. This episode is brought to you by Pando, the always-on employee performance platform. How much do you love the performance review process? Yeah. It's time-consuming, subjective, biased, and there's rarely any transparency. With the rapid shift to distributed work, it's a struggle to create the structure and transparency that you want to help your employees have the highest impact and growth in their careers. Pando is disrupting the old paradigm of performance management, including a continuous employee-centric approach,

Starting point is 00:25:53 so employees stay engaged, see their progression in real-time, and know exactly when and how they can level up. With Pando, managers can leverage competency-based frameworks to effectively coach and develop their teams and align on consistent growth standards, resulting in higher-quality feedback and higher-performing teams. Visit pando.com slash Lenny for more info and get a special discount when you sign up

Starting point is 00:26:16 and reference this podcast. That's pando.com slash lennie. For someone that actually wants to do that and learn to code, which I love that advice, do you have any resources, places that you point people to for learning to code, getting started down that path? It depends on what type of learner you are. There are some people that like to learn offline.

Starting point is 00:26:36 So just go to Coursera, there are so many courses. There is an amazing one, actually. Introduction to AI by Stanford, let them encourage people to take a lot of that. But I know a lot of people don't like, don't have the time, don't have the discipline to actually, you know, take time off or like after work, after they put their kids to sleep to just do it. So if you enjoy learning with others, if you enjoy being part of the team, if you enjoy going through a journey together, then I recommend these resources. So there is something called Career Fundry, which is a fantastic on like a coding school, general assembly, and then coding dojo. I was actually gave it talks

Starting point is 00:27:15 ages ago at the Dojo about Python and all it takes is just a few weeks of your time and passion and just for you to roll up your sleeves and just realize that this is not intimidating and realize the benefits you can get by learning. Awesome. Thanks for sharing those. We'll include links in the show notes. Going back to a PM, trying to become better in AI,

Starting point is 00:27:39 if you think about a PM that's kind of early in their career and wants to become a very strong AI PM. I know you have a whole course about this, which we can talk about now or later, whatever, is easier. What should that PM be doing? We talked a bit about Learn to Code, maybe, start playing with tools. What else do you suggest PMs that want to become really strong AI PMs do now and invest in? So I do have a course that's coming up on February 6th on Maven,

Starting point is 00:28:05 which is for current and aspiring product managers that want to build AI products. but I also have offline recordings. I have the same course on an offline basis on my website. I'd be happy to talk to your hearing you about this. What I feel people should understand is what it takes to manage an AI product. Of course, people are very familiar with the stages of product development in general, but AI product development is different.

Starting point is 00:28:33 As I mentioned before, sometimes you're actually managing the problem and not the product, and you're trying to secure out if there is a problem, that makes sense to be answered by a smart solution. So it's kind of a very interesting and more complicated process than regular product management. So number one, figure out how it differs from general product management. Number two, if you're already at the company that is actually having AI researchers and AI research scientists, I encourage people to just retouch them and shadow them and spend an hour

Starting point is 00:29:09 of their week, just talking to them and experiencing what they're doing. This is going to open your mind. This is going to give you so much context as to what it is and the endless potential that you can identify there. Awesome. And is there anything else you want to share from your course that you think might be interesting to folks? So we talked about why it's awesome to be an AIPN, but I do work aloud that there are a few challenges that people need to be aware of. Number one, and I kind of mentioned it before is the uncertainty. You may have been working on all these incredible research and ideas in hypothesis, but then when you actually train the model,

Starting point is 00:29:50 the results you may be getting may not be optimal, may not be answering the questions or the hypothesis that you actually had in mind. So that's number one. You need to be able to encourage the team throughout this process because you're like the captain of the ship. You need to be the one that's kind of titulating the team. making sure I know what it's going. Number two, you are going to have to be a good point of lot.

Starting point is 00:30:14 You are going to have to change the action. In managing this from a leadership perspective can be tricky and it can be challenging. Number three, we talked about data, but getting good data is hard. Like, you may need to be creative, figure out ways for data collection that you never thought you do. You may get on the street and ask for people to actually contribute data for by he's your dream. You need to be able to and willing to do everything. And the last thing is, from a

Starting point is 00:30:43 career trajectory, usually product managers get ahead the more of the launch. But if you're in a research hour, you're not going to launch as often. So you need to make sure to clarify with hiring managers early on, hey, what does progress

Starting point is 00:30:59 mean? How am I going to get assessed in a research work which is different than what I've been doing so far? So it's challenging, but I always encourage people to flex different muscles, and this is like the zero to one muscle. The other thing is, it's just crucial when it comes to product on that one. This actually is a great segue to a question. I definitely wanted to ask, which is around getting buy-in for investment at a company for ML.

Starting point is 00:31:24 So there's sometimes like all this energy for like a zero-to-one. Let's just try something. Sometimes not, but that maybe that's, maybe there's a two-part question here. Do you have any advice on just getting buy-in for, we want to try something with a amount? It's going to take us six months. to figure out if it's worth the effort, but we think there's something here. And then sometimes there's a lot of energy initially, and then you get some win, like your search ranking is smarter and it's great.

Starting point is 00:31:48 But then maintaining that, having all these really expensive people working on just tweaking this model and continuing to make it smarter and a little more efficient, often it's hard to continue to get buy-in for that sort of team. Do you have any advice on initial kind of buy-in, let's try something here, and then down the road just like keeping a team going, trying to make a team going, trying to make this thing smarter and smarter. People should know that there is an excellent source of inspiration and something that can kind of do risk things, which is adjacent products.

Starting point is 00:32:19 Maybe the company has already launched a product that has been successful, those AI firsts. And whenever I tried to convince leadership about something that I want to do, that's kind of a big bet. I always use examples and I'm like, hey, this seemed crazy at the time. Here's how to work. What I'm proposing is very similar to this crazy thing. And then I propose a little contingency claim, like, hey, if that doesn't work out, here's the rollback plan. Here's kind of the maximum impact it will have done negative way, which is not going to do too much. And you kind of take it all on zero.

Starting point is 00:32:55 And it's interesting because the more you work on this specific company, the more trust you get. And if the culture is such, then failing is going to be welcome. So a lot of companies that welcome paying because you can just go ahead and do this road. Do you tell me if I'm wrong, but I feel like most investments in ML are not successes and often not great uses of time. And I'm curious that that changes with more tooling and more kind of public models that people can plug into without having to build their own. I wonder if it becomes like, oh, okay, look, we'll put in three weeks, we'll get something really useful. Exactly. And also, the other thing, and I wanted to add on the question you asked before about, hey, how do you keep updated about new niche? we shouldn't underestimate academia and research blogs.

Starting point is 00:33:40 And there's a website called Archive where you can see new papers come up because this is where Chachachypdi and like used to be there for a long time. Like there was a lot of information on this sort of thing. But it's now recent where we see that research scientists and research orgs are kind of not as silent as they used to be. So there are more companies invest on staffing, this layer between productionizing and research,

Starting point is 00:34:09 the more PMs you're going to add there, then the more you're going to see this bridge kind of creating good products that are created. So sometimes you have amazing ideas by research scientists, but you need a PM to take it and actually figure out ways to also monetizing, right? That's the other thing. If you're a PM, you need to come up with ways

Starting point is 00:34:30 to actually be able to monetize. And Chatt GPP is now free for everyone, But I don't know if you, if he saw, there was a sign-up forum that was kind of coming around saying, hey, would you pay for this? What would be the minimum you would pay? What would it? The maximum would you pay would you pay? What would you like to see if you paid? So having BM's bridge that tap is crucial for companies to be able to take the research and actually come up as meaningful use cases for users.

Starting point is 00:34:56 I think they actually started charging the other day. I think it's like $40, $42 a month to start using it, I think. people have been talking about on Twitter. I don't know if that's live yet. And then you talked about research papers. When I think that, I always think of Tyler Cohen. He has this awesome blog, Marginal Revolution. And he's really good at sharing insights from research papers that he's reading. So that's another place for folks to check out. He's just like this really smart dude. He's really excited about AI and GPT in general. So he shares a lot of really interesting insights about it all. Segwing a little bit to your course. I have a couple questions about it. One is just like, can you just

Starting point is 00:35:32 talk about like the broad framework of your course. Like how long is it? What do you learn? What are the workshops broadly? And then I have a couple follow-up questions. My course is three week long. It's meant for people that they're either aspiring or current PMs that want to understand how to sprinkle in AI solutions or the one of the full-time AI programs. Week one is more about introduction, what the product development lifecycle is for regular products and how it differs for AIPN specifically. And then we talk about idea, creation. How on earth do you come up with ideas? And I love what the job sets where he used to say, well, users don't know what they want until you show it to them. And that's exactly the mindset.

Starting point is 00:36:14 I want to embed to people and say, hey, people don't know how on earth do you use AI. People would never have much chance to be able to do what it is. And then we take that and we dive deep. And we talk about how on earth do you productionize something like this? What are the different partners you're working with? What is the research scientist? And how how on earth do you collaborate and how do you partner with them? How do you convince them of what you have in mind for their precious research to be converted into a product? How on earth do you convince them to trust you and how do you influence them? And then at the end, we're talking about how you actually will be able to pave your path to a PM all the way from interviewing for

Starting point is 00:36:54 this role from what good resumes look like and doing some local interviews because the more or practice, the better. How many workshops are there through the course? Nine workshops. Nine workshops. Okay. Of the nine workshops, which of them are you finding is the most exciting, game-changing for someone most interesting? So throughout the duration of all these workshops, people have homework and they actually

Starting point is 00:37:20 take home an exercise where they need to create and develop their own AI product end-to-end. And they can pair up with each other. By the way, there was this two students paired up, and actually where I would raise funding, which is mind-blown to me, which is really really great. That's awesome. But to continue it, the most exciting part is when everyone at the very end are actually presenting their work, and they're actually asking questions and getting feedback, and they're just really excited and cows for what they created. That's a good reminder of a lot of the learning that you do is just doing it, not just kind of reading about it and following Twitter.

Starting point is 00:37:54 Can you share any examples of stuff people built after the course? someone was able to actually, and I can't you not, create a little model that was able to take as an input x-rays that defined online and it was able to tell us what was wrong if something was wrong with that patient. And it's just crazy to think that you can do that within three weeks. Obviously, it was just by photos we were able to crawl online for x-rays. But the concept is there that you can build something like that. You can create. it and to take it a bit further, they wanted to create a lower recommender system and say, hey, we think this is what's wrong with you. Here are the Stapton's your problem. Obviously, we're not trying to play doctors or to pretend that we're medical in any way. But being able to see that actually functioning is just, it's very helpful. That's amazing. Do they already know how to code this team that built this thing? They did not. But part of the course is to teach people

Starting point is 00:38:55 the basics that you are going to need for the PM rents. And there are some no-coil tools, as I mentioned, that are going to allow you to drive and drop and train these models and input photos in it and be able to do it. Can you mention those tools again? Because that is really interesting. And it's just like a peek at your course. But if someone wanted to start building something like this,

Starting point is 00:39:14 what are some of these tools they could check out? One of the tools I would like to recommend to people is actually auto-m-m-el. This is offered by Google Cloud. And essentially, it allows you to train high-quick. while it would be custom machine learning models with minimal life forward. You don't need to be able to understand the shape code or anything like that. You need to have a lot of photos and images that you have already corrupted, but it's not going to do the collection for you. And a great application I had to see. There's actually a YouTube video about this is

Starting point is 00:39:43 there was this company that actually had a lot of wind turbines. And what they did is, in order to maintain these, they will actually have people manually, have huge ladders, and go take a look and see if everything was okay. So eventually they just got drones and they had these drones fly on all of these machines and take photos on everything. And then they downloaded all these photos and they uploaded on AutoML and they were able to see which ones need and maintenance and which they did not. And I think they reduced time from like three weeks of work to like a few hours of knowing which need maintenance and just be able to send people there. So it's this type of thing that you can do on your own by applying this sort of tools.

Starting point is 00:40:25 And that tool is called AutoML? Yes, AutoML. Amazing. We'll link to that in the show notes. Coming back to your course and maybe just a couple more questions, can you just talk about what it takes to build a course like the course you built? How much time did it take you? How much work did it take?

Starting point is 00:40:41 Anything there you want to share? I treated creating my course like the product. This is like what I did is I came up with some hypothesis as to who the audience was and as to what they were so keen to get out of it. And I started reaching out to people. And I started saying, hey, first of all, would you like to learn for me? Second of all, what would you like to learn? What are the specific questions that you would need answer?

Starting point is 00:41:06 Because these are people that are working full-time that have families. In order to take a break from all that, he needs to provide something to them that is meaningful. And there were quite a few iterations. In the beginning, I was focusing the course. more for software engineers that wanted to become AI product managers. But then I realized, no, there are a lot of PMs that want to become AI product managers. So I did a little online shift in fare. So what it takes is make sure you find the right audience. Make sure to figure out what that audience wants. Make sure to have the right duration. One week, I find it too short.

Starting point is 00:41:44 Two weeks. People still have been rushed. Three weeks is excellent because you give the opportunity to everyone to present and to keep to know each other on like an offline Discord community, which is another important part. And then the last thing, you need to have a personal relationship with everyone. So I've messaged everyone. I've seen everyone's application. I met with some people as well just to make sure to answer any questions and concerns because I wanted to make sure that people were comfortable,

Starting point is 00:42:12 just trusting a stranger like me and paying them to provide knowledge for their course. So it took quite a few iterations, but I was able to get there. And I'm very, very happy about it. And I recorded it offline as well for people. Has anything had to change in this course? Maybe it's just as the last question, things are moving so fast. Is there anything you've had to like rethink, redo since you first built it? I actually added bonus sections.

Starting point is 00:42:37 And one bonus section was Judge EBT and how it was trained. This is because I started this new cohort in December. And on day one, the question I go is, what is this? How did it start? What is going on? How did they train? So I added the beginning section for it and I put people to it. Amazing.

Starting point is 00:42:53 Anything else that you'd like to share before we get to a very exciting lightning round? It was someone that recommended. I actually did a course. And in the beginning, I was not. In the beginning, I laughed and I said, wait, people would want to learn for me, really? And of course they did. And I'm teaching so many people. So what I want to tell people is, don't underestimate this.

Starting point is 00:43:15 Try creating your own courses as well. people may want to learn what you take for granted. For them, it may be game-changing. It can be life-changing. So building courses is an amazing thing. And, you know, we're living in the full collaboration era. And so the course is content. So go try this. I find that teaching and at least crystallizing thoughts is one of the best ways to learn it yourself. I imagine you learned a lot about AI much more than even came into it with just putting it together into a course. Absolutely. And I got some uncomfortable questions that I had no idea how to tackle. Like, people on day one were like,

Starting point is 00:43:51 how do I assess the trade-offs between these two different moments? And it had to figure out how to answer these things and how to convert them in my course. So learning from the students, learning from the course, learning from explaining is just so viable, so skills that you can get. Well, with that,

Starting point is 00:44:08 we've reached our very, very exciting lightning round. I've got five questions for you. I'm going to go through them pretty quick. Whatever comes to mind, share. We'll see how it all goes. Sound good? Sounds good. Two or three books that you recommend most to other people.

Starting point is 00:44:22 Inspired. It taught me. It's all about how to create tech products. People love. Marty Kagan, right? Yes, that's the one. Great. Cool.

Starting point is 00:44:32 Anything else? Or that's the one that comes to mind? You look like a thing and I love you. And I have it right here. It's a great thing. It's super super cool. It's about how AI works and why it's making the world a weirder place. It's actually a very fun.

Starting point is 00:44:46 And there's one. One more, which is a book, a workbook I recently launched with Alana Karan. And it's about it's a workbook for women in tech trying to navigate working in tech. It's called Adventures of Women in Tech Workbook. So that's another thing that they want to shamelessly plug it. That's a great choice to plug. Where can folks find that? Is that on Amazon?

Starting point is 00:45:06 Yeah, Amazon. Amazing. What's a favorite other podcast that you like to listen to? I like BOS's podcast. I mean, if you're aware of it, Boss is the CEO of, Facebook. He has a great podcast. I have not heard it. I do know of Boz. I'll check that out. I didn't have a podcast. He had some great writing over the years. Maybe that's why he doesn't write it anymore. He has this podcast. What is a favorite recent movie or TV show that you've loved?

Starting point is 00:45:30 Oh my God. The White Lotus people were talking about this thing. I ended up, you know, just trying it out. And me and my husband, we just binge-watch the whole thing. It's just so different, so mind-blowing. Get you excited about going to Hawaii again. It's just, it's really good. Have you seen the second season? I've seen it, and it's so much better than the first, which is rare. I agree. Awesome. Love that show. What is a favorite interview question you like to ask?

Starting point is 00:45:55 And bonus point if it's AI-related. I love to ask people, how would you explain a database to a three-year-old? And I know it's kind of an AI, not very rare. But I love asking it because people are kind of thinking about things like, what did you just ask me? But it's so important to be able to explain things in a simple ring and have this storytelling to convince a kid and really explain technical terms to non-technical people. Favorite AI-based tool that you think people should check out? I mean, we'll talk about tragedy PT now. My head is on tragedy PT. This is what comes to mind.

Starting point is 00:46:32 Well, the lens out was pretty cool, too, right? We're all uploading our photos. We're able to see what people would look like as fantastic heroes. I have to say, I tried being the nail version because it was so much cooler than the female version. So that's when I recall. comment to people. Try the mail. That's fun. And there's actually a, they actually have pets now. That's what got me to download and pay for it. You can take pictures of your pets and they look so fun. That's like a killer feature right there. Good job, Lenza. And the app is Lenza, right? Yeah. Lenza. Amazing. Marely, thank you so much for spending time with me, sharing your wisdom. Two final questions. Where can folks find you online if they want to learn more and reach out?

Starting point is 00:47:10 And how can listeners be useful to you? Thank you so much. People can find me on Instagram. They also have a product channel on YouTube that you can check out. I just started it. I'm getting used to the whole process. I'm also kicking off a newsletter. Just any social, reach out, and you'll see all my links. How do they find the YouTube channel? How do they find the newsletter?

Starting point is 00:47:31 Typing Marilyn Nicke. Merrily, thank you again for being here. Thank you so much, Lenny. It was a flavor. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcast, Spotify, or your favorite podcast app. Also, please consider giving us a rating. or leaving a review, as that really helps other listeners find the podcast.

Starting point is 00:47:52 You can find all past episodes or learn more about the show at lenniespodcast.com. See you in the next episode.

Lenny's Podcast: Product | Career | Growth - AI and product management | Marily Nika (Meta, Google)

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.