Experts of Experience - #18 Innovation in Education: Duolingo's Approach to Immersive Learning with Megan Bednarczyk

Episode Date: February 21, 2024

Join us as, Megan Bednarczyk, staff product designer at Duolingo, gives us a deep dive into the use of generative AI in the language learning app and how users are getting the most out of the technology. Megan shares how GPT-4 opened up new possibilities for personalized and immersive language learning experiences and she emphasizes the importance of testing and iterating with users to ensure the effectiveness of the AI-powered features. Megan also discusses the strategies used to create an addictive and engaging experience for Duolingo users. Tune in to learn:Generative AI, such as GPT-4, has enabled Duolingo to create personalized and immersive language learning experiences.Testing and iterating with users is crucial in the development of AI-powered features.Creating an addictive and engaging experience is key to keeping users motivated and coming back to the app.Understanding the user's perspective and incorporating subject matter experts are essential in designing effective AI-powered products. Creating a fun and engaging experience is crucial for keeping users engaged and coming back to an app.Drawing inspiration from other apps and games can help incorporate gamification elements into different industries.Balancing business needs and user value is essential in monetization strategies.Using AI to solve user needs can enhance the user experience and provide personalized solutions.Avoid blindly following AI trends and instead focus on solving real problems for users.Staying informed about AI developments and experimenting with the technology can help uncover new possibilities.Starting with the user and understanding their needs is key to building successful products.–Imagine running your business with a trusted advisor who has your success top of mind. That’s what it’s like when you have a Salesforce Success Plan. With the right plan, Salesforce is with you through every stage of your journey — from onboarding, to realizing business outcomes, to driving efficient growth. Learn more about what’s possible on the salesforce success plan website.  Mission.org is a media studio producing content for world-class clients. Learn more at mission.org.

We're competing with TikTok, and we're competing with Instagram and Snapchat. If it feels like a textbook, nobody's going to spend time on it. We have to create fun they can do in their five-minute commute or the 20-minute subway ride. We're trying to make language accessible to all. Looking to these games, seeing how they use animation and delight to get people engaging with it and we can take some of those principles and apply it to learning, then that benefits both us and the people that are trying to learn a language. Hello, everyone, and welcome to Experts of Experience. I'm your host, Lauren Wood. And today I am speaking with Megan Bednarzek,

a product design expert who is currently the staff product designer at Duolingo, the ever so popular language learning app. In her role at Duolingo, she leads the Duolingo Max team, which is their latest generative AI-based subscription tier. I'm really, really excited to get into this because there's so many juicy insights about how they have been incorporating AI into their app, which I'm a huge fan of. Previous to Duolingo, Megan led design at Geneva, a group chat startup, and she also spent time at Digitas. Throughout her career, she's worked with a number of heavy-hitting clients such as HBO Max, Audible, JP Morgan, and Amex. So today we're going to dive into Megan's thoughts on UX and

the future of AI in design. Megan, thank you so much for coming on the show. Thank you for having me. I'm excited. So as I was looking into your background and Duolingo, it was ever so apparent that you've worked on Duolingo Max, which as I mentioned, is an AI-powered language teacher. And you worked on this product in collaboration with OpenAI leading up to the launch of GPT-4, which is super cool. And I can only imagine that there was a lot to learn considering GPT-4 hadn't even launched at that point. So I'd love to understand as you were approaching the launch of this product, just in general, what did GPT-4 open up for you in terms of what was possible

at Duolingo? Yeah, that's a great question. So we found out about GPT-4 about like a little over a year ago at this point. I still remember the first moment that we started playing around with the prompts. We had like this little Slack bot that somebody had uploaded. And it was funny, we were working on a different project at the time. And our CEO, Louis Van On, came over and he came up to our computers and he was like, you have to see this thing. So he had us test it out and he had us basically ask it a bunch of questions. And again, this is the first version of it. This is super rudimentary. So we asked it things like, explain this mistake that I made. We tried to make it practical to do a lingo. We're like, I said this thing and tell me what I did wrong.

And not only did it say exactly what we did wrong, but it explained the grammar rules and the reasons why. And we tried some other things. We're like, write a story about a girl who lives in Guatemala that dreams of being an astronaut, right? And it wrote this pretty beautiful, like wonderful story. And then we're like, okay, like trying to play around with like, here's some other things we like to test to do lingo. We're like, rewrite it at a beginner level. And it was able to take more of that advanced language and rewrite it in like really, really simple terms. We tried things like add this particular word, rewrite it using a specific kind of personality. We have these characters at Duolingo that have these different types of personalities. So that's kind of always one

of the things that we like to try. We hadn't obviously translated it to many different languages, explained certain words. And ultimately, we were just blown away by what it was able to do. For years at Duolingo, we had been looking at a number of gaps in our product. And what we really wanted was to put a tutor in front of each and every one of our learners so that they could actually have a really immersive and personalized experience on Duolingo. But a tutor is really expensive. And real and human tutors are definitely the best way to learn a language. But that's not something that can scale to, you know, all of the learners we have that don't necessarily have that money but still want that type of education. So after seeing the quality in this initial model, we were like, this is something that we thought was probably like four to five years away.

And we realized that we were actually there today. Obviously, while this instruction will never be as nuanced and personalized, it's really scalable. And we think that that's really powerful. So that was that big first moment. And honestly, we pivoted our entire team's direction and strategy in the matter of a week. We were like, all right, we're going to build two features using generative AI.

And so we're the monetization team. So we're like, okay, how do we give our users access to these features? And obviously, there's a cost associated with it. It's something that could go down in the future, and we expect it to. But we kind of had to package it underneath one of our monetization tiers. And for a while, we'd been wanting to expand our tiers to include like a higher tier. We wanted to like basically give people more options and expand beyond the offerings that we already had. And this felt like a really good opportunity and a way of packaging those generative AI features. So saw the demo, pivoted everything, and we're kind of off to the races in a very short amount of time. How long did it take you from that moment of, okay, this is what GPT-4 is,

and this is what I can do to actually launching a new product? That was about... It was still a very short. That was about like six months. It was between like October to March is when we officially kind of started rolling out that MVP. So we basically had two quarters to design two brand new, like brand new zero to one features using this new technology and figure out how to package it as the IR subscription tier. So very short amount of time. And partially that was because, one, when we first got access to that demo, we decided we were going to be a partner with OpenAI in their launch of GPT-4. As a company, when they first came out with this model, they wanted to release

it with a few different sponsored companies because they wanted to give everybody some examples for how this technology could be used for good. They cared a lot about that and they wanted to showcase important uses of it across a multitude of different businesses. And they were really bought into our mission and what we are trying to do at Duolingo. So it was a natural fit there. So we had to get all of this done by the time GPT-4 was announced to the public. So that was our pretty strict timeline there. I'm curious to know if you were also using it to help you build the tool and integrating

it into your systems and kind of redesigning how you were approaching the development of a product in using this technology? We did a lot, especially in those first few months, just to understand the intricacies of the technology and the different ways we could use it, both for ourselves and our own processes and also in the product and what it could actually solve for our learners. So we did a number of company hackathons where we'd all come together and just kind of go crazy with what can we do with this thing. And we found a lot of interesting ways to incorporate it into our own work streams in the way that it can support the human employees that are using it, not replace them. I think it's a very effective tool for

kind of helping you get the first start to something, right? Like that first draft or that first structure. Like I want to do this thing, or I'm thinking about this thing, or summarize these notes. So we use it a lot for things like that. But then you kind of always have to go back through and edit it and apply your own knowledge and experience to turn it into something that can be used. But yeah, absolutely. We've been thinking about how we can use it in our processes as well as, you know, actually creating a product around it. I think hackathons can be such an amazing way to just go like as far as you can go with something. I'm curious. I always like asking this question, like what got left on the cutting room floor that was a really good idea that you might come back to?

Oh, honestly, honestly, so many things. We kind of have a few different teams that are working on this technology. We have our team that like actually right now, you know, have has these live features in the app that are built on generative AI and using GBT4 in real time. We have other teams that are almost like R&D type generative AI teams, like thinking about like all these different features and all these different use cases of it. I think like something that we've like tossed around a lot that could be fun is like creating a type of like Mad Libs type experience where you put in like a lot of the Duolingo things you love, like choose this character and choose this setting

and practice this travel scenario to my upcoming trip to Barcelona in, you know, 2024. So like thinking about ways we like allow people to craft their own learning experiences through like plug and play is something that we definitely are excited about and maybe we'll incorporate in the future. That's great. How about the challenges that you faced? I mean, especially not only is the quick timeline a factor, but just using something that is completely new. And at this moment, when you were introduced to the technology, like no one's really had that much experience with it. Maybe you understand it conceptually, but like you're really doing this for the first time and

everyone is. What were some of the challenges that you faced? Yeah, I think honestly, one of the hardest challenges at first was just figuring out at which stage we need to incorporate it into the design process. And I think early on, we started to make some assumptions for like how it could work. And I think we were a little bit limited by like how we approach creating, you know, most features at Duolingo that are not reliant on this technology. And we were like, okay, one of the features that we built was this feature called Role Play. It's a conversation

practice tool. And we're kind of approaching it from almost like a two-structured way. We were like, all right, these are the components of a conversation. We're going to basically have people navigate this conversation. We're going to give them three tasks to complete during the conversation. And then once they've completed all three tasks, they'll be done. Let's take the example maybe of you're ordering coffee from a coffee shop. So we're like, okay, step one is to introduce yourself to the barista. Step two is to order your coffee. And step three is to decide how you're going to pay.

So that's kind of how we started thinking about how we'd actually have users engage with these conversations. Like every scenario would have these tasks to complete. But then once we started actually like testing this out and seeing how it played out and design and using this generative technology, you can't actually plug every single conversation into a formula. And if you said anything that didn't fit perfectly within how we had structured the conversation, it just wouldn't work and it would kind of go off the rails. So let's say I say, hey, I'm Megan and I want a banana.

And then the character starts talking about bananas and then I never actually get to the task of ordering my coffee so that I can't complete the conversation. But I'm still having a conversation and I'm still practicing my conversation skills. And maybe I want to talk about bananas and I should be able to talk about bananas. And the really magical, cool thing about generative AI is that I can talk about bananas and the character can respond to me and we can go off in that direction. So we tried to limit it too much and like, didn't realize that actually

we wanted to like lean away from structure and kind of allow for this free form conversation. And that's where it would actually start to feel really magical. I mean, it's, it's so true. I actually am currently using Duolingo to practice my Spanish and I am in the very, very early stages of it. And I'm not using Duolingo Max yet. But I'm like, okay, I am learning what coffee is and bread and all these things. And then I go to use it and this exact thing happens of, oh no, now they're talking about my shirt maybe, or they're like commenting on something because my accent was so good. And they think that they can go there. So that makes a lot of sense that, and such a wonderful application of generative AI as well. Yeah. And like, I think the big process learning for us there was that like we had to figure out our feature solution in the prompt before we ever jumped into design. So like whenever we had like a product strategy or idea, we'd create the prompt for it and try it out, you know, just in like a text editor type of

interface to play out if it actually worked. And then once we could validate that, then we'd actually move to the next step. So that became like a key part of like any feature improvement or update or new feature that we build using generative AI. You have to start with the prompt and you have to like test your product ideas early and often just to validate what works and what doesn't. It's not something that like you can imagine with just boxes on a screen, you have to engage with the actual tool. And that's really what the product is all about. So that was an important process learning for us. And now it's something that we incorporate into every improvement that we make to the features and every new thing that we do.

Cool. Tell me a little bit more about that process of testing and how do you structure it and really make sure that you are pushing the tool to the limits to make sure that you're getting those edge cases? So it's definitely something that we've tried a lot of different methods for and have slowly evolved our processes over time. But I think we've gotten into a pretty good rhythm now where we basically created an internal tool that allows us to test any new prompt update and basically simulate what's, we're still talking about our conversation practice feature because this is specific to that,

but we can basically simulate any conversation and any update we make to the prompt and test that conversation in this tool that we've built. One of the big things that we've focused on in the last few months or so is we have these characters in Duolingo and each of the characters has their own personality. Some of them have their own like little fan flubs. Like everybody knows Duo, but like in this feature called role play,

you get to talk to all of our other characters. So we have this character named Lily. She has purple hair. She's probably one of the more popular ones. She's this snarky, moody teen. And we have this other character named Oscar. And Oscar is this very pretentious art snob. So they all have these worlds built around them and these like really rich

personalities. And that comes across like when they show up in lessons, when they show up in stories in the app. And like it's really kind of one of the things that differentiates Duolingo from other products and differentiates like the features that we create and the way that we approach language learning in this fun and immersive way. And originally, when you kind of see that character face and you're talking to it, and that personality didn't come through, it started to feel really dry. It didn't really have that like hook and storytelling that made Duolingo special. So one of the things that we did was we basically wrote specific prompts. So I'm using this word prompt because this is like a key part of working with generative AI.

It's like a series of different prompts or instructions that we write to it to get it to act the way that we want it to act. That's the very simple way of doing it. You know, when you are engaging with chat GBT on the internet, you like ask, hey, summarize this paragraph or give me 10 words that mean

stellar. That's a prompt. And our system is a very intricate, complex system of multiple prompts working together that creates our feature. So we created specific prompts for each character that like basically had all these notes about their personality, like has a son, is really friendly, blah, blah, blah, likes these foods. We compare it to real life personalities of these characters that kind of reflected who they were. So we could actually pull data from the internet on these people and kind of incorporate that into their personality. But we wrote these prompts and then plugged that into our conversations. From a process perspective, it's going to take a little tweaking to get that right. So we had some creative writers write these prompts for the characters. And then using

this testing tool, we basically would have a bunch of conversations with the characters and see how it was starting to feel. And one of the things that we noticed as we're testing, let's come back to our character, Lily, is that she was so sarcastic at first to the point where she was almost mean. And we were like, this is so demotivating. Like if you're a beginner learner and you're just trying to have a conversation and she's basically bullying you, that's not going to feel good. So we had to tone that down a bit and get like the right level of like slightly sarcastic, but still encouraging and, you know, friendly enough. So I think that's a use case that like you really just have to get in there and test it. And, you know, we would have, we'll have our whole team come together, sit in a room for 30 minutes. We'll all test out different conversations. We'll

leave feedback and then we'll, you know'll make iterations and updates to the prompts and kind of go through that cyclical process until we get somewhere. And I think the reality too that we've all had to become comfortable with is it's not perfect. It's never going to be perfect. There's always going to be instances where it doesn't work in the way that we want it to. So our goal is we've kind of formulated these rubrics in our head or like there are certain things that we have to achieve with each different iteration. We're like, this is really bad and this can't happen. This is less bad and let's try to avoid it as much as possible, but we're okay if it happens from time to time. And just kind of like make our own assessment of like what's really, really crucial knowing that,

you know, we cannot control everything with it. We just have to be comfortable with that. Yeah. So it's really a matter of just like pressure testing it constantly over and over again, going in and asking questions, using it, seeing what comes up and just like putting it to the test. Exactly. And, you know, we test it internally and then we actually test it with our users or our learners, as we like to call them in Duolingo. I'm sure you'll hear me use learners so often. And sometimes I say that and I forget to say like, that's how we describe users. Great. Well, now we know. And I know. So, you know, we'll test it internally. We'll get it to a point where we're like,

okay, this feels good to us. And, you know, let's keep going along with the character personality example. We're like, okay, we feel good. And then we like run it as an A-B test with users. So we'll have like a control group that's using the current one. We'll have this one. We'll obviously outline like, this is actually what we're hoping this feature improvement does for us. Just like you'd approach any A-B test

for any feature improvement. Like we want this to improve engagement or we don't care what this does. Like we don't care if this actually improves any metric, but we think it's a better experience and we just don't want to like hurt these other metrics. So then we'll test it with users, see how that goes. And then, you know, either kind of come back and make iterations and try it again, or launch it and build off of that latest version. When you're in the phase of not yet giving it to users, and you're really putting yourself in the user's shoes, and I'm sure you're doing that throughout the entire design process, as a product designer, what tools or techniques do you use to really

put yourself into the user's shoes to build that experience that is going to be ready to even test with them? That's a really great question. And the lucky thing for us at Duolingo is we have learning scientists embedded on every single product team. So everybody on the team is learning some language and has varying levels of experience, which is a good thing. Like I am personally I'm learning French. I'm a pretty beginner French learner. Some other members of the team are very advanced French learners. And, you know, we have a multitude of different languages. So we have some experience just with our own

journey in Duolingo and our own stage in our language learning process. But what we really rely on to make these calls is these learning scientists. And these are people that know, you know, have PhDs in how to teach a language and like have an inherent understanding of the needs of somebody who's like a very beginner learner to somebody who's a very advanced learner. And they're really the ones that like kind of help us make the call, especially when it comes to these meatier learning questions. It's interesting because honestly, everything is impacting that fundamental piece. Even character personalities.

If we go too crazy with character personalities, they start to use vocab that's too advanced for beginner learners. So we need to make sure we have our learning scientists gut checking that. So that's really the most important thing, especially for a product like ours. And it actually really started to shape roles of people that we embed on each team. So we have somebody on our team that's a learning scientist and prompt engineer. And they're really the core person that's writing most of the prompts for the products. And they'll work with engineers and front and back and engineers for how that's actually incorporated into the actual code. But they're a subject matter expert.

And they also have this prompt writing experience. And that's really the magical formula of how we create prompts that are relevant and addressing all of our learner needs and actually solving for the goal that we're trying to solve. So I think that can be applied to really any company. I think people writing the prompts need to be the people that are those subject matter experts on the thing that you do. And whether or not they're the actual prompt engineers or not, they definitely need to be part of that process. So they're kind of like user advocates in a way. They're the person inside who understands the end user the most, and they're actually helping to build the product through that experience.

So finding people who can kind of have both of that is the sweet spot is what I'm hearing. Yep, totally. And then obviously we, you know, we'll run user research and run qualitative studies and like actually run it with real learners that give us like a better sample size. But yeah, we always have our learning scientists as kind of the voice of the learner in a lot of these decisions and in every step of the product process. When you're using Salesforce to tackle your company's most important goals, failure is simply not an option. That's why their most highly skilled advisors, Salesforce CTOs,

are available to help you succeed with expert guidance and implementation support at every step of the way. Learn more about Salesforce CTOs at sfdc.co slash professional services. How did you do this in other roles? I'm just, I'm curious and, you know, take AI out of the picture because obviously that's come up so quickly. But in the other work that you've done, how did you approach having that person internally who could really help to architect the product from the user's perspective? Or what strategies do you use to have that person or that mindset, let's say, if that person didn't exist? Well, I mean, I think even beyond generative AI, we still have learning scientists embedded on all teams. And whether they're writing prompts or whether they're just advising the direction and the strategy of the product, they're working hand in hand with product

designers on pretty much every single initiative at the company. For example, this is related to our specific conversation practice feature, but it's something that could exist on anything outside of generative AI. We're like, okay, freeform conversation is very hard. And how do we give beginner learners the tools they need to be able to respond in a conversation? And how I as a product designer work with the learning scientists is we discuss like different methods. Like what is the right amount of support that we can give to them so that they don't feel like totally lost or confused, but we're also not giving away the answer. So it's still a learning moment. And they can kind of help us like fundamentally form those strategies of like, okay, we could give them this resource

or this resource or this resource. And then I, as a product designer can think about how that actually comes to life in the product itself and how that, like how we can make sure that we can represent that in like a clear, simple, easy to understand way. And then, you know, we'll then work together from that point to like align on an actual version of that, test it out, see if it's like accomplishing what it should accomplish, and then kind of go forward from there. So they're embedded in every team regardless of generative AI. And I've seen that model at other companies as well. It's like you really, it depends on the thing that you're building. I guess if you're working on like a social or a chat app, you can kind of put yourself in those shoes pretty easily since we're all using those apps.

But specifically for a learning product or something else that requires that specialized knowledge, it's really important that that subject matter expert or voice of the user is really embedded in early product strategy and throughout. Definitely. And I can imagine really going and either being that user and being able to experience it yourself so that you can embody that mindset. Or how I've also seen it done is just spending a ton of time with users and sitting next to them as they're using it and understanding what's going on. So you can really like pull that inside. Yeah, I think it's so important. And, you know, we do a lot of user research and I'm a team lead alongside my PM partner and engineering partner. And we all make it very clear to our entire team, like we all need to attend user research sessions.

Like this is not just something for product people and designers and all engineers should be there because, you know, we're all working together collectively to build the right experience for our users. And ideas don't just come from leads or ideas don't just come from product and design. Like a lot, some of our best iterations and ideas come from leads or ideas don't just come from product and design. Some of our best iterations and ideas come from our engineers. And they're the ones that understand the tech and the limitations and possibilities there so deeply. They can sometimes suggest some of the best product iterations. And I think that's why it's so important for every member of the team to spend as much time as possible just understanding what the experience is like for our learners. I've definitely seen that in some of the

companies that I've worked in before where the engineers like to do what they do best, right? Sit and code and be in their bubble and get really deep into something. And because I typically work on the customer success or the customer experience side of things. And I'm usually the one trying to bridge the gap for the two and like pull an engineer, like get them into a conversation with a user if I can or whatnot.

And I've always found that to be a bit of a challenge, but I'm glad that you underscore the importance of everybody on the team getting to have that real user experience and connect with and empathize with the user. Everyone needs to do it. It is. It's a super fun experience. And I'm curious, how do you think about creating something that is innately addictive so that people want to be using it all the time? What types of thoughts or strategies do you put in place there? Yeah. So I think one of the fundamental things that we kind of all use as a principle in anything that we approach at Duolingo is we don't look at other language learning apps and products as our competitors. We look at any other way people spend their time when they're on their mobile device. So we're not really competing with another language product.

We're competing with TikTok and we're competing with Instagram and Snapchat and Be Real, although I guess we're past the Be Real days point. It was a fast one. But we're competing with all these other very engaging social platforms. And it's really just like what percent of a single person's time is spent on Duolingo. And to actually get them to use our product, we have to create something that's engaging and entertaining and fun to use. And if it feels like a textbook, nobody's going to spend time on it. They're going to spend time scrolling through TikTok. So I think just coming at any product problem

with that mindset is so important for us because we have to create fun, simple, bite-sized experiences that people feel like they can do in their five-minute commute or the 20-minute subway ride that they have or between when they get home from work and their kids get home from school. We know that we're trying to make language accessible to all. And to do that, we need to make sure that people can actually get their practice in a way that works for their own schedule. So bite-sized element and simplicity is really important there, but also just making sure that it is engaging and entertaining. And I've mentioned the character personalities that we have, but besides that, we have a lot

of mechanisms for thinking about how we can continue to get people to return to the app on a daily basis. And really all of that is for the sake of learning because to learn a language, you have to kind of constantly be immersing yourself in it. Like if I do an hour of French today and then don't do it for a month and do an hour of French again, I'm not really going to retain anything that I learned a month ago. And the best way to learn a language is to continue to practice it, continue to learn the vocab, learn the new grammar, practice the things that you just learn and do that on a consistent daily basis. And that's really the only way you can learn and grow. So to do that, we have a bunch of strategies for encouraging people and making it fun to

come back to the app. We try to create these smaller milestones along the way because learning a language is hard. And from when you start day one, like I'm going to learn French today to like actually being a fluent French learner, that's a really, really long journey. And that's going to take a while. So we need to make sure people are motivated along the way. So one of the things that people really resonate with is obviously the streak. And that's just the way to get people coming back every day to the app, continuing that streak, keeping that up. It's something that people are so proud of.

Like, honestly, even whenever people will apply to jobs at the company, they'll reach out to me and be like, I'm on my 2,000-day streak. And I'm like, that's better than me. I'm like, but people are so proud of it, which is really, really cool because it gives them this milestone of like, I am taking a step every day to learn a language. I'm taking five minutes away that I maybe would have spent on TikTok, and I'm going to apply that to something that's meaningful and useful to me in my life, which is really cool. We've kind of capitalized on that as well. One of the more recent things

that has become very popular is our Duolingo widget. And I don't know if you have it installed on your phone, but we basically have it. People call it the Duolingo Tamagotchi. So basically there's like different chaotic poses of duo that changes throughout the day as you get closer and closer to not completing your streak. So like at the beginning of the day, he's like really happy and he slowly transforms to like chaotic and then like very, very angry if it's getting to be about midnight. And a lot of people say, a lot of the users that we talk to will be like, I have to keep Duo happy. Like that's the thing getting me to come in and do my language learning every day to make sure I don't get angry

Duo on my phone screen. So things like that and kind of capitalizing on the chaotic and fun side of the brand and using that as a way to get people to come back to the app is also really important. Yeah. I mean, the gamification is real, right? And I was a huge Tamagotchi user. So that actually resonates with me a lot. Where do you look to for inspiration with these types of features? Honestly, all across the ecosystem.

And again, like I said, we don't really look at other language learning products. I think obviously to some extent we will redo a little bit, but it's like, what are the things that get people coming back to TikTok and Instagram and other things? What are things that work and how do we apply that to language learning? How do we apply that? And I think the fun thing for us, both from a growth perspective and even a monetization perspective is all of this is in service of getting people to learn a language or from the monetization lens, our Duolingo subscriptions make it possible to have a really, really good free product.

So by getting more people who are able to pay for the subscription, it allows all these people to learn languages for free, which is really, really important when you think about that at scale. And that's really the core of our mission. We look all across social apps, other apps, all across the ecosystem, way beyond ed tech, and also mobile games too.

I think that's something that obviously we are known for at Duolingo is how do we apply these fun gaming mechanisms and use them to make learning fun. And one of the things that we looked at recently, I don't know if you saw, but over the holidays, I think Monopoly Go was number one in the app store for a period of time. So we're all like, what is Monopoly Go? And where did that come from? And if you play it, you understand it's really, really engaging. There's all these different tactics of earning money and building properties and like celebration moments where like I win a bunch of cash and there's like a big, you know, but get people engaging with it. And we can kind of take some of those principles and apply it to learning. Then that benefits both us and the people that are trying to learn a language. 100%. That's so much fun. I mean,

it sounds like you get to play a lot of games. I wanted to come back to something that you were speaking about earlier, and especially being on the monetization team. When you were launching Duolingo Max, the need to bring in money as a part of this new product, obviously using AI adds an extra expense and there's like a business need there. How do you approach one solving for the business need, but also creating an experience that has value that is applicable and value that people want to pay extra for? How do you think about those kind of bridging that gap? It's a great question. And kind of coming back to just technology and technology especially, that's very, very present in news and everything in the current time. People can kind of hype up

these technologies in a way and everybody is like, I need to do AI something. Two years ago, everybody was like, I need crypto to be a part of my business, even when it didn't actually make sense. And I think for us, we were excited about AI because we were excited about how we could use it to fill these learner gaps or needs that we'd been trying to fill for years, but just couldn't because building a conversation practice feature without generative AI requires so much human labor costs and such an intense content need that it's kind of near impossible to make it make sense as a feature. So we'd been wanting to do it for a while. And for us, AI was like the key or the tool to unlock this thing that we'd already been hearing for years was something learners wanted and learners needed and learners were saying they would pay money for. okay, we have AI. How do we make Duolingo AI? Or we have crypto. How do we make Duolingo crypto?

That never happens for a reason. There's never a dual coin. But AI was something that we were like, this could help us solve these problems that we inherently know our problems learners have and have said that they would pay money to solve. That's really the basis of why we built the features that we did, which was this conversation practice feature. And then this other feature that we call explain my answer, which gives people more personalized explanations or like grammar tips behind the mistakes they make in their lessons. So I do think that that is so important. And that's the way that we approach any new avenue within our company or business. It's always based on making sure that there's a want or a need or a market for it.

And that was the case with generative AI for sure. I think with AI, there's so many use cases. And I think I hear that from a lot of people that this enabled us to do something that we weren't able to do before. So it was an unlock of a problem we were already trying to figure out. I'm curious if you have any advice though, for the folks who are maybe in the camp of, we just need to use this technology. Everyone's saying like, you have to be using AI or else you're going to be left behind. And I think I also see people scrambling to try to figure out like a new AI feature just to be able to say that they're using an AI feature, but it might not actually solve a user need.

So I'm wondering if you have any advice for folks who might be in that camp of getting pressure to use AI, but not really sure where to apply it. Yeah, it's a little bit of like a chicken and an egg thing, right? Where you're like, you should understand how it works and really understand the intricacies of it and how it's evolving because it's going to evolve very quickly over the next few years. But also you don't want to build something just to build something if it's not actually solving anything for your business. So I do think that there's a lot of value in investing time and thinking about what it could do.

And like I said a little bit ago, the way that we approached this was doing a ton of different hackathons where we just had people mess around with it, break it, try crazy things that had no business value whatsoever. It was just a really cool idea. Try things that were actually really centered on learners and then kind of come together and share that knowledge. And even the things that were just like crazy fun things that like had nothing to do with our language product helped us understand like the details and the intricacies of the technology and how we could apply it. So I think it's important to whether or not you're building a product around it or launching a product to users. I don't really, I mean, maybe there's some valuable data you could gain from doing that just for doing it. But I think the better approach is

just like consistently staying on top of understanding what it can do, trying to think about all these different ways you could potentially use it for your business or just use it in general so you have an understanding of how it works and continuing to do that every few months. And I think that's really the best way to at least stay in the know of what it can do. Because maybe what it can actually unlock for your business doesn't exist today, but will exist in five months. So it's good to at least understand and maybe a new need will arise. So that's great. Where do you go for inspiration or what resources, blogs, podcasts, thought leaders

do you like to follow to really stay on top of trends when it comes to AI? I don't know if there's any like one particular place that I go to, but what we do have at our company is we have like a Slack channel that's just like generative AI news. And it's like a bunch of different people in here. And that's kind of where we all share more of like the business news of everything happening in the AI world to like new models that emerge to like crazy things people have posted on Reddit or whatever that they do

with it. And I think it's good to look at it from all angles. Like look at the really, fun things people post on Reddit or threads or Twitter. It's hard for me to call it X, but X, I guess. And also, stay on top of more of the actual news and the new models that emerge. I think it's good to look at it from all those different angles to stay on top of what's happening. Every time I see a new AI product come out, we'll always download it and we'll play around with it and just like kind of constantly get a sense of what are the different ways people are using this technology in different ways. Are they doing certain things that we didn't think of or are they solving a problem in a different way? I think I always have in the back of my head all of the core problems we still want to solve with it. So whenever I'm downloading a new tool, I'm thinking about, have they figured it out? Normally the answer is no, because it's new. It's a new technology. I think we're all still figuring out how to use it. We're

all still figuring out how to incorporate it into like our products and also just our processes. So I honestly think like there's not really one good source. I think you can just like follow people that like you think are coming up with like really interesting creative ideas and just stay on top of it in the news. Myself and my product partner, Edwin, spoke at two conferences this past year at Config, the Figma conference out in San Francisco, and at Design Matters in Copenhagen. And going to these conferences was also really useful for us, especially the Design Matters conference. There was a lot of people speaking about generative AI. So it was like a really cool gathering. We got to talk to

a lot of other teams that were using similar technologies, both in like similar applications and education and other applications. And we could like really get into the weeds of like how people were approaching problems in different ways. So I think it's really cool to connect with other industry professionals working in the space and just kind of like get a sense of like how they're approaching different problems and that could involve new things. That's great. There's so much information out there and I love having a Slack channel and just a way of sharing that. And then also connecting with other professionals who are thinking about things in the same way that you are and unlocking, taking different approaches to potentially the same problems and sharing that knowledge. That's great.

Just one thing I'll add to that. I think the best way to really understand it, you kind of have to just play around with it yourself. Whether it's for your company or not, come up with a personal project and be like, can I use AI to accomplish this? Think about how you would write a prompt to do that. See what the limitations are. I think it's something that you really need to like feel and test and see for yourself to like understand what the possibilities are. It's actually a personal challenge that I've made to myself recently of thinking about how I can use AI to solve like every problem I have. Like whenever I'm like stumped about something, whether it's like a personal thing or a professional thing or a client thing, it's like, how can I use AI to do this? And it's

been fun to think about it. And most of the time I'm like completely stumped on it, but then you do a little Google and you're like, oh, someone did this. And then you can kind of like inspire yourself on different ways of using it because the possibilities are truly endless, honestly. So Megan, I have two last questions for you. These are questions that I ask every guest that comes on the show. The first is, I'd love to hear about a recent experience that you've had with a brand. And it can be any brand, whether it's an app or a hospitality experience or whatever it may be.

I'd love to hear about an experience that left you impressed. What brand was it and what was the experience? Yeah. So one of my favorite brands or subscriptions is Nully or Newly. I actually don't know how it's pronounced, but it's one of the two. And it's basically, it's a clothing rental subscription. It's like one brand in the same portfolio of consumer brands as Free People and Anthropologie and Urban Outfitters. And I've been using it for the last four years or so, kind of in an effort to reduce reliance on fast fashion and waste less money buying clothes that I will wear once and then never again. And they just

sit in my closet for months on end. And basically what this allows you to do is like every single month you pick six items of clothing, you wear it, you can buy it for a reduced price if you want, and then you return it and pick the next set of clothes at the end of the month. And I think the best thing about subscribing to it besides really good price, great clothes, good renter reviews that make it easy to actually find the right fit and size for things is like they're very low-key approach to the subscription. And I think there's a lot of consumer mistrust in monthly subscription services because people get this fear that they'll be trapped in it. And I think a lot of fitness

brands or even gyms like Equinox make it near impossible to quit or very expensive to quit. And then I think that also shows up in apps as well. I used ClassPass for a really long time. And ClassPass is great, but to cancel ClassPass is like a 20-step user flow that I almost get so annoyed with that I don't cancel, which is probably what they're trying to do. So I guess it's effective. But I think people have this fear of signing up for these programs because of that. They don't want to get stuck in it. They don't want to waste money or get stuck paying money that they don't realize they're spending. But Nully makes it really, really easy to pause a subscription for any month where you don't need a rental or end the subscription if necessary and the click of a button on the website. And they have great customer service specialists that will answer you in less than 24 hours if any issues arise. And if there's any problem with any of the items in your box that comes,

they'll automatically award you extra bonus items the next few months. So it's really just that amazing customer service, this low commitment in a way that makes me more committed approach to subscription services that makes it one of my favorite brands. That's amazing. And I really appreciate you calling out the making it easy or allowing you to actually like control your subscription because I am a part of a number of subscriptions and it can be just really frustrating and it makes me dislike the brand when I can't pause it because I'm not going to be able to use it. So that's great. Last question for you. What is one piece of advice that you think every customer experience leader should know? I'm going to bring it full circle a little bit here.

I think the most important thing is that you really, really need to start with the user and make sure that whatever you're building, whatever feature or new product or whatever it is, whatever you're building is actually solving a problem that exists in their life right now. And when I say exists in their life right now, I mean, like now, not in 10 years. I think people people have been trying to make VR work for a really long time. And they should. And we should we should understand that technology and see what it can do. But it's not really solving a need that we have right now. And I think that is kind of what we all see happen. I mentioned with crypto, now kind of AI is just like the popular thing to do or the popular thing to use as a company. And you should understand the intricacies of it and you should understand how it works and explore what that looks like. But as you're building a product, you need to start with the user. You need to make sure that you're using whatever technology you do in a way that's actually solving a problem for them. And you're doing it in a way that's different

and unique from other people. I think when you think about generative AI, people can access it through chat GPT. They can access it in so many different ways. So when something is universally available like that, you have to really think about what is the unique angle or edge that you're adding to it that are going to get people to come to your product. So for Duolingo, what that looks like is bringing in our character personalities. We know how to create a strong narrative. How do we make that clear? How do we tie it to the specific learning journey you're on and make sure that like that particular conversation is suited to your level at the stage that you're at

and you're, you know, Spanish or French or whatever language learning journey. So I think those two things are really key and honestly can be applied to a lot of different trends, both in tech and in business. Gotta think about the user, put them front and center. Exactly. Amazing. Well, Megan, thank you so much for coming on the show and sharing all of your

knowledge about generative AI, building for the user, product design in general. It's been really insightful and we greatly appreciate it. So thank you so much for coming on the show and we hope you have a beautiful day. Thank you. And likewise, it was fun. You are a business leader with vision. You've seen the future as an AI enterprise thriving with Salesforce's agent force, and it is bright. Getting there? It's a little fuzzier.

Don't worry. Salesforce CTOs are here to work with you side by side and turn your agent force vision into a reality. We're talking expert guidance and implementation support from the best of the best. To learn more about Salesforce CTOs, visit sfdc.co slash professional services.