$100M ARR Founder: Build an Agent Harness
00:00Today we're talking with John Nerona, co-founder and chief product officer of Gamma.And we're going to talk about agent harnesses, all the opportunities around it, vertical agent harnesses.I can't wait to dive in. And I want to give a special shout out.And thank you to Anthropic, the team behind Claude, for sponsoring today's episode.John, thank you so much for coming on the podcast.By the end of this episode, what are people going to learn?Thanks for having me, Greg. I'm really excited to be here.Today we're going to dive deep on what it means to actually build an agent in practice, including demystifying what is an agent harness and what that looked like for us at Gamma, which is a creative agent harness, an agent for visual communication.
00:46And not just, you're being humble. Gamma is, I mean, it's grown extremely quick. I think it's hit 100 million ARR. You've just relaunched the platform, huge, huge release. And there's a lot you're learning. And what I hope to get out of this podcast is really just little insights around.opportunities in agent harnesses, what you're learning in real time, and other opportunities thatexist, startup ideas and stuff like that. So, John, can you commit that by the end of thisepisode, I'm going to learn some of those things. I'm in. Let's do it. Let's make it happen.So I want to start off by, like back here, you can see, what is an agent harness? Because I thinka lot of people, you know, scroll X, they see agent harness, but they don't really understandwhat it is. So can you clearly explain what it is?Absolutely. Let's break it down to the two words, agent and harness. And I'll use maybe Gamma's
01:38journey to illustrate both. So for everyone's context, Gamma is a tool primarily for makingpresentations, also documents, websites, social graphics, that sort of thing. We've actuallybeen around for several years, long before agents were ever a thing. We started the company in2020. It really took off in 2023. And early on, we were just making these one-offLLM calls to do things like write a single slide or make the outline for a deck. And wevery manually strung them together into these preset workflows to do things like generate apresentation with 10 slides. The big revolution that's happened in the last year or two is thisancient revolution where the most powerful AI products now are not stringing together these
02:14individual LLM calls by hand, but they're actually having one higher level loop of an LLM run thatwill do some kind of work, inspect it, use tools, and then turn that into something over a wholeseries of executed calls. And so we actually went through this whole journey over the course ofthis year of rebuilding our product to be an agent. And building your product to be an agent involvesbuilding a harness. So the harness is all the tools that agent uses to execute things in itsenvironment, inspect its own work, and deliver something finished to the user, includingsoliciting from the user what their actual preferences are. So probably the most famous agent harnessis something like Claude Code. Cloud Code is a harness for programming. So CloudCode teaches
02:57the agent how to go through all the files on your system and see what's there.teaches it how to like run tests and even prompts it with certain rules like go into plan mode in certaincases or run these tests after completing execution or do this stuff on GitHub. In our case, Gamma is nota coding agent, it's a creative agent. So it needs to do things like understand your art style andcreative preferences, help you craft a story. And we care a lot about visual design. We're doingthings like generating images and laying them out. So we need to know how they look and make sure they look right.So our harness takes all of those things and turns them into tools that an agent can use.And I can talk through what that look like.
03:34Yeah, I mean, please do.And I think also I'm curious if you agree or disagree with me.I've been saying agent harnesses are the new GPT wrappers or new rappers, right?Is that just a good tweet or is there some truth to it?I think there's truth to it, although I always disputed this concept of the GPT wrapper originallybecause it sort of implies that 90% of the work is the LLLL.and then there's just this thin layer on top, which is the wrapping back and forth.I think maybe what harnesses illustrate is that actually the wrapping is most of the work.
04:07You actually have all these LLMs, which are quite similar in their core capabilities,not to say they don't have some variation,but generally all the magic of turning them into a product is building the harness around them.And even the model companies like OpenAI and Anthropic are putting so much of their workinto harness development as well.So they are themselves doing wrapping of these things to achieve different capabilities.Could you walk me through what happens, like in 30 seconds after someone types a prompt on gamma?Like, how many models does it touch, like, realistically, you know?
04:36Because I think a lot of people are not sure how that actually works.Let me walk you through it.Let's go through a sample prop.So a very typical use case for gamma, someone will paste in a bunch of messy notes or a document or PDFand say, help me turn this into a 20 slide presentation.So the moment you put that prompt in, a whole bunch of things actually happen under the hood.The first thing is we actually spin up an entire computer in the cloud.This is what they call an agent sandbox.
04:59This is what the opening eye ones broke out of famously in the last few months.So we actually spin up this entire sandbox where the agent gets all these files representingthe user's prompt, the file they brought in, also other context about them.We then prompt the agent to go look through all that file and their prompt and run certain tasks.And for us, actually, one of the first tasks is usually to figure out what other information we need from the user.And so typically we will actually pop up a form and ask a few questions.questions. Questions like, who's this actually for? You know, is this like an educational presentationfor school? Is this for your work thing? We will actually call out often to different
05:35connectors. So we've built integrations to tools like Notion or Slack or linear where we canactually pull in all this relevant context. Just already in that first maybe 10 seconds, we've nowmade calls using probably three or four different models, including expensive and cheap ones.We'll talk about actually how we manage costs as well. Just to sort of source informationand ask a few questions.We've rendered a UI around itwith some generative UIto actually show you what those choices are.
06:02Then we're going to actually write an outline for your deck.That's usually several more LLM calls.And we will style a template for you.So we have a library of templateswith different visual styles.We're going to use vision models,try to figure out which one's a good fit for you.We're going to do some cool search embedding things
06:18to sort of look at the space of thousands of templatesand find the right one.And even customize it if you said,yeah, but I want it red or blue or whatever.it is. Then we're going to once again show that to you and let you edit it and inspect it.We haven't even started making the deck yet. And already we've probably made something like 30 or 40different LLM calls to maybe four or five different models doing different tasks. Now you're going tohit that generate button and we are going to build up a presentation for you before your eyes.
06:45And there's actually a lot of pieces involved in doing that. We have to write the words on theslide. We have to lay out all the things. In our case, we generate a lot of imagery as well,whether decorative or more substantive,we're often making diagrams or infographics or charts.So actually all of those are now more callsthat are going out to different models,often in parallel.There's a huge performance consideration
07:06because if we did these one by one,it's going to be quite slowand you're going to basically leave before we ever finish.So we do a lot of work to make it all pretty speedy.And then in the space of about a minute,you're going to actually see that whole presentationstream in front of you and generate.And now we've made probably close to 100 different model calls
07:23and I would say probably 10 to 15 different AI models have been used somewhere along the way.And we're still not done because you probably won't stick with the first draft.In fact, we don't stick with the first draft.After we generate that first presentation, we're now going to have an AI vision model look at every single slideand look for problems that it can find.Misaligned text, the person's head got cut off in the image.There's weird contrast.Something's not padded right.
07:50It's actually amazing how many things can go wrong in something like a simple presentationthat a human eye can detect right away.Or sometimes a human eye can't detectbecause it takes a designer's eye.So not everyone can, but we can train modelsthat can cache that.So we go through this whole round of visual QA.We actually will do multiple rounds of fixing things
08:06based on what we find in that QA and then redo them.And then after all those rounds of fixing,we will finally serve it up to you.And now at this point,we've probably used 20 different modelsmaking like maybe 200 different callsalong the way to deliver what looks to youlike one finished output.
08:23So what you've just described to me is like a workflow in a particular niche.In this case, it's creative output.Totally.You started with presentations.You've now moved, you know, it's essentially a creative suite.Yep.You know, where do you see opportunities to create agent harnesses outside, you know,if you were starting a company tomorrow, where do you see an opportunity to create a vertical
08:51agent harness?What's an example of a niche that you see under-explored?That's an interesting one.I mean, just leaning on my experience,I think where we see a lot of value of making our own distinct agent harnessis when that problem looks very different from just plain old coding.In our case, what makes it different is that it's very subjective.There's a lot of judgment calls to make along the way
09:11where two different people will disagree about what good looks like,which means you want to have the human in the loop.And also where the person using it actually really cares about the specifics of the output.And so I would think about maybe those two traits as something that go into, like, wow, these are a good case for making your own agent harness.So just thinking loud, some examples are like, I know a lot of people are making games with Claude Code.And I'm really bullish on this idea of like the personal video game where you almost like vibe code it yourself.Coding agents are already really good at making video games.But I think they're often not good at actually sourcing the players' opinions on where it should go and where it should evolve.
09:50And if you think about things like Roblox or Minecraft,there's all these things where people are actually shaping the world for themselves.And usually it's like non-technical people who are not coders themselves.I wonder if there's a harness to be built there where, in fact,the whole harness runs in like a 3D world or something.I think people should really think broadly and try to, you know,in general, my philosophy of business is try to be different,try to do something that looks nothing like what everyone else is making.
10:17And for us, that's been a challenge because we try to be different,but then others, you know, copy us who do the same thing.And we have to keep looking for more and more different.I think that's, you know, it's not even just for you guys.It's anyone who's building anything in software today.Yeah.As soon as you catch lightning in a bottle,totally.
10:34People are copying you.But I think the thing that people, I think the reality is, they're copying where you were six, you know, three months ago, six months ago, right?So you as the founder have to be like, okay, where do I, what's my strong point of view?what's my thesis on where the world is going?Yeah.And how do I essentially create like a product studio where, you know, every two weeks,you know, I'm building a feature, zero to one, exploring a, you know, a new piece of this puzzle.I think that's totally right.
11:09And I think for anyone thinking about building their own agent harness, the really important question to ask is, what's my edge going to be?And there's a couple different types of edge you can have.One type of edge is what you just said, which is you have a distinct vision of the world, and you are just moving faster towards it, and it's not the one that others are focused on.I think another edge is you have some kind of proprietary data.You've got some information that other people don't.Maybe because you already have a successful platform or something.I think a totally different edge, too, is that it's not just software, because software is getting very crowded, and it's a scary market to be in.But, you know, there's hardware.
11:44There's like physical services.There's all sorts of real-life businesses to be made that incorporate AI in interesting ways.How do you, you know, physical services, like, can you elaborate on what you mean?Yeah, I mean, to be clear, I'm a software guy, so I don't know anything about this.But if I were not a software guy, I'd be thinking a lot about how can I integrate AI into some kind of real-life experience that people are having,where there's actual some kind of human exchange of value, whether it's like, you know, a person giving you a massage or a restaurant that's serving you a meal,or whatever it is, I think there are so many times of, like, real businesses that are not quite reinvented with AI yet.I don't know that they all need their own harness, although perhaps some of them do.
12:26Like another category where I'm seeing a lot of harness development going on is, like, legal tech, for example.I suspect medical tech is, like, a big one that has a lot of interesting, like, data lock and regulatory stuff going on.Yeah, I think legal was one of those where it was, like, obviously this makes sense.Yeah, yeah. Accounting obviously makes sense.I think the thing I would be thinking about is like, would I be building a legal tech AI agent harness today?Probably not.Yeah.Just because it feels so crowded.
12:57Totally.So I like when, you know, when you're talking about service businesses, even boring businesses in general, like agent harnesses for HVAC companies.Absolutely.You know, if someone wants to build an agent harness startup in a niche, how do they,approach designing it in the sense of like you need the key insight, you need the niche.One really simple answer about what an agent harness is, is an agent harness is just a bunch oftools that an agent uses to complete a task. And so I think the question to ask is, what are theunique tools that either I already have or I can build that are not already out there? And
13:38ideally, I think as a founder, you should always ask the question of what am I uniquely suited to dobased on my life experience, where I am in the world, the people I know, that sort of thing.And how do you think about AI smell?Yeah, totally.You know, because I think a good agent harness is going to kill that smell as much as possible.And I know you have a strong opinion on that.I do.I have been using this term AI smell all year.
14:05I've been talking about deodorizing, spraying it off, all that sort of thing.Maybe for people just don't know the term, AI smell is what we use to refer to this fact that whenpeople make work with AI, you can often tell they used AI. I think it shows up most in writingwhere when someone writes you an email made with Cloud, you're just like, you don't normallyuse that word. That's not what you sound like. And I noticed it a lot with AI images now, where Ijust like walk around in San Francisco and every cafe, every restaurant has a poster where I canjust smell like, that's nano banana, that's GPT image. I do this stuff all day, so I can like spotthem a mile away. And I think more or more people are starting to recognize that AI smell as it
14:41diffuses just through the economy and normal people. So for us at Gamma, it was actually a hugegoal of this launch. We had kind of been a victim of our own success as one of the early AI tools,where when nobody used Gamma, if you used it, your stuff would look really cool and stand out.But once everybody started using it, everything started to look like Gamma, and peopledidn't want to be recognized as using AI. And so kind of our whole goal of this launch we justpulled off was we need to rapidly expand our sort of creative landscape.The truth is there's still going to be AI smells,but there can be a thousand AI smells
15:14instead of one monolithic one,and we can help people jump to different little pockets.And we needed, this is another form of a harness, by the way.We needed to create our own automated systemfor constantly cycling through different creative looks.We actually use the analogy of fashion.Fashion has a smell too.You're like, I know that H&M shirt you're wearing,
15:33like I have the same one.And, you know, with fashion,there's kind of this dynamic where people are cycling through different looks.You know, right now we're into like really big wide jeans, but soon it's going to be skinny jeans again.And so we realized we had to actually ride that cycle and give people the tools to adapt to it.So is there an eval for this looks generic?You know, evel is such an interesting topic because I think evils work really well for things that are objective.Actually, if you don't mind, break down what an eval is.
16:02Yeah, totally.So an eval just means a, you could think of it almost like a software test.it's a hard-coded question that you ask in a way of judging it.So a simple example in our world of gamma is we often want the agent to ask questions when you do something ambiguous.So we'll give it an eval like when the user says make a deck about tigers,the agent should ask something like, is this for kids or for adults?Because there's very different kinds of tiger presentations you can make about that.And so we typically use evals to test out new prompts or new models.
16:36And usually you score them.can put a bunch of evils together into a benchmark where you say, oh, it turns out model B scores3% higher than Model A. Every time the labs come out with a new model, they're like, oh, Opus 7 crushes,like these 15 e-vals. The problem for us in a very creative domain is that evals have always beenhard for very subjective taste-oriented things like, does this look cool? And even more so,very personal ones like, is this me? Does it sound like something I would say? There's no oneobjective eval for this. And so, you know, we've tried to create some evals that are proxies for this,like does it follow instructions? Does it, for example, if you provide your brand guideline,
17:13does it stick to it? That's an objective question that we do test ourselves on. But we've neverlimited ourselves to just evils. And I think this is actually interesting insight, too, of how you cancompete in this world. Ultimately, what founders can bring to their domain is taste. They have theirown sense of what looks good. They're actually talking to their users and those users have one.And the less verifiable a domain you're in, like say fashion, the more I think there's room forthese unique perspectives. I think when it comes to clothing design, there should be 100different harnesses. I think, you know, Nike should have a different one than like Adidas becausethey have a different perspective on taste. Also, for us, because we have a super diverse user base
17:50who are all different, we actually rely a lot more on AB testing. We will actually run two models ortwo prompts at very big scale and measure across that wide user base, what's actually performingbetter? Because that captures subjectivity. And what's the metric that matters most there?You know, we have a bunch of different ones to give you a few examples. We look at things likeafter you generate, how much time do you spend editing what you made? Do you end up sharing it orexporting it? Are there signals that show your investing time? We also have a freemium product,so people will use our product for free. And then if they like it, they'll upgrade to paying.that's kind of my favorite signal because it's hard to fake.
18:25Generally, people only pay if they're really seeing value out of a tool.So we use that one as like a North Star.Sometimes we'll even look at like downstream retention.Like are people still coming back to the product a week later or a month later?It's obviously slower to measure, but it's also one of those hard to fake signals.The chart that's going viral this week is I think an A16C chart that says I think only 2% of people are spending on AI tools.What's your opinion on that?Yeah, it doesn't surprise me. I think in general, subscriptions are always a bit of a niche thing towards people that are doing some kind of valuable work.
19:00I don't think consumers like subscription. I as consumers don't like subscribing for things. I'm still always looking to jump on somebody's Netflix account or whatever it is.No, I will not share my password with you. Are you sure? What if I explain what an agent harness is?But I think it's different for work. I think for work, it's a totally different story where, you know, work subscriptions are what has driven, essentially the entire SaaS boom, the last like 10 to 15.years in software industry. And those things will keep being successful. And I think that same chartsays that when you actually zoom into professionals, the numbers are rising like crazy in termsof who's paying for AI. So that's certainly our main focus. And I think most people's main focusin AI is how to find those prosumers, the professional consumers, and how to break into B2B,where there's a totally different kind of spending going on. If someone wanted to build the gamma
19:46for X, what would you pick? Like if someone listening to this is like, wow, like, you know, I'minspired by what Gamma is building.Yeah.But I think that, like, you know, basically what you just described, but like for a different niche.Yeah.Anything coming to mind?It's such a personal thing because if, you know, if it was my thing, I'd be building it already.I think you need to do the thing that I don't know anything about, whether it's like Pokemon
20:14cards or fashion or sneakers or whatever it is.Like find your domain where you know what's going on and build it.And you personally, is there like a harness?that you wish existed, that doesn't exist, that you'd use tomorrow if it came out?You know, I'm really intrigued by this whole sort of like personal agent space.And I haven't gotten to use all the latest ones, but what I feel nobody's totally cracked is like work collaboration.I think if I could build a harness for anything, it would be Slack.And I'm a little disappointed that Slack themselves have not built the harness for Slack.
20:46What would that look like in your mind?So there's just this absolute flood of communication for anybody whose company uses.Slack or Microsoft Teams, of course.There's so many messages going through.You're constantly trying to stay on top of like, well, which ones?Do I actually have to do something about?I've seen them nibbling around the edges of this of like you can summarize 30 Slackmessages into one.
21:08But I want something that just like runs it all truly for me.And it learns me.It learns how I would reply before I reply.I want to be able to open any Slack thread someone's ping me on and have the first draftdrafted for me.And it learns you not by answering like a quick.or onboarding by actually like your action.By observing me.
21:26Yeah, by observing.I have sent approximately a billion slack messages.That should be all you need to know how I'm going to reply.Right.Even in the email doesn't feel totally cracked.I've seen Gmail has the little things that are like, oh, you know, like, yeah.So I'd love to go to the dentist on Thursday, which is not how I'd ever reply in my life.But I think there is so much more to just like represent me in a professional setting authentically to me.
21:49Like maybe that's what I'm really craving.And it's a little bit of what Gamma does is I want an eight.agent that can learn my voice.And of course, that's hard because it's personal.And this is why the agents haven't done it.It's really hard to make agents personal.So, you know, at Gamma, you're building something on top of other people's models.What's your advice for founders building on other people's models in general?
22:13I think there's no shame in building other people's models.We, you know, we have done this from the very start, often bouncing between anthropic, open AI, Google, models.and so much of the discourse has always been like, no, you've got to train your own stuff.You've got to fine-tune it because you need the moat.I have always felt that that's unrealistic for early-stage startups in particular.I mean, first of all, the money involved in training your own models is really hard.But also, the models out there keep on getting better.I rarely would want to be competing against the models that are out there.
22:42I think, though, it does change as you get further along in the journey, because you start to amass real usage.And, I mean, let's be honest, just amassing real usage is the main back.people should be focusing on. By all means, use the models that are out there to get thatusage in the first place. Once you do, though, you can start to do really interesting stuff.We've actually had great success, fine-tuning models to bring our costs down, and in some cases,even to start to do more personalization for people. We also have started to identify specialized tasksthat, you know, the big models are bad at. Like I mentioned earlier, this whole process ofQAing a slide and telling if it's good. We've actually not had great results using any of the
23:16models out there for it. So we are actually doing some of our own research to build models forthose specialized tasks. I think people should absolutely aspire to that, but by all means, use thegreat models that are out there. What about what about, you know, current, what are your currentthoughts on frontier models or cloud models versus open source local? Yeah. I think, you know,cloud models are underrated. Again, I think open source gets a lot of hype and excitement, but follow themoney. It's all going to the close source models because they're really darn good. So much work has beendone for you. And the great thing for all of us is,is there is intense competition in the world of closed models.
23:53I think in a world where, you know, like Anthropic had just purely run away with it,this would be actually a different storybecause we wouldn't begin to choose between like great open AI models,great Gemini models.But I feel like all these companies are just a few months behind each other.And for us in the application layer, that's the best thing ever.It keeps the heat on them to keep doing better and better.You've once said that you were surprised with how shallow your AI stack was.
24:19Is that still true today?You know, I would say that our AI stack is broad but shallow.What I mean by that is we actually use a ton of different models,but we often use them in fairly simple ways.Like we get so far with prompt engineering,with providing,and maybe your listeners have heard of the term context engineering increasingly,of just like providing the right context to the models.
24:40I really view that as kind of the whole game here.And it comes back to harness building.Can you get the right context from the user?Can you express the AI's output to the user in the right way?And can you distill all that into things that make the LLM's job really easy?When you make the LM's job really easy, you can use any off-the-shelf model and get good results.There's a big debate prompts or fine-tuning.Yeah.
25:02You have a strong opinion there?Yeah.I think for us, it's 98% prompts is where we get the value out of.Because prompting is easy and it's portable.So, like, and by the way, nobody's writing prompts by hand anymore.Like, Claude's writing your prompt forward.And so it's really easy.Yeah.
25:16And so I think most problems can be solved with a prompt.including breaking up a prompt into different pieces.And fine-tuning to me is more of an optimization on topwhen you want to do things like control costsor eke out that extra 2 or 3% of quality.When a new model drops,how quickly are you implementing it?Or are you kind of like waiting and seeing?
25:37Or how do you think about it?No, usually within the hour,if it's a model that I have reason to believe is going to be good.So, you know, not every model is interesting,but I would say for us, like any modelfrom Anthropic or Opening Eye is interesting, generally if they're dropping it. And I think I mentionedwe have all these different jobs in our harness. Some of those jobs, we use a like, you know,fable or astral level model for. Some of those jobs we use a little haiku Gemini Flash level model for.
26:04Some we use a little baby GPT Ludo model for. And so anytime any model comes in this class that canhelp us fill out that sort of Pareto frontier, I want to try it and see how it does. And generally,because our use case is so subjective, yes, we'll run some evals. But I just want to run it on asample of users.And we're very lucky to have a big user base where I can start a test in an hour and get datatwo hours later on how it's doing.So one of the things I learned, because I'm a co-founder of a design agency called LCA,and we actually worked on a project together at Gamma.
26:34And one of the things I learned working with you guys was you guys run a lot of experiments.Yeah.I'm curious about maybe one experiment that the result surprised you.Because you guys are just, it feels.like you guys are running a lot of a lot at a time. We run a huge number of experiments. In any given time,we probably have 10 or 20 running. And we're cycling through. Many of the experiments last less than a day.So walk me through like the process of an experiment because I think it's really, really interesting.Well, I'll walk you through one that I just started in the Uber ride over to talk to you. I just fired
27:10off eight different experiments because Claude Haiku 5.5 came out yesterday. We were really excited to get to use it.because, you know, I mentioned that we have a lot of these tasks that don't need that, like, superfable level model. They're quick and cheap, like picking a template from a library or sometimes, like,writing an outline based on something. And, you know, we've been looking forward to new small models we coulduse to test this. So this morning, I fired off eight different tests for it. And right after we gotoff this podcast, I'm going to check the results and probably cancel four of them and see how they did.And we're testing it for things like making micro edits on a slide or, you know, like revising an outline,whatever it is. In terms of the kinds of things we learn from these tests, often the great thing
27:53that we learn is that we can use a model that is half the cost of another model and get theexact same results. But sometimes we actually get the opposite finding. We find like, wow,actually the big, beefy, expensive model is necessary to achieve a certain task. Sometimes we evendiscover things like good friction. Like we actually just ran an experiment a few days ago of a fast modeof our product. And fast mode just means skip through some of those steps that I told you aboutearlier, like showing you an outline of your presentation or asking you questions up front.And the hypothesis was, of course, if you remove friction in the process, then people will getthrough it faster and get to that sort of magical value moment more quickly. I was really rooting
28:34for this test. I wanted to see it win, but it did not win for us, actually. It turns out thosespeed bumps we had along the way, like seeing your outline and reviewing it, you know,choosing the kind of high-level arc of your story,answering those questions led to much higher satisfaction ratings.Even though it was slower,the people that got through felt more invested in their output and did more.And they actually converted to paying at a higher rate.But I think what's also cool is a test is never just one answer or another.
29:02You can dig a little bit deeper.So we actually shipped a version of fast mode for brand new users on mobilebecause on mobile you don't want to be typing a bunch of answers to a question.I just, right before this, I posted an X article about marketing engineers.My thesis is that marketing's changed.If you think about the madmen era, it was like, you know, you did one big commerciala year in the 50s and 60s and 70s.Then it became in the 90s and 2000s, the digital marketer, maybe a few more experiments.
29:36Then we had the growth hacker era, a few more experiments.per week. I think in 2022, I would say, AI-assisted marketer, more experiments. And, you know, I think we're nowentering the age of the marketing engineer where you're, you know, as a marketer, you're runningdozens of experiments per day. How are you able to do that? You're using AI agents to go and dothat for you. I'm curious, your POV on that. Do you think that marketing has changed and like,where do you think it, where do you see it going? It's funny because before I worked at Gamemy work at a company called Optimizely.This is where my love of AB testing came from.
30:14We were AB testing as a service.And a lot of our vision was about personalization.This is idea that as it gets easier to change your website,we are going to create a hundred little micro variations of it,and everyone will get served a slightly different version,and that will optimize conversions in all these interesting ways.Surprisingly to me, that vision, I would say largely has not come true out on the internet.I would say websites are still mostly one-size-fits-all.
30:39so are mobile apps by and large, except for these specialized areas, which are like hyper-personalized and succeeding.When I say specialized areas, I mean things like your TikTok feed, your Instagram reels.We know that companies are spending billions of dollars to eke out every last, you know, a dollar from you in those surfaces.I heard that McDonald's is testing personalized pricing.No way.That sounds like a backlash waiting to happen.That's why I saw it.Yeah, yeah.
31:06But I think maybe that McDonald's example illustrates what.why this isn't happening. First of all, I think there's a real societal allergy to this kindof thing in the same way that if you found out that we both bought the same airline ticket,but they charged me five bucks more because I was in San Francisco. Well, I think in general,like even surge pricing, like, I feel like, when I had surge pricing last night coming homefrom dinner and I was like, I was like, oh, you know, like, it's not a good feeling.Yeah, yeah. And, you know, I actually think in this age of hyper-efficient, you know, like performancemarketing. Actually, what's working is companies that really doggedly invest in their brand.
31:42This Apple mindset of just protect the brand at all costs. And so I think that trend is going tocounter some of what you're describing of everybody's going wild with their marketing agents.That said, I still buy the thesis that like the marketing engineer is rising. And I just love thatterm marketing engineer because it sounds like a contradiction, you know, much like GTM engineersounded like a contradiction. These are supposed to be different departments. But I've loved seeing howeverybody's skill sets are just blending.We're seeing this with Gamma, actually,where people are more and more using Gamma,
32:12not to just make one presentation here or there,but creating these really cool automationswhere they're doing things likeevery time a new deal opportunity is created in Salesforce,let's automatically create a pitch deckbased on the call notes that have happened.How do you set that up?We've actually been building kind of connectors
32:29where you can automate more of it.So you can just plug into Salesforce.But I would say there's still a fair amount of plumbing goingon where people are using maybe a tool likeN-A-D-N or Zapier or something totrigger it all together and wire it.That is a good example of a thingwhere it's still the domain of specialists,
32:45I would say. There's like these really smartpeople or agencies that know how tohelp a business like wire things up.I think there is still a business opportunityin taking these practices that are doneby the top 1 or 2% of companieswho are really savvy about engineering itand just turning it into a super easy thing
33:02that everyone does. Another example of this, to go back to thepersonal agents is like what's happening now with tools like muse and instinct. You could do someamazing stuff with OpenClaw or Hermes if you could get your Linux box set up in the right place andavoid getting hacked. Personally, I skipped out on that whole trend. I was like, I'm a busy dude.I don't have time to set up my open claw. I'm going to wait a few months for them to just make iteasy for me. And now I got Muse and Instinct blown up my phone because they made it easy for me.Right. So you don't have a Mac Mini running. I never got my Mac Mini. I'm ashamed to say as an AIfounder. Going back to marketing engineering, in your mind, what does a top 1% marketing engineer
33:40look like? Well, one thing that comes to mind is they're really a marketing engineer data scientist.So they are like ultra data driven and grounded in what's performing. And this is something that'sgotten easier than ever, like absolutely clot as my data scientist. And so like I would say now everybodyon our team is constantly armed with like very fine-grained analysis of like failuremodes and opportunities.They're looking at traffic on our site.They're finding gaps.They're, you know, like I look at our, you know, the person we have on our team who's
34:10closest to a marketing engineer.And he is just constantly thinking of new ways to do SEO, ad creative, at just anunbelievable scale for one person.You know, they're like, they've gone from being like individual creatives to people whoare running this army of agents.And they are just in this high level supervisory role.Yeah.I mean, it's basically a department.
34:31It's department.Yeah. It's like we're getting to the place that if you're a top 1% marketing engineer,your surface area of what you're working on isn't just, well, the service area is huge.You're looking at SEO, GEO, you're looking at social content. You're looking at paid ads.You're looking at pretty much like all creative brand, you know.Absolutely.I think that's what's interesting to me because I think that the people that figure,out how to become a top 1% marketing engineer have an unfair advantage around getting their product
35:09seen.And the other 98% are quickly becoming dinosaurs if you can't keep up with the people in the marketthat are doing this.I've kind of had this idea that all of us over the last maybe like year or two, at leastall of us who are using these tools, kind of got this rapid series of promotions.We went from individual contributor to manager to director to like VP.We now have these like layers and layers of agents on.us in a pyramid. And it turns out that job's actually kind of hard. Like, there's a reason that
35:38like few people do it because you are sort of constantly barraged by information. You're having tomake so many decisions. You kind of are like overwhelmed and confused by all of it. You needtools to even make sense of it all. You also need people to help you make sense of it all. You kind ofneed your assistant and you need your chief of staff and you need all of these things. And so all of usare having to rapidly upskill to essentially how to manage a thousand people.And building that skill is one of the most important things we can all do.Also, I think building tools to help the average person become a VP of their own life is stillunderexplored, going back to my Slack idea.
36:16I also think, you know, experiments in general, it feels like building a startup todayor building a business in general is, it's like you have, it feels like almosta little more science than art.Yeah.Now, in the sense, like,the ones who are going to winare the ones that have the experimentation muscleand just trying a bunch of different things.
36:40That's why, like, I love talking to youbecause it feels like that experimentation muscle is,I mean, that's your, like, your thing.What's one thing at optimizelythat you learned that every AI founder should, you know,would be good for them to know.That's a great one.You know, one thing that comes to mind for me,
37:06I think from years of helping other companiesdo experimentation, was that so much of it just comesto how many at-bats you have.It's all just sort of a numbers game,just like sales, just like dating,just like kind of anything else in life.You've got to really try a lot of different stuffversus expecting that you'll run one big test
37:24that magically unlocks your business.I like how you called it a musclebecause I do think it is.First of all, because it's trainable.You know, yeah, maybe some people are bornwith more muscle than others,but if you put in the work,that muscle's going to grow.
37:37I also think, like,your first experiment's not going to work.Totally.Just like, if you're trying to get fitand you're trying to build muscle,you have to just show up to the gym every dayand you have to put in the work.Absolutely.
37:50And then two months later,then you might see a six-pack, right?But you can't expect it on day one.I think that's the hard partabout being an experimenter is like there's like a trough of sorrow that you have to go through.Absolutely.And I think what a lot of people end up doing is they're like, you know what?I'm just going to go with my gut.
38:10Absolutely.By the way, the same is true of things like posting on social media, writing a blog, sending cold email outbound.There's a trough of sorrow in all of it.And I think all success in life comes from swimming through some kind of trough of sorrow.What's something that's underhyped right nowThat you're likeYou know what?People are just not talking enough about this
38:33I was talking with a friend of mine about voice AIThat felt like underhypedRight now people are talking about personal agentsAnything that comes to mindMaybe it's a lame answerBut I actually think AI image models are underhypedIt's a lame answerbecause we've been talking about image models for years.
38:55Ever since stable diffusion came out in 2022,people have been making cool stuff.But I think, ironically, as they got good,people stopped talking about them.As they stopped messing up in so many waysand just became sort of like part of the fabric,we sort of like forgotthat we have this unbelievable technology
39:13that can make anything.And I think if you look at the way people are usinglike chat GPT images and a few other things,it's kind of just the same.three or four use cases over and over.You know, they're making like maybe the menu for their small business.They're making like a logo for their dad's side project.There's like a handful of things they're doing.
39:35But these models are capable of so much.And we have a really cool vantage point to this at Gamma.We didn't talk as much about images,but we use probably like 10 different image models as well for different tasks.And the pace of innovation there has been so much.and just this sort of like unexplored capabilities.For us, it's things like making beautiful diagrams or infographics to go in a presentation,making cool little textures that can be the background of a presentation,
40:02making custom icons.Like all these things are possible now.And most people just don't realize it's there.And I think you can pay attention to almost like any domain that's creative and find ways that you could be using these things.And especially the average person just has no idea.You know, even one that I think is like probably many AI native people are doing, but most people don't realize you can do, is using AI image models for like clothing and like home decor.You know, like you found a new couch you want to buy on Amazon.You can actually just chat GPT to say what would that look like in my house and then say like go find me a few others and visualize those two.
40:39Like so many of these things are just waiting to be more deeply productized.Do you think there's an opportunity to build vertical products there?like just focused on, you know, interior design, for example,or do you think the big model companies are just going to own that use case,so it's not even worth trying?I think there is so much room for these vertical use cases,and it's all about understanding the specific needs of the people that do them.Like, let's just take this furniture one, for example.
41:09I actually just bought a couch recently, so I...Congratulations.Thank you.It's really comfy.And I went through this, and,it was so interesting, like, trying to do this. And it turns out the AI image models areamazing at showing me what the couch will look like. But they're not actually showing me exactlyhow it's going to measure, right? And there's a huge difference between, let's say, 68 couch
41:29and a 58-inch couch if your walls a certain size. So, like, could you actually show me theunits in the display? That's a great example of a harness, right? The models are actuallycapable a lot of these things. One way they're capable is you could imagine doing a thing whereyou render the couch in my room, but then you use some good old-fashioned programmingto draw a line across in pixels and measure how long it is and see if it actually came out of the right sizeand prompt the model to tweak it to get exactly right. I don't see the labs doing that anytime soon.Or prompt, hey, like this couch isn't the right couch for you. Yeah, totally. But here's another couchfrom restoration hardware that I think would be perfect for these reasons. Totally. And dude,
42:09what I learned about couch shopping is there's hundreds of websites that sell couches and you'll have toclick through a lot of little buttons on all of them. I just,I ended up on Google shopping.It was not a pleasant experience.Maybe somebody's already built the great product for this,and I missed it, but I haven't seen it yet.I think there's been examples.I think Levels I-O. has like a...
42:26Yeah, he has like an interior, I think it's called.Yeah, yeah.But I think there's still room.Totally.I mean, like, I think, especially as the models are getting better,the image models are getting better,the concept of a harness is getting more and more mature.There's more here.
42:43I think it's safe to say that also the consumer experience today around couch shopping or furniture shopping is an optimized.Totally.And what's cool about that specific use case is it's high ticket.Totally.So, and there's margin, right?Like if, for example, like, I don't think it's crazy to do a deal with restoration hardware where they're like, we'll give you 5% commission if you recommend.sell us more couches. Yeah, absolutely. I think the other version of this, when you think aboutwhat will survive competition with model companies, and we're certainly facing this question
43:22constantly at gamma, there is the pure consumer version, which to use the couch analogy is like,help me buy a couch. But then there's the like pro sumer version, which is, hey, I am an interiordesigner. Maybe I'm a part-time interior designer, but I also have like two kids at home. And that'swhat I do. But I, you know, I do it by the hour. Those people,need a tool and they will seek out the best. And if it helps them do their job better, they willhappily pay a subscription for it. Then there's also the true pros. There's restoration hardwareitself has people who work there. What tool are they using? I bet it's not just chat GPT to do all ofthis. I think there is actually an opportunity to build companies for those. And I do suspect that
44:01highest tier of professional, the bigger companies, is where you can really build something durable.Yeah. And I feel like it's under explored. Yeah, it is. Because the narrative I think is, there'sthere's not that much opportunity creating vertical SaaS or vertical AI products,which is unfortunate because I think that there is.And it's a big world out there.That's maybe one other thing we've learned from the gamma journey.Our user base is very broad because if you just think about how many people on earth make presentations,it's a lot of different kinds of people.
44:36For us, less than a quarter of our revenues in the United States,It's spread across many different job rolesfrom like consultants to teachersto marketers to salespeople to lawyers.I don't know lawyers make presentations.Turns out they do.And, you know, the more you can tap into little niches,you know, like in many ways, a big company
44:57is just a thousand little niches piled on top of each other.So if you can find something that cuts acrossand you don't have to start broad,you can start with just one,but then look for the adjacencies you can tackle.100%.And you actually don't know where it's going to lead, right?You didn't realize, like, hey, lawyers are creating presentations, and then you end up, who knows, you end up creating something specific for lawyers.
45:18Absolutely.And then all of a sudden, that becomes a huge part of your business.Yeah, totally.And that's why I'm always like, get started.Like put something out there.Put something out there.You got to just find out.That goes back to experimentation.
45:31Experimentation doesn't just mean run an AB test with like 10,000 users in each arm.It usually just means put up a landing page, get a product someone can try, which has never been easier.If you, I know you're not leaving Gamma, but like, let's say you were to leave Gamma tomorrow.What's something, what's a company you would start to build?Oh, that's a good question, man.You know, it's, I just love circling this space of, like, enabling people's creativity.I think there's other kinds of, like, creativity that, like, intrigued me, though.Like, as crowded as the spaces, I'm so intrigued by just, like, vibe coding as I practice and just the way.
46:08I had this experience maybe as somebody who knows a lot of other tech people and lives in San Franciscoearlier this year where every time I would catch up with a friend I hadn't seen in three or four months,I'd be like, what are you up to? And they'd be like, check out my app I made. It sort of took the world by storm.I want to know maybe what's the next vibe coding or what's the next frontier of vibe coding?As people can make anything, where will that take them? And so within that, I'm certainly intrigued by video models finally,getting good. I'm intrigued by voice models getting good. Like, what's the version ofsome kind of thing you could call and it would do something for you? Games are so interestingto me. I just think, like, the video game industry is right for some kind of disruption,
46:52but I haven't cracked it yet. I don't know. I think on the video side, it does feel likewe're hitting like the vibe coding moment for video. Yeah. Opus 5.5 has, is obviously... It makes somecool stuff. Make some cool stuff. Yeah, so I think the question is, in a world where anyone cancreate video, what are the types of videos they can create?And then I'm also interested, it's like, as a founder, I'm like, okay, in a world whereanyone can create, what are some startup ideas that I can create?Totally.On top of that.
47:21And that goes back to agent harnesses.Yeah, absolutely.Everything goes back to agent.Everything goes back to the harness, baby.Dude, thank you so much for coming on the podcast.Is there anything you want to leave people with, one thing?I would just always encourage people who are thinking of starting a business to know,know that it is a long and sometimes lonely journey. And because of that, you really have to love
47:43the thing you're working on. One of my kind of rules for myself before we started Gamma, andit would stick around if I ever did another startup, is it has to be something that I coulddog food, that I could be a user of, and that I could believe in. And to be honest, that actuallyrules out a whole lot of boring businesses, but it was good for my soul. I just needed to havesomething that I wanted to spend hours every day in. And so I always encourage people to dig deepand think about what is your passion and the thing that only you know, the thing that you want tospend eight hours a day on that when you look around, your friends don't want to spendeight hours a day on and zero in on that as your North Star. I think that's the important point, too,
48:23around like the thing that you want to spend eight hours a day on that other people don't.Totally. Because like you might be really into sports or something, but a lot of other people areinto sports and that might be very competitive. I think like if you're if you're choosing a startup ideaor choosing a business, I always say like marry the niche date the product. I love that.Like fall in love with who you're trying to serve. Yeah, yeah. And because we live in an age where you,you know, you're going to be iterating on the product so many times and it's going to look sodifferent. But if you love the niche, and that's why I always encourage people to build media,build a following in that niche because then you can sell them a bunch of different products.
49:06Now you've built in audience. I think that's totally right. You know, Jeff BeBe Stubbered on the Vision, flexible in the details.Well, I'm glad I'm in good company with Jeff Bezos.You sure. Thank you. I appreciate you. Thanks for having me. This is great.This is awesome.
There aren't comments yet for this episode. Click any sentence in the transcript to leave one.