The Peterman Pod - OpenAI Codex Tech Lead On How His Career Grew And How He Uses Codex | Michael Bolin
Episode Date: March 9, 2026This is Michael Bolin, the tech lead for the open source Codex repository and a former distinguished engineer at Meta. We talked about his career path, how OpenAI engineers use Codex and the differenc...e between research-led vs engineering-led company cultures.🔸 My keyboard project: https://read.compose.llc/p/our-keyboard-design-reveal𝗣𝗼𝗱𝗰𝗮𝘀𝘁 𝗹𝗶𝗻𝗸𝘀:• YouTube: https://youtu.be/hN5ZFzWFhhg• Apple: https://podcasts.apple.com/us/podcast/the-peterman-pod/id1777363835• Transcript: https://www.developing.dev/p/openai-codex-tech-lead-on-how-his𝗧𝗶𝗺𝗲𝘀𝘁𝗮𝗺𝗽𝘀:00:00:00 - Intro00:00:56 - Chickenfoot00:02:45 - Working at Google00:06:34 - Overhauling Facebook's build system00:16:36 - Rewriting Facebook's IDE00:26:01 - Struggles after Principal Eng (E8) promo00:28:39 - Building a virtual filesystem for Facebook00:35:47 - Delayed Distinguished promo (E9) and learnings00:39:56 - Joining OpenAI00:43:05 - Research-led vs engineering-led cultures00:44:53 - The story behind Codex00:51:00 - How he uses Codex00:57:00 - Why Codex's harness is open source00:59:50 - Top technical book recommendations01:05:02 - Why deep technical skills are still valuable (for now)01:11:07 - How to start projects well01:14:27 - Advice on writing better and career planning01:17:06 - Advice for his younger self01:19:10 - Outro𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗠𝗶𝗰𝗵𝗮𝗲𝗹:• His personal blog - https://blog.bolinfest.com/• Twitter/X - https://x.com/bolinfest• Threads - https://www.threads.com/@bolinfest• LinkedIn - https://www.linkedin.com/in/michael-bolin-7632712𝗪𝗵𝗲𝗿𝗲 𝘁𝗼 𝗳𝗶𝗻𝗱 𝗥𝘆𝗮𝗻:• Newsletter: https://www.developing.dev/• X/Twitter: https://x.com/ryanlpeterman• LinkedIn: https://www.linkedin.com/in/ryanlpeterman/• Threads: https://www.threads.com/@ryanlpeterman• Instagram: https://www.instagram.com/ryanlpeterman• TikTok: https://www.tiktok.com/@ryanlpeterman
Transcript
Discussion (0)
The questions that you asked the agent is going to affect the quality of the thing that you get out.
This is Michael Bolin, tech lead for the open source codex repository and former distinguished engineer at Meta,
and we talked about how his career grew.
So I was like, all right, hackathon, totally going to like make a new build system.
You know, like relatively cookie, had something that was dramatically better, like at least like twice as fast.
He shared about how open AI engineers actually use codex.
I almost feel a little bad writing code by hand.
What percent of your code would you say is human written?
You know, because every time I sit down, I'm like, should I write this?
I mean, the answer is almost always no.
And we discussed AI research versus big tech engineering cultures.
What do your thoughts on that research-led culture versus engineering-led culture?
I mean, it's certainly an adjustment, right?
I mean, I think of anyone who comes here and says otherwise it's a liar.
Here's the full episode.
I was looking deep into your website, and there was something, most everything I could find more information.
Well, there's this one thing that you seem to be pretty excited about the time, but I couldn't find any information about it because all the links were dead.
What is chicken foot? Oh, man, that's strangely relevant because it was my master's thesis project.
It was a Firefox extension, probably one of the very few that was written in JavaScript for Firefox as a thesis project.
And it was actually a little coding tool in a sidebar Firefox.
And it was like a little REPL.
It was called end user.
The idea was end user programming for the web.
And so there were functions like enter and click and things like that.
And you'd say enter.
You know, pass a string arg and it would like find the box and you'd say click search or whatever.
And and you know, a lot of the work was all these heuristics that we built under the hood that have you said, you know, enter first name,
finding the like the words first in name and finding the text box that was closest.
And then like using that and then through JavaScript like,
using that as the input.
And I just think about that now,
because it's really funny, we did a lot of work.
And that's,
you know, a lot of what these agents are doing right now, right?
It's like similar.
But now actually truly in natural language,
like not in this little like JavaScript repel.
Oh, interesting.
So it parsed the front end and had this repel
where you could say, I don't know,
find the first name field and, you know.
Yeah, we'd use like the, you know,
the accessibility tags and alt, alt text for images.
And we, you know, and it worked well.
It was really good at Craigslist, actually, because that's one of the simplest websites you could use.
But I had friends who, like, made, like, automated tasks and, like, made, build money off of the things they automated with this tool, which is really fun.
When you entered the industry, you were really excited about going to Google and specifically working on Google calendar.
Yeah.
So what drew you to Google and what was that like?
Yeah.
I mean, you know, so I, you know, got online in the 90s, right?
I remember, you know, browsing the web.
And it was at a time you'd go to, like, you try five different search engines to, like, hopefully find the thing that you wanted.
And I, and it's funny because I distinctly remember my roommate in March of 2000 saying, like, hey, there's this site that this search engine that looks a little bit better.
I think it was still Stanford.comgoogle.edu.
And I was like, oh, this is better.
And, like, you know, and then you started reading.
And they were just so different, right?
Yahoo was all, like, very cluttered and, like, they tried to be very sparse and were more.
I think principled about it at that time.
And then, you know, like a lot of things, you start seeing people who, you know,
who gets a job there.
And you're like, oh, man, they're taking really good people.
Like, I want to work with those people.
And I felt like they just, you know, really got the web.
Like, especially at that time when like Microsoft didn't, around that time,
Microsoft killed the Internet Explorer project altogether.
And I was like, this is the portal into the web.
And you guys are, you know, dismantling it.
Whereas, you know, Google was way more, you know, web forward.
And so between that and, like, the quality of engineering and the,
and the impact they were having, you know,
certainly a place that I was really excited to go to coming out of school.
And what was the culture like there at the time?
Like, I think in some of your writing,
you talked about like products versus infrastructure.
You know, a lot of companies, especially once they get big like that,
you know, it's kind of like whatever they were good at first is,
I feel like the founders always have a soft spot for, right?
So certainly information retrieval and infrastructure, right?
were like key to, you know, growing that, that company.
And whereas I was drawn there at the time because, like, you know, they put out things
like Gmail, right, which were vague, right?
But it didn't, it still, I think, didn't have the same cachet inside the company, right,
as the search and that sort of thing.
And so, you know, when I was working on it, calendar, it was still, like, mostly consumer.
It was, like, kind.
It was also sold to enterprises, right?
But like, we were not the ones making the money.
Like, we were probably a call center, right?
At that point in time.
I saw eventually you ended up leaving Google.
And from your post, it looks like it was pretty bittersweet.
So what left you to leaving Google?
I had been there for four years.
And, you know, I mean, realistically, like, investing was a thing.
Like, finances definitely changed after that four-year period ended.
But, you know, also it was just tough.
Like I had a, I guess, a bad habit at that point of working really hard on things that like really were important to me, but maybe weren't important to Google, right?
So like I worked on calendar.
That was pretty important.
And then it worked on Google tasks, which was like a very small feature within calendar, right?
Like probably at least two to three orders of magnitude fewer users, right?
But I was really passionate about it.
And then I was really passionate about the like the JavaScript infra and the closure that that Sweetie
of JavaScript tools.
And, you know, it's great.
I mean, I enjoyed, and I'm proud of all the effort that I put into the things.
You know, then I went and wrote the book on closure because I was so motivated,
but career-wise, maybe not the best move, right?
You're like, but I'm doing all this high-quality stuff.
And you see other people getting recognized.
They're like, but I'm working so hard.
You know, like that's, you know, kind of like the harder, not smarter type of mistake.
And so, you know, it's kind of like, I should go.
somewhere or try something else and see if like the things that I'm excited about or the things
that like the company I'm working at is most excited about. Later you came back into big tech and you
start working at Facebook. I understand that you were kind of like a JavaScript expert at the time
and then one of your first big projects at Meadow was kind of build tooling in the Android code base.
So I want to know the story behind how you got involved in that. So at the time it was like
Facebook's going to make a phone, right?
Actually, there was some failed projects,
but it was like this time it's really going to happen.
We're going to partner with HTC.
We're going to like fork Android and, you know,
and do some stuff.
And, you know, so that seemed super exciting, right,
as a person, you know, just coming into the company.
And I, you know, I've done like quite a bit of Java.
I was more JavaScript.
But at this point, this is also where
you know, they called it
what, FaceWeb, the version of like
they kind of like put HTML 5 Facebook
on the phone and like that was clearly
not working and
it was clear that like
you know, mobile was going to be the future.
That was kind of make or break for the company.
And suddenly,
you know, a friend was like,
hey, like I know you really like JavaScript
but like you should
really pick Java or Objective C
and like get good at this if you want to be a product person.
And that was really,
really great advice.
And so I was like, well, I like Java.
I don't like object to see.
So like, let's go.
And so that's how I, you know,
found myself on that project.
And I, you know,
we had a very short timeline.
Because unlike almost every other project,
there was a hard deadline.
Usually, you know, you ship when it's ready.
This was like, nope, we got to send a bill to HDC
because they're going to like burn it into phones
on like March 1st or whatever it was.
So, you know, it was really a scramble.
But and the original Android code base was, or a bunch of it was also like inherited from a contractor that Google paid.
Like, it's like Facebook didn't want to make like a native app and they paid some guy.
And then it was like the app in the store and they're like, here, the code's yours now.
And we should have thrown it away, but we didn't.
And so a lot of things that were frustrating, but also like iteration time, right?
I think for everybody, if you had been a web developer for such a long time,
you used to this edit refresh, and then, you know,
like the Android build system was rough, right?
It was just, um, some build tools and ant and it, like, there were no,
there was no way to modularize it out of the bot.
Like, we had to hack it up just to even have like four modules or five modules or
something like that.
Um, and it was so painful to get anything done, uh, that I was.
I was like, I need to fix the build system.
I was like, I know I've done a lot with Java.
I know it's not like fundamentally this slow to like iteratively, you know, build this
sort of thing.
And so, you know, Facebook has this hackathon culture.
So I was like, all right, hackathon, totally going to like make a new build system, you know,
like unceremoniously in the style of Google's build system.
And there's actually another build system called FB build that was already a different mirror.
of the Google build system as well.
That ones were written in Python.
I only did C++ plus.
But yeah, and I was like, either I'm going to make this work
and then I'm going to be happy to be a developer
or like, I don't even know if I'm going to make it here
because it's just going to be like tearing my hair out, you know,
working on this thing.
And if you didn't fix this, you would have quit the company.
Well, I was, I mean, it was like, or I'll at least switch projects or I need to
a way to find a happy place, right, and come at work every day.
Like, I came. I want to write code. I want to do work. I just want to, you know, try to do my
best work. And so it was funny. Actually, I give people credit. A lot of people told me what I was
doing was a very bad idea. Almost everybody, except one person. I was a senior Android engineer.
But nobody said no, you know, whereas I felt like at Google, like people said no more.
And so I just rolled with it.
And, you know, I quickly, relatively quickly had something that was, you know,
dramatically better, like at least like twice as fast or something like that.
And so that, you know, it turned a lot of heads.
And people were like, okay, you know, I guess we, yeah, we'll go with that one.
Right.
I think the interesting thing to me is it seemed like all the odds were stacking.
A lot of people who are an engineer noticed the same problem.
would not have chosen to start the project because there's an existing one.
And then also Google's got some competing one.
Maybe you're not going to beat that one.
What gave you the conviction that your project could beat the other ones and become the default?
Yeah.
I mean, I guess a couple things.
I mean, one, like I said, I, you know, I had done other Java stuff.
I was like, like, this shouldn't be that slow.
Or just as a software engineer, like, this fundamentally should not be as slow as it is.
And there, most of the, actually, most of the pushback was of the nature like, well, if we deviate from the standard thing, you know, we're not going to be on the standard thing.
And what if it gets like 100x better next week and now we're stuck on your thing?
Which is really funny when you think about it because there's like so many things like, you know, making their own, you know, P.TB virtual machine at language.
Like they've certainly embraced doing their own thing many times why this one was different.
I don't know. I mean, it was just, I guess everything about mobile was like, how are we going to make this work?
I guess there was just like a lot of fear and, you know, again, like timelines. It's like we have the senior person, like why it seems like it sounds like they're going to go off and do a science project. We have a hard deadline. Like is that the best use of our time.
But, you know, it worked out. It worked out. I will say another thing too was I also tried to to couch it a little bit. I was like, hey, I'm just I'm just trying to make an answer.
build system. I'm not trying to take over the company. I'm not trying to change anyone else's
project, right? Because I certainly felt that that was going to cause more, I was going to invite
more friction. And so, you know, I knew that it, I tried to make it in a way that it could
support more of the company, but I never, I never pushed it. And so I was like really heartened
when the year or so later, the iOS people were like, hey, our build system sucks. Can we, you know,
can we use buck? I was like, yeah, sure. Like, I don't.
I don't, you know, come on board.
Yeah, that's also another interesting, because you came in and you didn't have a whole lot of credibility.
You were just hired.
And then you're trying to build something and get every, and also everyone's saying, don't do this.
And then you have to convince them, hey, this is the right direction.
How did you influence that change without any credibility?
I mean, I borrowed some.
There's one senior Android engineer, you know, who was also ex-Google, John Perlough.
and he was like, I think he said, right, he's like, you're going to do it, just do it fast before, like, anyone gets, you know, like Kansas Zee or something.
He's like, well, I'll support you if you, if you get this done. So he was one of the first people, and he was a very prolific coder.
So he was, like, genuinely very happy to have a faster dev cycle. So he was one of the first people to, you know, give it kudos. And I think that that helps a lot also. But I will say, you know, I made a big, some big mistakes on my way in there.
mentioned like about having no credibility is like you know i came from google and i was like oh this is
where you know the ex bell labs people are like all these like great people and then you go to like
facebook you're like oh it's a bunch of college kids how can they possibly know what they're doing right
and you know i made a few times certainly i made the mistake like well at google we did it this way and
people were like we really don't care and you know and they were and they were right and and
most cases right that like just because it worked there didn't mean it was going to work here
One last thing on this topic is, I mean, what you built was way more performant than anything else by many multiples.
What's the intuition, like the technical intuition behind what was done and what you did that made it so much more efficient?
I think the big thing was that I just sat down and looked at the tool from Google and I was like, what is it actually doing?
Right.
and where.
And I think the big thing is that the Google one would kind of just,
if you changed anything, it would just start over from scratch.
And so that's why it was so slow, especially for incremental builds.
And so then I started really understanding, like, okay, but what depends on what?
Because there is these, I'm sure, like, Android resources, which has like a very bespoke
thing.
And it was complicated.
And because it was somewhat complicated, I think that's probably why it just by default.
I blew everything away and started over.
But once I got in there and I was like, oh, okay, you know, we can take advantage of if these things don't change, then, you know, we can cache this result from this step. We don't have to redo it. And like suddenly, you know, that made things like quite a bit faster. And then also just even supporting the idea that you could have more than like the four modules that we had. I mean, I think I think every time someone added the module, you know, they had to add like 200 lines of XML of like Ant build script that like nobody really understood. And so it was not. So no one, no one.
No one wanted to module-larize anything, right?
You didn't want to be responsible for that.
So a big thing with Buck was making it,
that it was like a lot simpler to add a new module.
And so then that also meant we ended up with more modules
and then builds are more incremental as a result, right?
So it's really a change in the mindset.
It was less redundant work, basically.
Yeah.
After you solved this build problems, I guess,
in Android builds.
And obviously I went later to further parts of the company.
I saw that you started to work more on like the ID,
What was the problem that you saw in IDE's that made you want to get into that?
So, yeah, I took, I did a brief stint after Buck on Messenger, iOS Messenger.
So I was like, okay, I've done Android, maybe I should actually branch out, even though I don't like it.
And I still didn't like it.
People only realize in Objective C, there's a thing that happens for a long time now called ARP, like the
automatic reference counting.
Like nowadays, the compiler injects it,
but you used to have to like actually add code
in Objective C to do every like reference count
like explicit.
Yeah, or like every reference that you created,
like you would have to do it.
And the and most people have never seen this type of code,
but the iOS messenger code that came in through acquisition
was so old that it was still written in this other way.
And it's, it's incredibly painful.
I guess now you'd have like codex cleaned it up or something.
But at the time, we just sucked it up.
And just Xcode just didn't feel right to me.
I didn't like header files and implementation files.
I still don't.
And also, I would say, for both Android and iOS, you know, Facebook always had like the biggest app, right?
We had like one app with every, with every feature in it and we, you know, ship it.
As opposed to Google, like they'd have like a drive app and a, you know, a sheets app or whatever.
But they also owned one of the other.
of the platforms so they could put 20 apps by default on there. And so what that meant was is that
Facebook was always hitting the scaling limits of every mobile developer tool before everybody
else, which was painful, but, you know, actually as a DevTools person, it was kind of interesting
because we got to solve problems that nobody had solved before. And like, you know, not just as a
science project, but because it was like real, you know, business value. And so, you know, Xcode, similarly,
we talked to Apple, we like, hey, X-Code.
because not really scaling to our project or like,
your project's too big, you should make it smaller.
Like that was kind of the feedback that we got from them.
And so it seemed justifiable to like go and build an editor.
It was similar thing.
It was like, I was like, what is, you know, what is an ID, you know, doing?
Right.
It's like, it's talking to clang, like the language server and that sort of stuff.
I was like, we could build like a nicer shell on top of that.
You know, then we had started by that point, um, deviating from Git and switching to
Mercurial as a company.
I was like, no one's going to support Mercurial out of the box, right?
And so, like, that should happen.
And then, like, oh, actually, if Buck is going to be our build system,
then, like, we really, like, you know, like, that's,
X code is never going to do all of these bespoke Facebook things, right?
So it seemed justifiable to, like, invest the time to, um,
to try to, like, improve that experience.
And I didn't feel that way about Android because IntelliJ was actually, like, quite good.
And we had figured out how to make that work at scale.
But, but Xcode was a little more difficult at that point.
So there was Xcode, which was.
bad and fifth existing needs.
And then there was another IDE that another team was building.
I think it was web-based, if I ever...
It was web-based.
I'm not like it that part.
That part's fine.
It was, but it was built off of an abandoned Google open source project written in Gwit,
which is Google Web Toolkit, where you would write Java,
and it would code gen to JavaScript for everything.
And I tried.
to build on what they had. I tried to build some credibility. It even actually sped up their build a bit.
But again, kind of like looking at the iteration times. And also I was like, this is an abandoned
open source project. It's written in Google Web Toolkit. And I was like, we are the React
company at that point. Right. Right. Like how, why would we not empower, you know, people who
want to build dev tools to use, you know, to build on a, you know, technologies that we are the leader in
and actually really like because we think we're actually really good.
So I was like, you guys are crazy.
And so I started and it's similar to the thing where I mentioned with Buck,
I was like, I started it as the Java build tool,
not the everything build tool, right?
I was like, it was a similar thing.
I was like, hey, I'm going to go over here and start this other editor.
But we're just, we're just going to focus on iOS.
Like I'm not trying to take things over.
Yeah, you didn't want all the friction.
And I mean, that team, they have.
had all the existing users, though, right? They had thousands of engineers. Maybe a thousand,
I think. Eventually, leadership sided with what eventually became new Clyde, which is what you built.
But why did they side with you when you didn't have any users on your shared profit?
Yeah, I mean, I think it was a combination two things. I mean, I think the arguments that I made in favor of,
like, the technology stack to build on, another was that
New Clyde was a desktop application,
and part of it was if this is actually going to be an iOS,
you know, Xcode replacement, like people are going to want to use,
it needs to be able to talk to the simulator or plug in the phone or whatever.
Like in theory, we could go through the web and do all those things,
but it just seems like a lot more, a lot of extra work.
And I think the other part is that, you know,
I had built up some credibility with the buck thing that,
they're willing to like take a gamble.
Like, well, the last thing, but it worked out.
Like, like, willing to try this one.
I saw that this, in combination of some of the other work, led to your promotion to E8,
which is also known as principal in the industry.
What was your reaction to that at the time?
Yeah, I mean, certainly I was very excited.
You know, I felt like, you know, in the ways that I had been out of step, I felt like, you know,
when I was at Google, I was like, this was also.
also not just, it was also validation that like, oh, now I'm also growing not just like technically,
but like understanding like what it means to kind of, you know, do the things that like are in line
with your employer. And so like that also just, you know, was equally valuable or satisfying.
I know that new client was open source and I think Buck was as well, if I recall correctly.
What's the rationale for open sourcing if, yeah, like what are the pros and cons of open sourcing
technology that you build. Yeah, I mean, let's see. Buck is the more interesting one. New Clyde
kind of didn't really get adopted externally. I think in both cases, I think, you know,
all these companies have benefited so much from open source. Like, I think it was just a feeling of like,
if it's not really the secret sauce, like, let's share this with other people, right? Like,
I mean, done like Codex as well and then a number of other things in my career. So I do feel like that,
that sharing of information is just good.
Even if no one uses your tool
just as seeing as a reference of how a thing could be done
could be great.
You know,
in the better scenarios,
you actually get,
meaningful contributions back and things like that.
And,
you know,
I remember, like,
I think Uber adopted buck.
I think Airbnb did.
Like it was funny,
right?
Like I said,
like Facebook was the biggest app
so we would hit all these problems first.
And then this next,
wave of companies would start anything these things.
They'd be like, let's check out what these guys did.
I think, I mean, honestly, we also, you know, at Google that was, it's internally, it's
blaze, externally, it's Basel.
So we were open source first and we always did, we always did kind of order like, if we put
some pressure on, like, you know, we get, and it ultimately did.
I think, I think you've gotten a little bit of credit from that unofficially from some of the
folks who worked on that.
But I think also that, you know, we, you know, it's also like a recruit.
recruiting thing too or just showing like hey if you want to like you know being at like the you know
the leading edge in like whatever area this technology is and you want to do this all the time not
just as like a drive by contributor like you know this is what we do this is who we are and then
this decision to open source is this was this a bottoms up decision where engineers said we're just
going to do this or is this kind of also leadership buy in hey this is going to be valuable and you
got like a dock yeah yeah i think i think i think both
cases it was certainly like bottoms up. I don't think so you know things like React and
pie torts like those are like the big success stories right where where the value back to
the company is like unquestionable and then there's this like longer tail of things where I think
depending on the economy and other things the managers get grumbly if they feel like their
engineers are like doing too much open source or things like that so it's almost almost always
feel like like bottoms up. Yeah, I would say that's, that's often the case. But it wasn't like met with
a lot of resistance, you know, and you usually get like a good conference talk or two and a nice
blog post. And those blog posts like, you know, do, I would say, pay dividends over time.
Right. Drawing recruiting. Yeah. Like they have like a longer shelf life than I think people realize.
Okay. So you got your, your EA promo at this point. And I imagine the expectations maybe in your mind are
kind of going up a little bit.
And now you need to go and find a E8 problem, I guess.
So what did you do after you got Froden?
I think that's where I did a little over my skis.
And I tried to help with WebSpeed, I think is what I did.
Because again, it was a thing that was like a really big problem for the, like the,
just the load time of Facebook.com was not in great shape.
You know, the architecture was a bit stale.
and it was just
the problem was so big
and I didn't have
I think the
you know a lot of people who had worked on web
at Facebook had worked on it for like a long time
and like I had you know put myself in mobile and dev tools
I actually wasn't in that world
and I remember I was like
ah you know what do we do I mean actually I sat down with the other person
we started like compiling V8 from source
and trying to see
If we could like change the way that we generated JavaScript,
if it was like, it would be friendlier to VA.
We're just trying like wacky things,
none of which panned out, by the way.
And I just, it was, it was, there's different,
there's different people who are like good for different projects.
And I think that was just not the one.
Like, I'm better in a project that involves writing a lot of code from the beginning.
And I think a project like that was like more about like looking at data
and talking to a lot of people.
And like, I just, that's not my strength.
I think you mentioned that at this point in your career, the idea of a hero quest.
Yeah.
What's that mean?
The idea that, you know, that it's like, you know, it's embarrassing because there's like something about ego.
This idea that, you know, there's this like, you know, Gordian and not and all these engineers are like,
if only someone would come in and like solve this, you know, engineering problem.
I was like, I know JavaScript right up the book.
Just come in that way.
and but I didn't I definitely did not you know and I think that it's been an important learning and I've had to relearn it like at least one other time that you know I can do a lot of things but there's a smaller subset of things that I genuinely enjoy doing and that I'm going to be a lot more successful right if I stay in the things that I genuinely enjoy doing you know I try to expand that over time but you know we don't all have to be.
the best at everything, right? I think that's
she accepted and
you know, embrace it. So then
how'd you find a EA problem after
that? Like, what was the next day?
That was, uh,
I think a bit of luck as well.
I guess, you know, we had, we had, sometimes
we'd have these little summits
with like smaller groups
of engineers about like brainstorming
what's going to, you know, bite us
in the future. You know, I mentioned us. Like, you know,
the repo keeps growing, you know,
you know, we're going to hit some scaling problem.
point. And a person or my manager, I think he became my manager, Brian O'Sullivan, he decided to get
some people together to like work on making a virtual file system to try to get ahead of that problem.
And so you got myself and Adam Simpkins and Wes Furlong, you know, both tremendous engineers.
I actually thought I was the worst engineer on the project for quite a bit.
You mentioned a virtual file system. What? On a highlight,
level, what's the benefit to a company like meta for if you had something like that?
If you have bought into the model repo philosophy, right, put all the code in one repo,
you know, most things that people do need only a subset of that repo, like they have to look
at the files at any point in time. And so the idea is that, you know, you design all your
tooling around those virtual file systems that when you say, clone.
in the repo or then you update to a different commit or anything like that.
You don't actually have to write out every single file in the repo on disk when you make
that change, which is the default in your traditional file system because that is going to grow
proportionate to the time with the size of the repo, right?
So at some point you're going to be very sad.
And so, you know, there's actually kind of two, at least two parts to it.
One is building this virtual file system that's like, hey, I know the user,
at this commit. If the operating system asks me for the contents of the file, like, I can go get it and
it will appear like they had actually laid out all the files. The other part of it, which is a part
that maybe I was actually better at, I think, is then try to anticipate, well, we have all these
tools in our tool chain that are used to just reading all the files. You know, you just rip-grap,
it just reads everything, right? That sort of thing. And how do we start changing also our development
and flow and our tools, such that they are designed with the virtual file system in mind,
because if you just materialize all the files with your tool, and I've actually just lost
all the benefits that you've made.
So on a high level, it's basically just lazy loading a huge file system.
So it's more efficient because you don't need to do everything at the beginning.
Yeah.
And so that part of this project that you said you're better at is integrating everything into
this on top of that primitive.
Yeah, yeah. So one thing I did, actually with Hansen Wong, who's here, actually I'm on the Codex team with me now, was I was like, oh, okay, you know, the traditional way you have, you know, you want really fast file search in your, in your IDE and your editor. And the way most of these things do is the first thing you do is they walk the entire file system to find out what the files are. And I'm like, well, that's going to be a, that's going to be a serious problem, right? It's going to undo all the benefits. Like, and, and then it's one of these things is like not, you know, first it was like, okay, how.
How can we implement file search that's not going to undo all the benefits?
And then it was, how can we do it even better than how it's done today?
And so we ended up building this file system called Miles for my files.
And it, you know, on like kind of like a cron job, it would index,
it would ingest all the new commits that had come in on, you know, on trunk.
And keep track of like which files had been added or removed.
just like the names of the files, not the contents, because all you need is the names.
And then Hansen had some clever ideas about how we maintained that index,
such that we could support fuzzy file matching.
So instead of just substring matching, right, like you type, you know, like a camel case thing,
you want to be able to type just maybe the uppercase letters or just your spelling is bad or what have you.
And so it came up with this really interesting way to represent, you know,
kind of all the files that had been seen at some point,
and then some bits to represent,
you know,
if I'm at this commit,
was this file present or not present at that point in time?
And then also,
when you asked,
I mean,
you sent a query to this thing.
So you'd send,
like,
what commit you're at?
And, like,
if you've added any files locally or removed any locally.
Um,
and I think we got this,
you know,
like,
you know,
over a million files and like,
I don't know,
like 10 to 20 milliseconds or something like that.
So that you would,
be in your editor, you know, typing. And so that was way faster than, I think, then, you know,
like what X code or is there like, like, UMDS code or something like would give you, um, out of the
box. And so, so that was, you know, so like it was solving a problem for Eden, that was the name of the
file system originally, the virtual file system, um, you know, in anticipation before it was even ready.
Um, but then it was so fast, um, and it was available as a trip service internally. Like,
people started using it for all sorts of other things.
things. I think when I left, like, I don't know, I think there were these like 30 servers running
miles, like spread around the globe. So clearly that was more than just people like personally
searching for files and then you did that much, you know, capacity. Interesting. Yeah, I mean,
when you talked about, I guess, the implementation details, most, most people, they, they don't
use leak code on the job, but that sounds like, I'm just thinking about the, I don't even know
how you have with that. Is it like a tree, like a try thing? No. So that's, so that's,
That's what's funny that try.
Yeah.
This was,
this was pretty cool in that,
um,
we,
we had kind of like two parallel arrays.
One was like the file contents and one was,
uh,
maybe an integer into that,
um,
index.
I forget.
And then we had a,
we had a 64 bit,
um,
uh,
mask that was like,
it was like,
so you had like 26,
lowercase,
26 capital,
10 digits and like,
maybe dash and whatever.
And it would, and it was a, every bit was set if the, um,
if that character was used at all in the file that you were searching.
And so the first thing was we could like blow through that list and, um,
you know, exclude a bunch of things like right off the top.
But we also like very designed like all these arrays were in parallel to each other so
that for like cash wise,
we knew it would be very efficient for, for the CPU to like read memory.
you know, linearly.
And then it led to itself to, you could just partition that array.
It, you know, led itself to parallelism.
So it's really, yeah, I should really probably write it up at some one.
Because that is really cool.
It was funny because it wasn't just kind of like out of the textbook.
That's for sure.
I understand that, yeah, you worked on Eden and then obviously this Miles as well.
And then these eventually led to another promotion.
But prior to that promotion, there was some learning.
you might have had about influence and conflict and orgs,
so maybe if you want to share that?
Yeah, I mean, that's one of the things, I guess,
about being an EA who primarily writes code, right?
There's other people, I'd say actually the majority of people
about level or higher are not writing code, right?
They're actually kind of exclusively spent doing more like influence
or working across teams, you know, writing the big Google Doc
and getting everyone on board and all that sort of stuff.
And so, you know, as an E8, trying to have that level, like, you know, that level of impact
that you're expected to have, it's like, oh, it's hard to do that just writing code.
So it's like, I need to spend at least some time probably, probably influencing other people.
That would probably be good for me.
That would probably make my manager happy.
and I think, you know, sometimes you're just so confident in some insights you have that, or certainly I was,
that I would just, I just came down like way too hard, I would say.
And that did not go well for me.
And yeah, and so like a promo got delayed because I was very excited.
or anxious about when Microsoft acquired GitHub.
And, oh, yeah, because New Clyde was built on this, you know,
primarily GitHub technology.
And I was like, that's going to go away because VS code is going to not make them be a project anymore,
which was true, which didn't happen.
And I was just so anxious that this was like now a risk, right?
And I was pushing people.
But, you know, people were, I didn't.
really account for the fact that people were happy with what they were doing at the moment and
like didn't want their cheese moved, you know, like right out from under them. And so,
you know, I got like a little bit of talking to. I, you know, I had to wait a little bit for
promo and things like that. But I, you know, I actually got some coaching, I think after that point.
I can't remember the exact chronology. To work on that, like specifically. What's the number one
thing you learned to that coach? I would say, for me, it's.
is like, I think I'm more aware of, like, things that trigger me.
Like, that would be the technical decisions or what have you.
And recognize when that's happening, I'd be like, okay, let's not act in that moment.
Or, you know, if I don't think I can have the conversation, like, or I'm not the best place to have it.
Like, maybe I go talk to the person's manager instead rather than like, you know, like bull and china shop to the engineer and be like, so I have this thing.
I'm thinking about it this way.
I'm kind of riled up about it.
Like, help me.
Like, how can I, you know, work with your team or whatever?
Like, that's happened to you.
Well, it's interesting to me in your career because the promo got postponed and you were,
you were seeing that VS code was going to come up and the thing that New Clyde was built on
was going to go down.
And you were actually right in hindsight, too.
Like, in the future, that is what happened.
And you were battling for what was.
right, yet you were told calm down a little bit.
So, I mean, what are your thoughts when you saw things play on?
You go, actually, I was raised the whole.
I did have some conversations like about a year later.
And I was like, hey, can we balance the ledger a little bit?
Like, it didn't be pretty hard for that thing.
And it was kind of good.
And we worked it out, I would say.
Okay.
So I guess the learning is just how you went about it, not what you were like.
Yeah.
Yeah, I would say that's true.
You seem to have a great time at Meta and eventually you left.
So what was the thing that drew you to Open AI?
Yeah, I mean, a number of things.
I mean, one was, so I interviewed at the end of 2023 with Open AI, I believe.
And when I had spent 2023 at Meta, trying to do LLM-based developer tools, right?
with, uh, there's a, you know, we had our own, like,
a version of, even MetaMate.
It was like a little version of, uh, GitHub co-pilot, right?
That one, we did like a paper and a talk on that one.
It was a code composed, right?
That was code composed.
Yeah.
And, um, you know, we, uh, like now, uh, there's a lot of, uh,
enthusiasm around, you know, delivering quickly and pushing the boundaries of that sort of stuff.
And, uh, you know, I mean, truth with it.
I mean, you know, you get feedback on some of these things.
like, why is this not, you know, GBT4?
And I was like, well, we're on like Lomba 2.
It's not the same thing.
And, you know, I was not a researcher, right?
I was like, but I, and I, you know, just wanted to build the experiences, right?
And that sort of thing.
And so that was, that was one thing was that, like, I wanted to build stuff and I
wanted to go to the place where I could actually built with the best model.
You know, two, again, was similar to like, you know, Google,
originally, like, seeing the people who were coming here to Open AI. And I was like,
well, those are people are really respect. I was like, those are, I could actually work with more
senior people, you know, like meta eights and nine's at OpenA. I felt like that I could
where I was. And, you know, third was just like this also seems like, and it has been just a very
special place at the point in time. So I told a lot of people, I was like, you know, I felt like this
would be like the most similar to starting at Google in 2000.
So not like 1998, but like let's say 2000, right?
Like they kind of got some footing and got some, you know, product market fit.
And so like just as a personal level, that was just a very exciting thing.
And, you know, the last one was that.
Funny thing is that when I really enjoyed calendar because it was consumer and I shipped
to a lot of people, I went to Facebook because I thought huge consumer,
place. Ended up not doing consumer at all, right? Ended up doing developer tools. But like,
you know, my users were my friends at work. It was just like 20,000 people, not like a billion
people. But it was good enough for me at the time. And then, you know, thinking about open AI and
then the chance to actually, you know, come back to consumer or at least like have a large user base,
right? And so now working on Codex, you know, I would have, like over a million weeklies or
I figure how much it is right now. And just keep it.
growing like like like you know hockey star more like vertical line than hockey stick even um that uh you know
so it's like way more than the 20 to 40,000 developers you know it could affect it at meta yeah absolutely
I'm deaf tooling for the industry almost yeah um meta when I think of the company it's kind of
of engineering driven like engineers are king and queen I guess and it kind of like drive everything is
very bottoms up from that sense
And I feel like a lot of the lab companies are also like that on the research side.
So rather than engineer is the first class citizen, it's like, let's make sure the research goes well.
And for good reason, right, like, that's also part of why you came is the models are good.
But as an engineer, you mentioned, you know, you weren't doing research.
What do your thoughts on that research-led culture versus engineering-led culture?
I mean, it's certainly an adjustment, right?
I mean, I think of anyone who comes here and says otherwise is a liar.
You know, if you've been at like the fangs or whatever.
And but, you know, they are, you know, you talk about impact, right?
Which I think is important.
Like a real, like you genuinely, you know, mean it.
Like, I, you know, I love the work that I do on Codex on the harness.
And I think we do a lot of very meaningful things.
But, you know, if the model weren't very good, it wouldn't really matter what we did, you know, on the harness.
Right.
And so I don't, so, you know, that's how it is, I would say.
But I, you know, I feel really great.
Like, you know, we sit right next to the research team.
We're really close with them.
And so that relationship that we have, that we get to co-develop the thing.
I mean, that was another reason, you know, leaving meta, I mean, was that for, in the LLM space was like, you know, I want to build the product.
with the people who are building the model so we can do this thing together.
I mean, you know, maybe you could do that there sort of, but not.
Anyway, the impact was not, you know, quite the same.
So you mentioned, I mean, when you got here, it sounds like you were working on Codex
and starting that project.
So I understand with the initial launch of Codex, CLI, it was not exactly what you hoped for,
like in terms of the, how it was received, but later it kind of really all came together.
Can you tell that story?
Yeah, sure.
I mean, it's been a wild ride.
So Codex, CLI, we launched it in April 2025.
It was kind of like this, like, one more thing moment at the end of the 040 mini live stream.
And so we demoed it live.
We open sourced it, you know, for repo at that point.
A lot of people tried it out.
You know, everyone was excited to try a new coding agent.
You know, and it was pretty good, but it was pretty rushed to get it.
out the door, right? There was, um, which in some ways was good for engagement because now
open source. We were getting poll requests, you know, coming in all over the place. And, uh, I forget
it. You know, we're, I feel like we're like 10 to 20,000 stars in like a week or two or something
like that. So like that, that part was fun. And, um, you know, there were people who did like,
and, you know, it's like nobody affected or anything like that. But then, um, but that,
I think we didn't, we were maybe quite staffed to like really drive that the way that we needed to,
because, you know, we're trying to cover multiple things as a company.
And so, you know, just a month after that and the, you know, team of like seven engineers
and I forget how many researchers launched Codex Web or Cloud Web, where, you know, you can use
codex in a container or you could just, you know, kick off a new thing from your phone, which is super
cool.
And that was like a more well-staffed effort.
And I think, you know, that one is still the, I think the, I think the,
the long-term vision is correct.
But I think with a lot of this stuff that we've seen,
you know,
you have to bring the users with you.
And I think that was just like a little bit ahead of its time.
And people were still actually more into the local coding agents.
And so we kind of, you know, we saw, you know, the, like the web product.
Like there's a big adoption initially.
It wasn't, you know, as sticky, I would say, as we had hoped.
And then through the summer, you know, both products still kept
kept, you know, we're working on both.
But then somewhere that midsummer, again, like I said,
is like this moment, like local agents are still the,
I guess the stronger product market fit.
But again, I still think in the limit personally, right?
Like, you're going to need more machines than just your laptop to as a place for agents to run.
And so we shifted quite a bit over the summer.
So we brought more people onto the Codex.
Yeah, a few of the kid bangs.
Bring more people on.
You know, GPT-5 was going to come out, and that was looking really, really good.
And I think I was personally excited about because I prototyped it a couple of times before,
was that in addition to the CLI, then we also had enough staffing.
We also started working on the VS Code extension.
And, you know, and I felt actually very strongly that, you know, the terminal is good for a lot of things,
but it has a lot of limitations always.
It's just like, you make a lot of compromises to make, you know, a nice UI in the terminal,
whereas VS code, we didn't have to make quite as many compromises.
And so I think, like August, I think it was a crazy month.
I think GPT5 came out.
I think we released our new refresh terminal UI.
Also the GPT open source model came out.
We supported that in the, in the TUI as well.
So that was really cool that we had, you know, an open weights model and the open harness.
And then later that month, the VS code extension.
you know, came out. And so we were just like, you know, shipping like, like crazy. And,
and that's really where, like, that confluence of things, I would say, um, is where we started to
see the inflection point that's brought us to like, you know, the vertical growth that were,
that we're at today. Um, and so that's, you know, that's been really exciting. I mean,
you know, some of these things, you can, you can go in the repo check for yourself, right? And,
and, you know, got check on some of these things. Um, terms of number of people. Um,
a number of commits and all this sort of thing.
And yeah, so it's been quite a ride.
You mentioned the local versus the remote version of these coding agents,
and it sounds like you have a lot of conviction
that the right long-term direction is not the local versions.
It's remote in the cloud.
Why is that?
Well, I think about, I think that the people who actually, for whom it is sticky now,
and I think there will be more of it,
is that you imagine, if you imagine you want to just automatically,
for the agent at every like GitHub issue or linear
thing that comes in or anything like that.
I mean, obviously, you know, there's,
there's costs with those things, but or you can,
it could be abused.
But like, you know, let's say for an internal private repo, right?
That you, yeah, want to be able to just have it be a piece
in any sort of automation pipeline, right?
And so you can't have all that happening on your, on your laptop.
And so I guess, I mean, you know, as an individual I see,
and maybe we'll still personally spend more time with the local agent.
But in terms of compute time of agents doing work, right?
I think getting that set up in the cloud.
I think, you know, a little bit, it's like a little more to get set up right now.
But once you have it, it's quite nice.
I see.
Okay.
So you're not seeing that product, the local one will change.
You're saying that like across the industry, the compute that goes towards agents,
majority will be in the cloud.
Yeah, I mean, even in, I think when the VES code extension first came out, you know, one of the things was the ability to take the conversation that you were having and, like, hit a button, have it like transfer to the cloud if you were set up to do it.
Right.
And so I think that we'll continue to see that where you may, yeah, you're working on something.
You want to throw it over here.
You're like, bring it back, you know, when it's done.
Your codex usage is five-x since the beginning of the year and over a million people are using it now.
I'm curious, has your AI workflow changed a lot since you've started to use this newer version of Codex?
Yeah, it has. It has. I'm a much bigger user of the app now, maybe than I thought I would be.
Yeah, like, for a while I was very strongly in like our VS code extension. Like I need, I want that sidebar. I want all the code there next to me.
I feel like these things should be, you know, together.
I'm not a person who doesn't look at the code.
I'm not a, like, I don't, for projects that are like true prototypes that are throwaway,
like I will actually not look at the code.
It's very freeing.
I understand.
I'm so excited about it.
But for the code that goes into codex itself, I'm like, no, I still need to look at this,
right?
This is pretty important.
This is like what's going to, you know, affect everybody else.
But, you know, you start to get a sense of, okay, like, I'm confident the model's going
to be able to do this, you know, this change and this.
sort of thing.
Like I, you know, I don't need to babysit it.
I'm just like right a lot up front.
And so, you know, I have like four or five clones of the Codex repo of my machine.
And I, I have enjoyed in the Codex app, like the multitasking.
I just is, it's just a lot easier because now you're just kind of hang out in one window.
And, you know, I try.
And it's, it's the game exactly.
But it's like, you know, it's like, how much throughput can I get, right?
In terms of how many balls can I juggle in the air sometimes.
it feels like.
It can feel like a little hectic when you're really trying to context.
Yeah, yeah.
The context switch.
But at the same time, you're like, I'm getting a lot more done, you know, and that,
you know, it almost feel a little bad writing code by hand sometimes because you're like,
I could have, if I had asked this in the right way, you know, it's like, when you started,
it was like, oh, I'm just going to change these three lines.
And then like, you know, 30 minutes later, you're like, oh, I kind of, you know,
But we all like to type still, I think, and that sort of thing.
What percent of your code would you say is human written versus the model generates it
these days?
Oh, man, it's models that would be like 80 to 90 percent.
I mean, yeah, yeah.
I mean, like, especially like debugging a test or like the CI thing is bad.
Like, I'm like, hey, you know, you know how to write, you know how to print debug, whatever.
Like, and that's great.
That's really freeing.
Digging into the problems that are.
suitable for LLMs and the ones that are not.
Like what do you need to see where you think,
okay, I need to go in and, you know, write it myself?
What's that 10% and also what's in that 80, 90%.
Yeah, I mean, that's a good one.
I think about that.
You know, because every time I sit down, I'm like,
should I write this?
I mean, the answer's almost always no.
There's things that are like lower level and someone, you know,
so Codex, the harness, the part that I, you know,
spend my time on is in Rust. And that means that, you know, we can do and we do do like operating
system specific things in that code base. And actually, and I spend a lot of time on on the
sandboxing, right? So the thing that really upholds like, you know, the security integrity of what,
of what we're doing and the model can't go outside, you know, the bounds that you set.
you know, I do more of that by hand because I need to make sure really sure that that's all
correct, right, that our test coverage is good. Or sometimes I'll, you know, seed it. And then,
you know, once I've got like the groundwork and like, okay, I have the pieces that I, that I
had a lot of feelings about and then could let it like fill in the rest and that sort of thing.
But, you know, a lot of like refractors and things like that. Actually, I think that like a lot right now is
like, you know, I haven't built up a big, you know, PR that does too many things,
but I know it does too many things.
So they're like, okay, please, you know, please split this up into reviewable sized commits,
things like that.
Like, I think about how much time I'd probably spend on that sort of stuff before.
And now I can.
Right, I delegate, freeze you out.
What about, like, code review?
Is there, like, what percent of lines of code do you and at OpenEI generally?
Are people reviewing manually or other agents that are like, you know, imagine you write at rest or something.
I don't really need to review that.
I mean, do you see?
I would say I liked the approach where, you know, the agent should do, you know, multiple rounds of review or whatever until it's confident and that it's like, I'm worth a human's time of looking at it.
But we still do look at it, like ultimately before it goes in, I would say generally speaking.
I think there's, you know, we have like our.
agents empty file like everybody else does.
There's still sometimes you find like a gap in knowledge that needs like that some
context that needs to get added back into there or it's yeah, just just things that we
haven't yet memorialized that the agentness.
But like as a human I still happen to know and things like that.
So we do catch things.
But you know, I'll say actually now that people are also using AI to write their like
pull request summaries or summaries across the team, I would say, are getting
way better. So now, at least also when I'm going into review, right, it's been reviewed by Codex.
There's a summary that's like that we actually make sure it has like the why and the what of why,
you know, the reason behind the PR. So that is certainly I think helping like get through these
PRs faster, which is good because there was a lot more review to do. That's, I mean,
that's amazing. I feel like, and 50% of diffs have like, you know, almost blank. The test plan. So, yeah.
know, I don't know, arc build or whatever you're the, oh, I know, man.
I want to talk about the codec CLI that's open source.
Why is it open source?
You know, for something that's that critical to like what's going on on your machine, right?
Like that's one of the aspects of open source.
I'm not, you know, the most zealous about this sort of thing.
But I sympathize with it, this idea that like, hey, you're going to put this thing on my machine.
I care about what it's doing.
And I think in this domain in particular, it's really important that people can look at it, you know, and have an idea of what it's doing.
Because, you know, people have a lot of questions about AI agents and that sort of thing.
And so I think, I think this area, this domain, I think it's actually really important.
And also we, you know, we have gotten a lot of, I think, like, great competitions and
reports and things like that that we would have missed out on.
And then also, I think it's, you know, just, you know, sharing with the world with, like, how this is done.
So we do it through code, right?
I've put out one blog post about how the agent loop works.
Like, there is a plan to do more of that.
I'm actually excited to.
It's just, you know, it's just time.
Is this the limiting factor?
It's the restaurant.
Really not.
Yeah.
It was funny because I had two candidates who came through.
And one's like, he's like, hey, I wrote it, right?
I was like, no, no.
I wrote it.
And another person came in.
He's like, oh, you can tell that, you know, that you did not outsource the.
Okay, good.
Right.
I was like, oh, thank you.
So, yeah.
You mentioned the blog post.
How does Codex find what is available to it in its environment?
When I'm running these things, I'll see it's kind of amazing.
It's thinking to itself and like discovering.
all these things in my terminal.
So, yeah, how does that typically work?
Yeah, I mean, there's a few.
I mean, there's obviously what is, you know,
when Codex is based training,
like it loves to use Ripgrep, it uses Ripgrep very well
to find all sorts of things.
And then there's, you know,
if you have your agents MD file and you're saying,
hey, in this repo, like these tools are really important, right?
You should use these, right?
You're the readmys or whatever.
Or obviously, if you use MCP
and you associate these infuriate these.
CP servers with your, you know, where you're working on, right?
That, that injects the set of tool definitions, like, at the start of the conversation,
right?
So then I kind of, yeah, so that's not even like discovery on Codex's part, right?
It's kind of just like put front and center there.
I see.
So some of it's the harness explicitly throwing that into the context.
And then there is a big chunk, though, where the model is just doing all the heavy lifting
of finding things.
Is that right?
Yeah.
Kind of like reflecting over your career, the breadth and depth of your work, if we look across all of it.
I mean, it's insane.
You're JavaScript's front end.
Then you have all these dev tool, you know, build, fuzzy file search, you know, virtual.
Now you're working on codex.
I'm sure you had to continue your engineering education to kind of get through all these projects.
What are the top technical books that have helped you educate yourself?
Yeah, I mean, you know, so one was this book on operating systems.
It's like a thousand pages or something like this.
It's the Addison Wesley book, I'm trying to remember the author.
But it was funny because I was when I was working on the virtual file system.
So I had actually gotten to that point in my career without ever writing C.
Like undergrad, like it was more theoretical.
Like we just never had to touch it.
And then yet I'm like working on a virtual file system project.
And that was like very low operating system.
That's why I guess.
You know, joke that was kind of the worst engineer of the project.
And, you know, and someone said something.
And I realized, I was like, I don't know what they're talking about.
This is kind of embarrassing.
And I was just like, you know, what book do I have to read?
And I think my manager, Aaron Kushner at the time was like, he was like, well, there's this thousand page book.
And I was like, done.
So if I bought it, read the thing cover to cover.
It like took it to Hawaii with me.
I took it everywhere until I finished it.
It's kind of amazing and sad or weird, like how.
how far many of us can get in software engineering without really having a clue how computers work.
There's just so many of those levels of abstraction.
But, you know, in one hand, it's very freeing.
And then on the other hand, it's little bananas.
And I think that what I would say to people right now is, is, you know, actively trying to go, like, deeper through the layers and understand these things.
and like because many times I saw other people do it and now I can do it is that there are problems
some other people could solve that I couldn't solve because I just didn't know that like
there was this cruff between these two layers and if you got rid of it right here you get like a 10x
improvement or something like that right if you if you're just operating so high up you don't
really know you know what you can what you can break down you know so that and so in terms of
books like I've enjoyed the like the O'Reilly the Rust book
books, you know, be kind of rust.
But they're also like well written and thorough.
And then honestly, another thing I've been telling people that's not in the book
category, but probably more fun.
And I learned a lot, actually, are these CTFs sort of like capture the flag, like security
type competitions.
And just, you know, it helps with like kind of like adversarial mindset.
It helps with, I think, they're usually just.
like it's like a it's like a decathal like a computer dicathlon i feel like right because like
there'll be multiple challenges and maybe this one you need to like understand assembly and this
one you need to understand like what someone's like janky php admin pages do bang and all
the sort of thing and it and it just forces this um breadth on you in a way that's kind of hard to
to generate otherwise and it's also into just more fun because it's like a game and that sort of
thing. Can you give some context on what
CTF is? Yeah, so it's
it can be a number of things, but
it's usually a competition, usually
in like the Infosec, the security
domain, and there'll be a
set of, there's like the
Jeopardy style one, which is like there's a bunch of
challenges and
like, you know, designed, like
crafted ahead of time and they all have
point values associated with them.
And so, you know, there's usually some fixed around
a time and it could be individual, it could be with a team,
and you're trying to solve these
things. And there's a flag. There's always like a secret, you know, piece of text in there that
usually has some format. And the way is that you, basically, if you can discover the secret piece of
text, that means that you have the challenges set up that you would only have gotten that if you
had figured out, you know, reverse engineered or whatever you were supposed to do. And then you
get submit that piece of text. And that's your, you know, token to demonstrate that you have
solved the challenge. And so it could be like a race to like solve everything first or get the
most points in a certain amount of time or different things like that. So it's like it's kind of like an
escape room, but in your terminal, you have only computers to figure everything out. Yes. I see.
And your recommendation is people who want to become better engineers, they should invest in doing
some of the CTF because they make you solve problems that make you a better engineer.
Yeah. And you know, and then you also develop skills and things that you just wouldn't have. You know,
if you're just like a, you know, writing React every day, like you're probably not going to
to like open up GDB and like reverse a you know a tick-tac toe game but like I did that because
of a you know a CTF challenge was to do you know exactly that right and and but then it's like
but then I learned how to use GDB and like and then then you start you know when you're faced with
other problems you're like oh my my just toolkit of ways to solve things is just much broader now
a lot of people especially earlier in their career they see the power of codex doing everything for
them in the terminal. And I just imagine them saying, oh, well, I don't need to learn GDB because
Codex knows it. So what would your advice be given the landscape of these AI tools for people
thinking about their engineering education? Yeah, no, that's, I mean, I think everyone's, like,
struggling to answer that question right now. I do think about it, you know, a lot. I don't have a
great answer. I kind of personally come back to my thing about, like, um,
I still think, like, trying to forcefully kind of go through the levels of abstraction and just understand, you know, more, like at a deeper level, how things work is going to be important.
I mean, I think this will change.
I'm sure this will change over time.
But right now, it's still kind of like, you know, the questions that you asked the agent is going to affect the quality of the thing that you get out.
Right.
So if you're not asking the right questions, you're not maybe going to get the best engineering solution out the other side.
you know, as things go on, perhaps that will also be, you know, another liar that's removed.
I mean, I think, you know, that's certainly where we're going.
I don't know what time frame will get there.
Things do seem to be happening faster than we expect.
But, yeah, I think in general, but learning how to ask what the right question is.
And I haven't, you know, for myself, even, like, totally pinned down what that means for someone who's, like, starting out new, right?
I, you know, I fortunately have experience to fall back on.
or like that's where that that that taste or that intuition of what to ask has as developed.
But, you know, if you're if you're starting out, I'm still not sure yet.
And it's also hard to say because we don't really know 100% where everything's going.
When you reflect over your career, the expectations at these really high levels is kind of crazy.
I mean, it's like already for most people, this E7 or senior staff level, is this unattainable.
level of impact. So someone who gets promoted to that level is thinking, I got to really work
super hard now because the bar is like up here. And then you went two levels past that.
And so I'm curious, in the day to day now, what are your thoughts on these like crazy
expectations? Is it stressful for you? I mean, it never, it was never not stressful, I think,
for me because I really, I think also because I sat in calibrations, right?
And I, you know, so for a bunch of other people, you know, you talk about their level,
you talk about their impact and, you know, you want to be really fair and have this integrity
around.
Well, this level means this thing.
This is the impact.
Like, that's what it meets all is.
That's when it exceeds is.
And then knowing that, like, then you're, you know, playing it out in your mind, like,
well, someone's sitting in my calibration and talking about this thing that they want to be fair,
right?
And it is a little scary.
like, you know, you get to E8 and you're like, oh, you know, it's a D1 or a director.
And you're like, how do I have as much impact as a person who has maybe like over 100 people in their org?
And so that's hard.
A lot of people do it.
Again, a lot of people who are in I see who do it, do it more by a different form of people management, right?
Where they're trying to, you know, write the right dock and get the people aligned and do that sort of thing.
And the reason that they do it and are not a director is that all.
Oftentimes, I've talked to people that were in this, like, they're like, well, I have this
technical credibility or because I built this thing, like when I go to the team and talk to them,
it lands differently than if like an engineering director dosers.
Some version of that, right, happens a lot.
And so then, you know, when you say you're influencing, you know, 50 to 100 people as a senior I see,
then you're like, oh, okay, that's like, you know, D1 impact.
You know, whereas if you're a coder, I see.
you know, being really thoughtful about the projects that you pick.
And it can't just be like, this is fun for me type of project.
I mean, you know, if you actually care about, you know, not getting fire or you're getting like Sam Beetsall or better.
And even when I would start a project, certainly like the later I went on, I would think a lot about like, okay, like I, maybe there's this feature.
I just want to write it because it's fun.
And I'd be like, oh, no, I should let.
somebody else do that and think about, okay, it's kind of like with codex now, like, what code
should I personally write to maximize my impact versus if someone else could do it about as well
as me, let's say 80% is good. I should probably let, you know, have someone else do that.
And but, but even then, you know, if you're leading a five-person project, is that still,
you know, still getting to E8, E9 impact, still hard.
So really finding that project that's a force multiplier.
So, you know, like the virtual file system was a really great project because that was going to, you know, we knew that down the road, like this was really going to unlock so many things prevent us from being completely blocked, right?
Actually, another big part of it is, and, you know, one of my managers talked to me about it is, you know, not actually recognizing senior managers enough who.
are the person who
pairs the senior I see
with the right project.
And, uh, because some
senior engineers are amazing.
Uh, you know,
fixers slash coders.
But they're not the idea
cummer uppers with. And, um,
they're the, but they're the person you want for like,
you know, very difficult, um, technical
projects. I don't want to like out anybody here.
But I have people in mind. And, um, but,
but, you know,
a lot of times it's the manager who realizes like, oh,
this project needs this person, right?
And, like, that person would have never realized it themselves.
One of your old colleagues, you know, out of Earth, I think high-ass-temya, what are the other engineers that you admire and why?
And, you know, he mentioned your name, of course.
And specifically, he talked about your ability to start projects was really good.
And obviously, I mean, look at all these projects, right?
I mean, a lot of them, you created them out of nowhere.
Like, it was just, you had an idea, and you went off and built a prototype.
You came back and you were very convinced.
seeing that as a better solution. Do you have any advice for engineers who they have a problem
on a solution and they want to like build a project from scratch? I mean, I think, I don't know,
a lot of good projects I think come from being a little bit dissatisfied about something, right?
And I think, you know, it's funny. Like sometimes I, for better for worse, I was just so
charging ahead and just like building the thing that like not really thinking about, like what the
best way to do it was. So actually a really funny example is Google Calendar goes are going back
really far. I was like, I want weather. I want to have weather. Little icons showing the weather
in Google Calendar. And I was like, I'm going to do it. And I had mostly been JavaScript,
especially and not done any of the backend stuff. And I just like charged through and I cobbled
things together and, you know, made it happen. And then my tech lead was like, wait, how are you,
how are you storing that information about
the weather and stuff?
I was like, oh, I just like, you know,
through a blob of like XML and everything like,
he's like, he's like, we should have talked about protocol buffers.
I was like, you know, like binary formats to save bytes that I like,
I was just so, you know, set on weather of all things that like I never bothered to ask
anybody if there was like a better way to do what I was doing.
I was like, it's done.
So I mean, I guess that's a unique skill of yours, just the,
digging into the dissatisfaction and solving your own problems.
Seems like almost every single one of these projects is,
I want my,
I want something to happen.
This shouldn't be this way.
And then you went and you solved it.
Yeah.
Yeah.
But I think another unique thing that I see is a lot of people,
they get to work and they,
I don't know,
they're in their dev environment.
I'm sure there were many other people who saw with the Buck story,
for instance.
They go, oh, wow, this build is really slow.
Oh, well.
Like I guess I'm going to go to the micro kitchen and get an area and call its builds.
What gave you the confidence to know that you could make it so much better?
I mean, that one was, you know, I have to credit, like having been a Google, I never worked on the Blaze team.
I never knew how that like worked, ever touched their code or anything.
But I was like, I know that there's a thing out there that has this shape and is a lot better than this thing.
So that's an existence proof.
Whether I could do it or not like TBD, right?
but I think that was helpful in that one.
You know, yeah, I mean, there's a lot of, I guess, prior art that gives me, you know, confidence
and things.
And I usually generally identify myself as a coding machine.
I guess with Codex and everyone's a coding machine.
But that, you know, I felt that I always had confidence, like, you know, build a prototype
quickly, right?
And at least answer that question or like the basic question.
Or like the basic question, like, spiritually, should there be a way forward to this thing
that I think should exist?
And generally, like, if you are determined, you'd find a way.
Yeah, I've noticed that pattern.
A lot of people go to big companies.
They see this world-class infrastructure, and then they go to the other company.
Oh, you don't have this.
You don't have this.
And they build their, you know, those new versions of it.
I've read a lot of your writing at this point.
And it is so clear.
It's some of the best examples of good technical writing.
What advice would you have for engineers who want to write better?
I mean, I think a lot of it is, well, I think reading other good writing is a certainly good start, right?
It started maybe, you know, consciously or subconsciously started to pick up patterns of like what is out there.
I think, you know, really a high level thinking about like what is what is it that I'm trying to convey?
What would someone actually really want to know?
And outlining a lot up front, right?
A lot of people will give this feedback, but it's impressive how really important that is.
and just being like, does this set of things like linearly follow, I think is a big thing.
And then asking yourself, oh, I went from this point to this point.
Was that too big of a jump?
Is there something that, like, somebody actually, like, would have reasonably missed?
And I think, personally, I feel like that's a reasonable feel for, like, where that gap would arise.
And then if you can kind of anticipate that and magically put in the example that, like, someone would have, that someone needed.
to make that jump, I think like that, at least for technical writing, is a big deal.
You have that career note that I love. And at the beginning of it, you lay out this three-step plan for impact.
Can you explain that three stuff on early? I feel like that's a good algorithm for people to use in their careers.
Step one is figure out what you really like to do, right? And as I mentioned, like, it's good to broaden that,
but it's also good to be honest with yourself.
So you don't know, and like the quote,
the hero quest that I went on that that didn't pan out because I was working on
stuff that didn't really, you know, truly love.
And then two, step two was figuring out what your employer is what's really
valuable to them.
Right.
And as I talked about it, Google, I didn't do a good job of that.
I did stuff that I was really excited about, but it wasn't, you know, it wasn't,
you know, ad words for Google or anything like that.
Yeah.
And then step three is like find that intersection and then just really lean into that.
And, you know, the more that you can do that, I think the more successful, you're going to be.
And the challenge is sometimes it's not always there, right?
And maybe you have to go, you know, find somewhere else to make that happen.
Okay.
And then, yeah, last question for you is if you go back to yourself at the beginning of your career,
knowing everything you know now, what advice should you give yourself?
I think I should have been open to learning more.
things sooner. And I,
you know,
to be a little gentle to myself and I think other people are in that situation is that,
you know,
there's so much to learn when you're starting.
And then whatever your first programming language is,
I think it's funny. I think everyone has a soft spot for it.
They'll like make excuses for it.
Like ever, like,
oh, it's a totally good language.
And I think it's because it's like the first thing that enables you to do a thing,
you know, do anything, right?
And,
and then it's like,
oh, okay, I can finally do something.
It's such a relief.
and and and but it's also like a hazard because like then you kind of want to hold on to that thing because like you're finally productive and now you're like it took so long to get to this foothold I don't know how long it's going to take to get to the next foot hold so I think like in my particular case I probably did maybe went too deep with JavaScript and like I said it took a long time before I wrote NEC and I think uh you know if I had been a little bit more curious and a little more flexible in terms of what like types of projects I was willing to do
to take on or things I was willing to learn.
You know, it came eventually, but I think that is probably the biggest thing that maybe
could have made a shift for me earlier.
Yeah.
I gather from your story, there was a point with the X code where you said you hated
Objective C.
And then you were coming up with like ways to compile the Objective C and the Java or
Java into Objective C or something like that, maybe.
And then, you know, also talking about the C++ for Miles, it was like, it was like, it
It seemed like a very concerted, okay, I'm going to learn this as opposed to like just kind of
being open to it.
So yeah, yeah, it makes sense.
Well, maybe with Codex in the future, it'll be less of a hurdle for people.
You could just kind of say, hey, I know JavaScript, write this in Rust or whatever.
No, it's true.
It opens a lot of doors.
Sure.
Awesome.
Well, thank you so much for your time.
I appreciate it.
All right.
Thank you, Roy.
Thank you for listening to the podcast.
It's a passion project of mine that I really enjoyed building.
Another passion project that I've been working on.
kind of in secret is building an ergonomic keyboard that I wish existed and I finally have a prototype
so I'd love to show you what we've built. It's ultra low profile and ergonomic and I couldn't
find anything like it on the market. So that's why we built it. I'll put a link to the keyboard in the
description. You can take a look and learn more about the project there. We could definitely use your
support. Also, if you have any feedback for me about the show, I'd love to hear it. Comments on
YouTube have led to guests coming on like Ilya Gregoric and David Fowler. I wasn't aware of them
until someone dropped a comment. Also, feedback in the comments helped me learn to reduce the number
of cliffhangers in the intros. So your comments definitely make a difference. Please keep letting me
know what you'd like to see more of in the show, and I'll see you in the next episode.
