Algorithms + Data Structures = Programs - Episode 291: autoresearch with Opus 4.8 & GPT 5.5

Episode Date: June 19, 2026

In this episode, Conor and Bryce chat autoresearch, the top LLM models, how to get the most out of them and more!Link to Episode 291 on WebsiteDiscuss this episode, leave a comment, or ask a question (on GitHub)SocialsADSP: The Podcast: TwitterConor Hoekstra: LinkTree / BioBryce Adelstein Lelbach: TwitterShow NotesDate Recorded: 2026-06-10Date Released: 2026-06-19autoresearchOxide and FriendsGPU ModeIntro Song InfoMiss You by Sarah Jansen https://soundcloud.com/sarahjansenmusicCreative Commons — Attribution 3.0 Unported — CC BY 3.0Free Download / Stream: http://bit.ly/l-miss-youMusic promoted by Audio Library https://youtu.be/iYYxnasvfx8

Each harness is a bit different. And like, I even asked it. I was like, Yeah, I, I've run into this a lot where, like, harness differences, especially when you're doing like multi-agent orchestration stuff, you become very tied to the particular way that a harness works and that the harnesses tool, tooling works. And, like, I ran it, I had this weird thing where, like, initially I couldn't get,

I couldn't get models other than Opus to, like, drive ClaudeCode well. And I thought that the models just were not capable of handling this workflow. But it turned out that really there was just, like, a bug in the structure of my skills, that when it was going to launch, you know, a sub-agent, the way it was describing what to do was just not correct in that there was a better way to do this. And with Opus, for, you know, maybe because they post-trained it to, for the harness specifically, or for whatever reason, it just was sort of smoothing over this, you know, this bug in my skills.

Welcome to ADSP the podcast, episode 291 recorded on June 10th, 2026. My name is Connor, and today with my co-host, Bryce, we chat about auto research, the top LLM's Opus 4.8, GPT5.5. Deepseek V4, Codex, Coddcode, and more. I'm good. We are live. I think I need a caffeinated beverage here. We're going to get the bottle. Oh yeah, it used to be a...

Well, one sec. I mean, it used to be bubbly. I don't buy bubblies anymore now because they're too expensive. And, well, I shouldn't say they're too expensive. They're the cheapest of the sparkling waters. Currently right now, I have a president's choice grapefruit sparkling water

that I like quite a bit. And I believe it's, I don't know, $5 or $575 Canadian for a box of 12. Whereas bubbly a lot of these times, depending on where you buy it, ma'am, it's like $10 for 12 cans. Anyways, the listeners do not care about the price of the different sparkling waters. How's it going, man? I imagine, so we were supposed to have Dwayne Merrill requested guest by our last. guest, Marco, and he has agreed to come on, but he's rather busy. And so he asked if he could come on. We can, we don't need to, we don't need to have Dwayne on. We can just have an agent,

the agent version of Dwayne on and just, just ask it. That statement is going to upset people because they're probably are not sure if you're joking or not. And he's joking, folks. He's joking. We're not going to have the, I'm joking. Yes, you are. We're not going to replace our guests with replicated LLM versions of them yet. That's also a joke, folks. Calm down. Everybody's getting upset about AI these days. And so Dwayne's not going to be on.

That means we're not going to be talking about the coupled lookback, Cub and other topics yet, which means we get to talk about whatever we want, which probably, if it's just Bryce and me, means we're talking about AI and LLM's folks. I heard on another podcast, Oxide and Friends, hosted by Brian Cantrell and Adam Leventhal, I believe. And they were saying that they were getting some feedback that like most of their episodes were becoming about AI and LLMs.

And they were those, that feedback was not happy about that. Listen, folks, it's hard not to be drinking the Kool-Aid and so excited about this stuff. And I know I asked Bryce how he's doing. We'll let him answer that question in a second. I'm so excited. I told my wife about this. while right after it was the wrong time to tell her about it I was picking her up from the subway she had a long day at a conference

and she's like six and a half months pregnant and she did not respond with the enthusiasm that I was hoping for but let me tell you buddy this is let's see if you can tell what do you see on my this is my pod god podcast app what do you see on the podcast screen here I'm not sure if you can make it out in the blurriness well I see acquired I see ADSP and I see a bunch of people with purple backgrounds.

Yes. Welcome to Personality Meta Podcasts. You can't make out who these people are. Or actually, yeah, you probably can now. It's a zooming in. We got Andre Carpathie, Boris Churney, creator of Claudeco, Demis, Hasabas, founder of DeepMine, aka Google DeepMind, aka GDM. We got Elon Musk.

We got Jensen Huang. We got Ilyas Oskivir, Miramaradi. And there's other folks, too. These are just, it's a starting list of the people that I care about. And look at that. Look at that. You hit on Jensen Huang. Let's see if I can do it.

Boom. And you've got a list. And today was the first time it actually updated. And so now you can subscribe to someone you want to hear from, not a specific podcast. And it brings me so much joy. So much joy, folks. Because there's a lot of podcasts, for instance, no priors, Sequoia Capital, a bunch of these like venture capital, blah blah blah talking about taxes nobody cares but every once in a while people that you are interested in

go on these podcasts and sure enough today it was training data which is i think the sequoia capital podcast jensen went on it it aired on youtube three hours ago it showed up in my podcast cue for this personality and i don't know why no one has done this before why is it me why is it some guy who's in research doing this it's fantastic and you know what the first problem what do you think the first problem that I encountered now that I have these cues are that I need to make an adjustment to. I did not expect this at all, but it was rather obvious once I saw it. And I was like, oh, yeah, that makes sense.

I can't imagine. More than 75% of the podcasts in each of these meta-podcast personalities are not in English. They are in Chinese, Japanese, German, like a lot of the Peter Steinberger. Well, then you just use one of NVIDIA's excellent, you know, speech. and translation models to automatically translate the podcast. False. I'm not spending money to translate podcasts in different languages.

One could do it, but you're just going to have a filter where you choose the language. They're not very large models. You could run them on a fairly small GPU system. I mean, there is the thing where I've heard that, like, on Twitter now, they auto-translate. And so, like, you might be talking with someone in Japan that's using Japanese and that's created some interesting interactions. But when I was at the Singularity Intelligence Technology Summit in Shanghai a couple weeks ago, all the talks were in Chinese.

And so I was like, okay, well, you know, I could just use Google Translate on my phone. Or I could go spin up a GPU node and like vibe code, a deployment of, you know, one of Nvidia's new, you know, translation models, which I did. And then I learned a lot about that whole space because it's like, you know, you. You have voice to text. You have text to speech. You have, you know, speech to speech. There's, like, all sorts of different, like, you know, types of models and specialized use cases here for, like, if you want a transcription versus if you want, like, simultaneous translation.

And then there's this question of, like, chunking of, like, you know, the longer that you wait, the larger chunks you have, the better translation you'll typically do because you have more context. but the longer chunks you have, the more like visible latency there is. Because like if you wait to buffer up a bunch of words that somebody's saying, like a whole sentence, you're going to be able to translate it better. But if you buffer up the whole sentence and then translate the whole sentence at a time, then there's like a more of a lag from when the person spoke to when you start hearing the translation. And one of the interesting problems I ran into, I wanted a smaller model that I could deploy

on one L40s. Tell people what that is, because honestly, I don't need. I've seen the brev instance, but I don't even know what it is. Like, I just know it's a GPU. L40S is like a, is a mid to low range GPU that is inference focused, like inference and data science focus. So it's not like a big like training GPU. So they're very common in like developer clouds.

Since they're not like a high end like flagship GPU, if you're just like doing like code development, you need a GPU to test on there. That's a common option to use. I've, maybe it has like 80 gigs of memory or 40 gigs of memory. I don't remember it. had some reasonable but not a, you know, flagship amount of memory and it's a slightly older chip. But so I picked a model. I was there live at the conference.

So there were like two constraints. Like one, I didn't want to, you know, go and get a beefier GPU. I could have. But that actually wasn't why I picked a smaller model. I was like, the conference is happening right now. I need to pick a model that is small enough that I can download it and like deploy it in a reasonable amount of time. And so like I, you know, I was chatting with my agent. and my agent's like, yeah, this should take like 30 minutes or so to set up.

But so the model that I picked was not a multilingual model. It was where I specifically downloaded a Chinese to English variant of this model. There were other models that I could have downloaded, which was just like any language to English or any language to any language, but those are necessarily bigger models. And the mistake that I made is that there was, it was a Chinese to English model. So I had to deal with code switching. So during the conference talks, someone would say something, hang on, one second, one second, Conner. Honey, I'm in the middle of recording the podcast. I understand, honey, but now, now, now Conner's going to have to edit this.

I picked this Chinese to English model. And, you know, then what would happen is a speaker would be, you know, talking, they'd be saying something, something, something, something, KV Cash, something, something, tense or parallelism, something. something, something KV Cash, right? Because if you're giving a technical talk, you're going to often use words that are typically English language words specific to your technical field. And if you have a model that only understands Chinese, then it hears KV Cash. And it's like, what Chinese word did you just say? And it gets very confused. And so that completely threw off the little system that I built. I mean, it did like an okay job. relative to Google Translate. And I later talked to some of the people who make the NVIDIA speech models.

And they told me, you know, the model that you used had picked a slightly older version of our SDK. You should have used this instead. So I'm never at a future conference where all the talks are in a different language. Maybe I'll have a chance to rebuild that thing. Anyways. Well, no, I just, I feel like it's so, it's so, maybe we should spend this episode. We should just talk about the different things. You know, you're using this for.

To be clear, this was a thing where I, like, built and stood this up. Like, it was a two-day conference and I, like, built and stood it up and had, like, a nice little web app and, like, app on my phone for this in, like, a matter of hours. It was very fun. Yeah. I mean, you can use, oh, man, it's just. But, but, but honestly, like, like, doing that, like, oh, building some little, like app like that, like, that's, that's, like, so, like, three months ago. I'm all about, I'm all about the auto, like, so. So, okay, the fun part for the listener here is somebody pointed, like, we have this codex leaderboard within a video.

Jesus. And how did you find out about that? I don't know if we should. I don't know. Somebody pinged me because I was 31 on there. So Connor is number seven on the codex leaderboard. And you might think that that, you might think that that means that Connor is winning. However, the reality is that I stopped.

I don't want to be high on this leaderboard when when people get fired for spending too many tokens. I had to stop use. So at NVIDIA, we have, you know, a corporate, like, subscription to Kodx, a corporate subscription to Klaude where we pay it, like, API billing rates. But then, like most big tech companies, we also have our own LLM gateway. And I had to stop using our corporate Claude code subscription because I just hit the cap too quickly. And likewise for codex, well, it's less about likewise for Kodak.

all of the like billing and like metering for these things like it's awful like with like I feel like I could never know you know what how many tokens I was using or like how much it was costing and so I just moved all of my usage to our invidia inference hub and centralized it there so the the 31 rank which I'm surprised that I was that high but I guess I did do a bunch last week before I split about a bunch last month before you're primarily using cloud code right? That was only a chunk of my usage. I was primarily using opus. I do primarily use Claude code as a harness. I do find it to be the strongest harness for the sort of stuff that I'm doing, which is long horizon tasks with complex multi-agent workflows that require multiple concurrent agents and multiple

concurrent agents that are running tool calls that need to be carefully synchronized with each other and concurrency safe. And I have found that Claude Codd codes model for subagents and subagent completion and waiting on subagents is better than codexes. Because in codex, I've seen in codex, like basically there's like a thing that you call. to like wait on subagents. And like you have to like explicitly call that by at least my understanding to get like notifications of when stuff finishes. Whereas in Claude code, if you start a background agent, it will eventually send a completion notification. And so I've had more issues in codex with subagents completing and the manager agent not being aware of that. And I also just in general have

found GPT 5.5 to, uh, I don't know, not be as good. The models that I, I'm working with these days is Opus 4.8, GPT 5.5, deep seek V4 Pro, and uh, uh, Nemotron 3 Ultra. And now I'm doing like complex stuff where I have like different, like I have like one model as the manager and then different models as the workers and stuff like that. Yeah. It's, and, and, well, I don't know how much we should tell people about this leaderboard, but I was, I was shocked. I, when I found out when because Bryce ping me yesterday and he's like congrats on being number seven and I I was like because the thing is is I keep waiting I know that you you've received your manager has received a

couple emails about your spend I have not been notified about any emails of what my spend is and so I just always assume if I haven't received any emails like I'm not hitting some threshold but the fact that I'm number seven on this list out of like 25,000 people using Codex is, I would expect there to have been an email, you know, like, is, I really hope this isn't costing the company. It's, it's because we're doing the same thing. It's because we're both doing, we're both doing like auto research. Yeah, yeah, orchestration of multiple agents running at the same time. And I mean, that's the thing is like, no offense. You can tell me if I have to cut this or we can keep this in. We had, but it was just like there was another guy I should I should find it but there's a guy that recently

joined in video from Google that gave an internal talk at a software engineering something let me find the name of this guy and I'm not sure if he's given the name of the talk was or the name of the event is agent native tools and coding and if I open it it hopefully will show me the name of this guy If I hit the play button, is it Colt, Mick, Anless? I probably butcher the pronunciation of that, of that name. But he gave, like, a talk that's, like, eight stages of agentic coding. And, like, level one is, like, you know, chat GPT or Gemini in, like, a browser using their web interface. And then, like, step two is, like, a CLI, you know, command line interface.

And then you get all the way to step eight. And it's like, you're developing. I think what he called it was a knowledge garden, and you have like an orchestration system where you've got... Anyways, the... We'll probably have to trim parts of... Good chunks of this. Anyways, we'll just say that the talk that Colt,

who recently in the last year, I think he said, had joined Nvidia that he gave, was, in my opinion, a hundred times more valuable because it shows what at the limit you can do with these tools. At the limit, this is what... you can aspire to be doing with these tools because like at the limit it's amazing how good these tools are and yeah so i'm going to i don't worry about how much the inference i'm using costs and there's a couple of reasons for that and i'll explain i would worry if the the thing that i was

doing did not seem incredibly valuable but i i'm very very confident that the thing that i was doing that I'm doing, the general area that I'm working in, is incredibly, incredibly valuable. So that's a big part of it. I know there's a bunch of other people doing similar stuff. And a lot of them are more worried about like token efficiency first. My first concern is capability and functionality. Can we build a thing that does what, you know, that does this incredibly valuable work. And I think it's a lot easier to first prove that we can build the thing that is incredibly valuable and then later figure out how to optimize it and reduce its costs. And I've just sort of started on that process on figuring out how to do that. But if you go in with a mindset

of, oh, I'm not going to use the frontier model because it's too expensive. then you won't know what is capable. And in particular, if you're doing something like auto research, the real value of auto research is in discovering novel ideas. It's almost like basic research, you know, like answering the questions of the universe. The value in auto research is when it can be an idea factory, when it can come up with things that are like net new novel that aren't in literature that you look at and you're like, hmm, you know, if I had assigned a, you know, a senior engineer,

a researcher, you know, to go work on this for six months, this is the sort of thing of novel revelation that I would expect to come out of that. And I think a lot of people don't have the right philosophy when they think about things like auto research because they're focused on, they're focused on determinism. They're focused on like an efficient and clean process. But that's not typically how humans birth like new ideas. Like if you have an open question, you have to have an open search space and there will be failure. That is part of the process. The other reason why I don't worry so much about my inference usage is because I think it's very clear that this is going to have a huge transformative impact on our industry. And part of my job, my mandate at Nvidia, is to figure out how people program our platform.

And, you know, I think of myself often as being a power user of things, right? I'm not necessarily a compiler engineer or a driver engineer, but I am a power user of those things. And so for me to learn what the best practices should be, I got to go around and try crazy stuff, you know? Like I got to go and see like, okay, like what happens if I go and throw, you know, 50 agents at, you know, winning the GPU mode leaderboard? You know, what can I learn from that process? So, you know, if you're just going and burning tokens and your only output of burning those tokens is like the output of the task, then that's maybe more questionable than if your outputs is not only what the task achieved, but also analyzing how it got there and learning from how it got

there. And using the data of, okay, I, I, you know, I spent 20 billion tokens to do this thing. That's a lot of data that you've generated. And how can you learn from that about how parts of your stack could be more agent ready, more efficient? You know, how could you improve your APIs, your tools, et cetera? So, like, yeah, I'm spending a lot of tokens, but I get a ton out of those tokens because now I have all this data and I can go and ask questions about this data. Like the other day, like, there's a particular command line tool in Vida and like people are always asking me, Bryce, how do we make our thing, our piece of software like agent ready and agent friendly?

And the other day, I was just like, you know what? I'm just going to go like, hey, like, go to cloud. Like, here's where all the data is. Just like go, like, go look for all the times that this particular tool was used. And like, give me, give me information about like every time we called this tool and did it succeed or did it fail? And then from that, I can learn like, oh, you know what? This command line tool, maybe we need to add this flag to it or something like that. So I think you have to, you have to treat your session logs are very valuable, I think. And it's not just like,

like, you know, it's not just the data, the hard data that you get out of it, but it's also like the learning experience that you, the engineer, get, right? Like, this is a whole new world, a whole new way of working, so you've got to go and kick the tire on it. And I think we all have to assume that there's a future world where token costs are a thousand times cheaper than they currently are. And if token costs were a thousand times cheaper than they currently are, then, you know, boy, like that would, that would enable everybody to do this sort of crazy stuff that you and I have been doing, Connor. And that is a very, a very interesting world. Yeah. I mean, I don't want to call people out, but I definitely have listened to folks on certain podcasts.

that I listen to. And when they whine about what these models are capable of, I think in my head like it's user error. Like you haven't spent enough time learning how to like wield these tools. And I think it's a privileged position that we're at Nvidia and we get access to these tools. And like you said, I mean, I'm more concerned about my spend versus you. But that's just because I think you're a more confident New Yorker and I'm an apologetic Canadian type. I'm just waiting for someone to send me an email.

But I do agree that there's like an immense value from like daily driving these things. And even like if I'm using it for some like, you know, random, not even like auto research thing, you learn things from like staring at the trace, realizing like, you know, one time I was doing something with cursor and it had like failed to recognize both the JSON files that were storing some data and then like a Python script. And I was like, without going into details, I was just like, whoa, whoa, whoa, like, what's happening right now? And I started having like a meta conversation being like, why are you not surfacing the information

in this JSON? And then it was like, there's no JSONs in these, like, this directory. And I was like, what? What is going on here? And then like 20 minutes later, after going back and forth with it, it realized that like somehow it had become, like, developed a blind spot to certain files in the repo. And then I was like, well, how do I make sure that this never happens again? again and it was like oh well we have these like dot for cursor i think it was like mdc which were like

the rules and it was like this is the first thing that i ingest and like i understand that you had like agents dot md or some markdown that stored some stuff that in that markdown it said you know blah blah and even without that i should have been able to look at everything in this repo but just for whatever reason i miss that but if you if it is essential that like this data is needs to be at the top of like the context for every conversation you have, just put like a little sentence in your dot MDC file or even just ask me to put that essential information there and like it'll never happen again. And that was like for something that wasn't like auto research related. I was just like asking it a question being confused and then was like, wait a second,

like we're at the point now where it shouldn't be making this kind of mistake. And and then I realized that like, oh, like it still can make mistakes. If you put the. important information in the wrong spot. And like there are like basically places for whether you're using, you know, clod code, codex, cursor. Each harness is a bit different. And like I even asked it. I was like,

yeah, I've run into this a lot where like harness differences, especially when you're doing like multi-agent orchestration stuff. You become very tied to the particular way that a harness works and that the harnesses tool, tooling works. And like I ran it, I had this weird thing where like initially I couldn't get, I couldn't get models other than.

Opus to like drive ClaudeCode well. And I thought that the models just were not capable of handling this workflow. But it turned out that really there was just like a bug in the structure of my skills that when it was going to launch, you know, a sub-agent, the way it was describing what to do was just not correct in that there was a better way to do this. And with Opus for, you know, maybe because they post-trained it to for the harness specifically or for whatever reason, it just was sort of smoothing over this, you know, this bug in my skills. I think you make a good point there that when something goes wrong with NOLM, you know, you have two options.

One, you can just tell it to fix it. Or if it's something that, you know, if you want to just put a band-aid over it, you know, you just tell it how to correct itself and move on. But if you're building like a repeatable, like, workflow or a skill, then you want to be like, no, hang on, like, you know, I'm going to be like a psychologist now. And like, you, like, you sit down with me and like, explain to me why you did this thing that you did. Like, don't, don't try to fix it. Just like, explain to me how we got here. And then, like, from that, you know, you can get insights into, into, you know, what you might need to adjust about, about your

process. And, you know, part, I think part of this, you know, having, having, like, more reliable evals and having ways of like testing and evaluating skills is really important because of this. It is very challenging for auto research to do evals because most of the crazy behavior happens 24 to 48 to 72 hours into an auto research run, right? Most of the interesting weird bugs happened, you know, after there's been some compactions, after it's, you know, there's some context rot. And so I don't know how. to, I don't know how to reliably create a CI system for stuff that needs to run like 72 hours and currently costs, you know, many tokens. I think at one point over the weekend, so you suggested to me on like Friday, they're like, oh, like, you know, we should look at GPU mode.

The GPU mode is this GPU programming community and they have these regular coding contests of which my bots currently hold. the winning spot for like six of the seven or six of the five of the six problems. But so like I just like spun this up over the weekend and like tried out a bunch of new things. I think at one point over the weekend it was like 500 million tokens an hour or something. That's that, you know, before cashing. So fewer than that are uncashed. But it's still a lot of tokens. Yeah.

Have you tried Nematron 3? ultra yet? No. You should try it. It's very fast. It's very fast. And yeah, I've been trying it out a lot, kicking the tires on it. I'll add it to my list of stuff to do.

Yeah. It was funny. We had this meeting on Friday, and we're Connor and I were just chatting. And halfway through the meeting, I'm like, hang on. Connor, I got to go check on my agents. And Connor's like, yeah, I also have to go check on my agents. Yeah. And then I remarked, what does it say about us that we're meeting for like 60 minutes?

And both of us need to go like garden our agents to, you know, kick them off. You know, one of mine just runs flawlessly by itself now for as long as I want. But other times when you're steering stuff, you do need to go check. And that was one of the things. You know, we should bring Colt on the guy that worked to Google and recently joined that gave that presentation about like the eight levels of agentic code. And because like one of the things that he remarked, which he's not the first person to remark this, we both have made the same statements and others have is that like, you know, we're, we're busier

than we've ever been before because these tools are like a unlock of so many different things. And well, also it feels like it's a race. It feels like it feels like there's a race, you know. I guess, yeah, a little bit, a little bit. Yeah, I mean, there's a lot of people doing this kind of research. and, you know, trying to figure out what's the best way to use these tools. But it's, yeah, it's this, it's this like you, you know, there was a period of time before I had figured out the right way to use these tools so that, you know, I could go to sleep and it would

just work. And like, it is, it is intoxicating while this thing is like making progress. And then like, whenever I would just like randomly wake up in the middle of the night, now I reach for my phone because like I serve my like little dashboards to my local IP address. So I just like, I go to my local IP address, colon, you know, whatever the port that I have it hosted on. And that was, that was another thing that I realized is I was like, wait a second, I have all these dashboards. I can serve all of this just like locally on my IP address with like, you know, I don't even know what the command is, but like, you know, usually on your desktop,

you go local hosts. But it's like one extra thing to do just so that you can look at it. And now, like, I go to watch TV, you know, I just like, well, I told you about my thing, my, where I had this Mattermost server where I can, you know, where all of my distributed agents that are running and multiple nodes are connected to and I can just go check on your phone. That's the, I haven't done that yet, you know, it's so that any time I'm going for a run. You got to go, you got to go cloud. You got to go. At the scale that you're at, you got to go cloud. It's funny. So I, I, um, I have a dashboard too and I usually watch the dashboard, but over the weekend, I didn't have to check the dashboard. I just like went to the GPU. Oh, yeah, yeah, yeah, yeah. You know, every couple hours I was like,

oh, okay, well, we've, we've claimed. we've claimed prefix some. Yeah, that is so funny, yeah, because there's a, there's a, there's a couple of those as well. And like, yeah, you, if you're in your house or whatever, you can go and look. But then also, too, yeah, I just go check. Then you look at the status to see if you have a submission in the last couple hours. And sure enough, it's just cooking. And anyways, we should, we should, we should, we don't need to necessarily stop talking about AI.

But we'll wrap this as episode one is whatever, our latest updates. Be sure to check these show notes, either. in your podcast app or at ADSP thepodcast.com for links to anything we mentioned in today's episode, as well as a link to a get-up discussion where you can leave thoughts, comments, and questions. Thanks for listening. We hope you enjoyed and have a great day. Low quality, high quantity. That is the tagline of our podcast. It's not the tagline. Our tagline is chaos with sprinkles of information.