How I AI - What a harness is and how to build one with Claude Agent SDK

Starting point is 00:00:00 A harness is some code around an AI agent that makes it more effective. Why we've seen people build these specific use case harnesses is sometimes with a specific job, you just want to micromanage a little bit. You just want to be more prescriptive about how that job gets done. I'm going to show you how it works and then we will talk about how I built it. So the interface I built for my harness is a terminal UI. The harness core is run on Claude, Agent SDK, and then it's connected to real tools. So it's connected to Century, Bersel, and then it's connected to linear and GitHub in terms of getting tasks done.

Starting point is 00:00:40 I think we all have done good work. But then now I've realized that these agents can help us solve very, very specific problems by constraining that work. It's really like changed my mind about how work gets done. Everybody's saying it's not the model. It's the harness. But you know what? not everybody is saying, what is a harness? In today's How I AI episode, I'm going to demystify the idea of a harness, write my own harness and show you how you can do the same, and explain to you

Starting point is 00:01:13 why a custom harness makes sense and could be better than using ClaudeCode or Codex alone. Let's get to it. This episode is brought to you by Bolt.new, the AI app builder for people who have ideas and want to ship them. Most AI tools spit out code that looks great in the demo and falls apart the second you try to do anything real with it. Or they lock you into their own platform with no real way out. Bolt is different. You describe what you want to build,

Starting point is 00:01:41 a startup MVP, a landing page, an internal tool, a side project. And Bolt generates production ready code in minutes. Connect Stripe or any other MCP, hook up your domain and deploy it live. Founders are using Bolt to build businesses doing real revenue. Product managers are shipping prototypes their teams actually use. Designers and marketers are launching campaigns without waiting in line. Anyone can build. Engineering can ship.

Starting point is 00:02:09 Everyone wins. You just need an idea and a weekend. Check it out at bolt.new slash how I AI. Before I get into how to build a harness, let's talk about what a harness is. is and I am going to make it as simple as I can for all of you. A harness is some code around an AI agent. Yes, you heard it here first. A harness is just code around an AI agent that makes it more effective. Can that code have AI in it? Sure. Does that code have to have AI in it? Not necessarily. What is the goal of a harness? To make the AI better. It is so simple. And I feel like the way that

Starting point is 00:02:52 people have been talking about this have made it such a mystery that I wanted to make it just very clear to you all. It is just writing more code around your AI to make it more useful for a specific use case. So what are the parts of a harness? Well, a harness is going to have specific context. It's going to be able to take specific actions and it's going to have a goal of specific outcomes. It's just as simple as that. And I want to talk about when it makes sense to build a harness and when it doesn't. And I think you want to build a harness when the same workflow needs the same setup and the same outcomes. And so it's kind of similar to when you would build an AI agent. And in fact, harness agent, sometimes you can interchange some of these

Starting point is 00:03:41 concepts. But really it's when there is a sort of combination of deterministic and non-deterministic workflow step by step, process tools, use cases. You want to your AI to follow up to do a specific job. Usually those jobs are like slightly more complex. And this is why you've seen these coding harnesses come out. Like coding is a job to be done. It needs specific tools. It typically goes through kind of a standard workflow. And so coding harnesses are very popular. But you could also do things like managing production incidents where you need to go through a specific process, getting PRs ready for release, handling support escalations, managing migrations, even non-technical use cases like doing research in a very specific

Starting point is 00:04:28 way or consolidating docs in a very specific way. That's how you and why you would use a harness. So how did I decide what kind of harness I would build? Well, I looked across my business at chat pardee and I thought, what am I doing sort of repeatedly and consistently that I think AI can be good at, that I think we can be doing better if we were more structured about the AI and how we used it. And I thought that fixing bugs, you all, if you've listened to this podcast, look, I ship code so I ship bugs. Fixing bugs is a very specific workflow where we've built some custom internal tools that I've been generally doing with cloud code or codex, but I had this hypothesis that I could do a better job of triaging bugs if I built my own harness.

Starting point is 00:05:15 And so I picked Century Debugging, and sorry for the Claude Slop content here, Century Debugging and Debugging Century Issues, really figuring out the issue, using some of our custom internal tools, and then doing all the follow-up actions we do when we close bugs was like a good first harness. It had coding in it. It needed custom content and custom context. There were like specific outcomes. I wanted to make sure that we followed, like tracking everything in linear and writing follow-up

Starting point is 00:05:47 docs that the rest of the engineering team could use. And so we chose debugging our century bugs by we, I mean me and codex, chose debugging century as a good use case to demonstrate how to build a harness. Now, why wouldn't I just use an AI coding tool directly? Well, I have been using AI coding tools directly. And I think the problem with using a general purpose coding tool and why we've seeing people build these specific use case harnesses is sometimes with a specific job, you just want to micromanage a little bit. You just want to be more prescriptive about how that job gets done. And so if you can identify the right workflows, you can actually be more efficient, more consistent, and that have better outcomes if you build a harness. So for this specific

Starting point is 00:06:37 use case, you know, with a direct AI tool like cloud code, I would have to explain what I want the agent to do. So I'd have to say, like, dear agent, please fix this bug, here it is, and send the link. Instead, it's harness. I can literally just paste in the link and the agent already knows my intent, already knows what the job to be done. A second thing that I wasn't that worried about, but is interesting when you build your harness is you can be really prescriptive about what tools it's allowed to do and what it's allowed

Starting point is 00:07:05 to execute and not. So for example, if you wanted to build an investigate only harness, you could make sure that your harness, your code editor never actually wrote code. It only explored and explained root cause. You can also repeat the same process over time if you encode it in a harness. And so if you want like a very precise step by step flow, including outcomes. So for us, every time we fixed a set tree bug, we want it documented and linear. We want a very specific report. We might even want to follow up with customers that it was impacted with.

Starting point is 00:07:42 You could encode that in a skill, but then again, you have to babysit it. When we built this harness, we knew it would happen every time. And then from a model perspective, you can do multi-model routing and all sorts of interesting things in ways that you couldn't with a general purpose AI model. So I'm going to show you how it works. And then we will talk about how I built it. Okay. So the interface I built for my harness. is a terminal UI, again, like quad code or codex, something you run in your eye, in a UI.

Starting point is 00:08:13 Just so you know, your harness does not have to be a TUI. It doesn't have to be a CLI. It doesn't even have to have letters. It could be a web app. I did it in a TUI, one, because I haven't built one in a while. I thought it would be fun. And two, I just want to show that building your own custom harness means you can build your own custom interface into these AI agents as well. So the harness is the whole experience, including the human experience that makes it more useful and easier to use. And so this TUI is pretty easy to invoke. I just run TUI. You can see it here. It's kind of cute. It's been made cute. I use this library called ink, which helps you make cute TUIs. I don't think they would say cute, but I'm going to say cute. And you can see here that this terminal UI really reflects the structure of the harness itself. So you see all the run.

Starting point is 00:09:05 that it's done so far, errors and how it's fixed things. And then sort of our harness process, which is it gathers evidence, it streams and activities, and then it builds some artifacts. And so I'm going to actually have it investigate the century error over here. It's one where our edit operations are getting dropped sometime by the agents. And that has now kicked off our specific harness. So what it's going to do is it's going to start this investigative. investigation run. It's going to kick off a clawed SDK session, which is a fundamental part of how I built this. It's going to go ahead and start gathering evidence and coming up with a root cause hypothesis of what's causing this issue and how we might fix it. Now, as you can see, I chose I

Starting point is 00:09:52 investigate, not F, fix. So the investigation should not touch and modify files. And again, this is something that I would have had to like prompt to the agent and say, I only want you to investigate. gate. I do not want you to ship a fix, but instead I can just click I paste in that century issue and it's off to the races. This episode is brought to you by customer I.O. You're here because you'd rather use AI than talk about it. With customer I.O, you describe the campaign you want to build and the AI agent creates it for you, the audience, the messages, and the timing. You review it, make any changes you want, and launch.

Starting point is 00:10:35 Instead of spending hours stitching together tools and workflows, you can focus on the work that actually drives growth. Every campaign is tied back to results, so you can see what's working and what to do next. More than 9,000 brands use Customer I.O to turn the data they already have into messages customers remember. Visit Customer.io slash how I.A.I. to try it today.

Starting point is 00:11:01 Customer I.O. More impact from every message. While this is running, I'm going to just go and show you a little bit about how this works and how I have actually built it. Okay, so this is the high level architecture of the app. So the front end is a terminal UI or a C-CL. Each invocation of the harness, we call a run. So it's running a task. Each task has a specific input.

Starting point is 00:11:28 Usually that's a century issue. And then there are specific flags I put on the harness that allow it to edit the source. source, modify the inputs or even message customers only if I flag and approve it. So again, this is just a little bit more control over how the agent works. The harness core is run on Claude Agent SDK. And so all the agenetic planning is run through the Claude agent SDK, which has some of the primitives of Claude coding, including grapping files and writing files and all those sorts of things that we find useful.

Starting point is 00:12:01 And then what's really interesting about this harness, and you've seen in other harnesses like OpenClaw, is it can create its own artifacts in its file store. And so we have this artifact store. I will show it to you in a minute. And it basically saves all the evidence from these runs to the file system for the agent to use in the future. And then it's connected to real tools. So it's connected to Century for Cell, the Claude SDK. It's running Sonnet 46 because I think that's the right model for the job and then it's connected to linear and GitHub in terms of getting tasks done. Now, what's really interesting as well is you can prompt this in a custom way. So instead of the general like you are Claude Code, make no mistakes, you are, you know, sort of model genius, I'm saying specifically that you're working inside the Chapyarty-engineering harness. It's Chapyardi-specific. It's not an open-ended coding system. We want to use these artifacts. as a source of truth, and here's the plan to attack a very specific problem. And what I want you to return is X, Y, and Z. And again, I don't have to copy and paste this. I don't even have to put it

Starting point is 00:13:14 in a skill where hopefully it will get invoked in the right way. I've actually encoded this in a very specific step in the harness to make sure that the model falls at every time. And so there's several of these types of custom prompts inside my harness. There is the artifacts that get generated, there are tool policies around like what tools can be called and which ones can't. And then I have just decided again to use Clotsonic 4-6, which is really, I think, the right model for this particular workflow. Okay, I want to talk a little bit about the code and how you generate this. And then just like a peek behind the scenes. I actually ran dueling Claude Code and Codex sessions and essentially said like, help me build a harness.

Starting point is 00:13:58 I think I want to use the Claude Agent SDK. hey, here's what I would like it to do and then like closed my eyes and tried to get it done. Honestly, it was not a one shot. I don't know if it was my prompting or the models were being funky. It was GPT 5.5 and Opus. But both of them really wanted to build something super deterministic. So they like really resisted putting any AI in the harness and I had to really prompt it very, very specifically to get what I want.

Starting point is 00:14:29 So I would say if you were. trying to do this, I would be very specific about the workflow. I would be very specific about the tools. I would be very specific about where custom prompts makes sense. And then I would suggest using an agent SDK, either from Claude or from OpenAI, to run most of it. Because without that prompting, I just did not get what I wanted out of these models. The second thing I will say, funnily enough, Codex did the best job of building the agent. But it used, cloud agents SDK to actually implement the agent. So we are spanning across models and spanning across coding agents here. But the actual harness itself is pretty simple. It's got sort of a

Starting point is 00:15:14 high level index of how you get to the TUI. And then it's got like, I don't know, eight files of specific things it can do. So it can hunt for bugs in Century. It has a sentry adapter to effectively use the Century API in a very specific way. So instead of using the MCP generally, instead of like having your coding agent wander through all these traces, I'm just very precise about exactly what I think you need to pull from a bug report perspective, what's useful, what's not, and made that connector really opinionated. It's got similar, a linear integration, a Vercel integration, a GitHub integration. So again, not like generally how you can use these tools, but specifically how you would use these tools when you're searching for a bug.

Starting point is 00:16:01 And then after those tools and data sources are used, the bug is identified and triaged. Then there is this artifact file here that outputs and spits out the specific artifacts I want to see after a bug run is done. And that artifact bundle looks something like this. So it's literally just the task run. which is all the messages, the report. So what was the sentry issue? Here's a brief on what we discovered. Here's any logs that we think are relevant, what the Claude Worker ended up doing, and then the summary of the output. And then we also output this beautiful HTML file that I can

Starting point is 00:16:43 show you, that shows you what happened and how it all worked as well as a worker report. So I will show you those outcomes as well. Just pulling up this code for you again, it's pretty straightforward. It's given me all the instructions on where to put my specific API keys. And then I can just run it in this very opinionated way. So in addition to running the TUI, which lets me sort of like navigate through the UI and use this harness, something I might want to do as a human, it also has built these really easy command line tools where if I just quickly want to run this harness against specific issues with specific flags on tool use, I can definitely do that.

Starting point is 00:17:26 And what's kind of interesting about this is, yes, I built this harness. And you can see here I built this like fun UI so that I could use it in a fun way and it makes for a better demo. But really, this harness is a structured way to give agents the job of running these investigations on a simpler basis. And so you can imagine while I design the TUI for human actually giving a kind of all intelligent agent a specific harness to solve a specific problem with agents in that, I think that's how you're going to get real leverage and really custom outcomes out of things like coding agents like ClaudeCode. And so going through this process has really opened my mind to we've gotten so used to like the open chat field. Like if I just type in, the agent will do good work. And I think we all have

Starting point is 00:18:19 done good work. But then now I've realized that these agents can help us solve very, very specific problems using other agents. And by constraining that work, we can actually get specific jobs done really efficiently and then use the general purpose agent to sort of orchestrate it. So it's really I changed my mind about how work gets done. As you can see here, again, it's just a couple files. It's really not too much. The adapters to the data sources, a couple workflows in particular, this bug hunter workflow which just goes through exactly how we want to hunt bugs, including how we want to put together summaries of bug reports, and then some files here in terms of running the TUI or the CLI. And then as I said, we have this artifacts folder that gets updated every time a run happens,

Starting point is 00:19:13 where I can click in and actually see exactly what happened out of a run. So let's go and see if this run happened well and what I can find out. So now I have the full context. Here's the investigation brief and I can go look for it. So this is Bug Hunter C7. Let's see if I can find this one. Here it is. Here's the investigation brief on that edit document operations dropped.

Starting point is 00:19:41 I have confirmed evidence. So it's saying, yes, there was definitely a century warning. It's impacted 150 users. it's still happening hourly. It's a warning, so it's not an actual error. And then Varsel logs were unavailable. And so we weren't able to use that data. And then it found likely root causes. So invalid original range or overlapping original range. And so it's identified a couple potential root causes as well as a blind spot in this particular function. It's told me exactly where in the product surface the issue is. And then how I

Starting point is 00:20:18 I would actually verify this by fetching a raw century event to see if the issues that they've identified are correct. It's identifying should it issue a linear issue and it says yes, we should definitely make a linear issue to fix this. And so this should get assigned to somebody. And then it doesn't recommend turning on patch mode and actually fixing this. So again, this is like a very specific outcome I wanted. I wanted to say like, what's all the evidence?

Starting point is 00:20:46 Priority rank the root causes. make a suggestion on the next step if we need to verify this more. Tell me if I need to assign it to somebody in linear and then tell me if you can fix it. And they're saying, no, I don't think I can fix it yet. I need a little bit more information. And all of that is built because I have done this like very specific workflow and encoded that in what we're calling a harness, which is just code around an agent. So how would you build your own harness?

Starting point is 00:21:17 I feel like hopefully you're still with me. Not too much of that went over your head. Just to reiterate, I just identified a specific workflow. I determined what the run against the task would look like. I made very opinionated calls to tools or data sources. So I didn't just say, like, use an MCP, although that could be part of your harness. What I did is I made adapters that made the calls to these external APIs and tools very specific. I thought about what the structured artifacts out of that workflow might be.

Starting point is 00:21:52 I decided what rules and permissions I wanted to give this harness and which ones I didn't. I decided whether I wanted to use Claudecode or Codex or a model router to actually run these things. And then I built a surface to interact with this agent. So I built a TUI so I could actually look and work with this harness in a way. It could be a TUI. It could be a CLY. It could be a web app. but I built some way to interact with this.

Starting point is 00:22:18 So this is what you need to do. Identify a workflow. Really write it down on, you know, proverbial paper, HTML or Markdown, figure out what sources of data you want and then plug it all into Claudecode or into Codex as I did and have it build your own harness and then test it against real data. So that's it. I just, I really hope that you walk away from this, realizing that these mystery terms like harness are not that mysterious. A harness is simply putting some structure around how AI works. Yes,

Starting point is 00:22:53 cursor is like a really complex harness. Yes, Codex and Claudecodecode are very complex coding harnesses. But the end of the day, they're code that wraps these AI agents and these AI calls to make them more efficient and doing a very specific job. And so whether you're doing that in a very prescriptive way, like I just showed where I want to show you how I triage set, country bugs, do the investigation and pass it on to the team, or you're doing it a broad way, like these general purpose coding agents that just have access to tools and context and methods that make the coding workflow better. That's all a harness is. You can think of harnesses that you can build. You can build them in the terminal. You can build them for CLIs. You can even build them as web

Starting point is 00:23:38 apps. I'm starting to hypothesize that a wrapper is just a harness and that is going to upgrade everything that I've vibe coded over the last three years. This has been totally a learning experience for me here on How I AI. This is my very first harness that I've built live on the show. I hope it's useful for you. And if you're interested in me building other things and demystifying AI terms, let me know in the comments. Thanks for joining How I AI. Thanks so much for watching. If you enjoyed this show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app.

Starting point is 00:24:21 Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at how-I-A-I-I-I-I-I-Pod.com. See you next time.

How I AI - What a harness is and how to build one with Claude Agent SDK

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.