The AI Daily Brief: Artificial Intelligence News and Analysis - Everything You Need to Know About AI Tokens

Starting point is 00:00:00 Today on the AI Daily Brief, an operator's cut episode with NewFAR, everything you need to know about AI tokens. The AI Daily Brief is a daily podcast and video about the most important news and discussions in AI. All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, Rackspace, Blitzy, section, and airtable. To get an ad-free version of the show, go to patreon.com.com or you can subscribe on Apple Podcasts. And to learn more about sponsoring the show, send us a note at sponsors at AIDDailybrief.A.I. All right, friends, well, Newfar Gaspar is back today, and Newfar and I have been cooking up a lot recently.

Starting point is 00:00:44 A whole slew of you have done our most recent program, have explored our most recent program, the Choose Your Own Adventure style AI summer adventure. Plus, we've been cooking up an expanded set of educational resources that we'll be telling you about soon. But one of the realities that both Newfar and I have been living in is every company we interact with dealing with the same questions of AI tokens and token economics. We are now firmly in the agentic era of AI, where companies have to think not only about how to get adoption and how to maximize AI's value, but how to do so in a way that doesn't just totally break the bank and where the right intelligence is being used for the right problems. Anyone who's ever built an open clock can tell you that getting the right models to do what you want them to without going off into endless cycles of spin take some real consideration. Today's episode is designed to be the ultimate primer on AI tokens.

Starting point is 00:01:33 What we're talking about when we say that term, what the new challenges are, and some of the key pitfalls to avoid as well as strategies to maximize the way that you and your company use AI tokens. All right, Newfar back with another operator's cut talking about the topic de jure, the topic on everyone's minds. We are talking tokens. How are you doing? I'm good. Very psych to talk about tokens. Yeah, I think it's a, I love this period in a discourse where we've gone from sort of pulling hair out free. out about the new change to actually settling into new tactics, new strategy. And I think this is a perfect fit with that. So tell us a little bit about what we're going to be talking about.

Starting point is 00:02:12 And let's dive in. Good. So the reason why I wanted to do this episode is because every room that I walk into these days, like literally every room, have some version of the same token conversation. Some practitioners feel like they are being watched when they use an expensive model. The regular users wonder whether one ambitious prompt will eat their weekly allowance. and the leadership teams, they see a bill growing faster than expected, and then they start asking a lot of questions on whether all of these tokens produced anything useful.

Starting point is 00:02:42 So I don't know where you guys are sitting, but there is a new anxiety around using too much intelligence. And I actually want to flip the conversation, first of all, to make sure that everybody understands what tokens are and what the bill actually means, and then how to spend them wisely rather than sparingly. So that's why I'm here and what I'm planning to do today. One of the places that I've found myself with this conversation is there's been such a visceral reaction now as the cost has gone up.

Starting point is 00:03:08 I have a bigger concern around people retreating back to known ROI biases and not boring, but ultimately low stakes use cases, let's say, as compared to what AI can actually do, that I've found myself in the position of having to defend things like token maxing and token leaderboards just relative to where the tone is shifted. I think obviously we'll get into today the smarter version of that conversation. So I'm excited for it. Exactly. All right. Because there is like a growing conversation around that, I think that there is a very like a better language around the feeling of tokens shouldn't be just a financial thing. And just recently the Open ICFO proposed a scorecard and it was called useful intelligence

Starting point is 00:03:51 per dollar. And that's built around the one question. What does each successful task actually cost? And that's the conversation that I think people should. have. And I want to help you with the whole story, what are tokens, what your work costs and where usage create value and where it quietly leaks value rather than adding. So to kick us off and how we got in here, I want to walk you through four eras of token consumptions. And probably we'll recognize where you are. And we all started by being basically token oblivious.

Starting point is 00:04:22 That was the all-inclusive era where model companies subsidized the usage and the flat subscriptions hid the meter and many individual users and many of them are still there, just see the ceiling, not a per token price. So that's where we started. And then, like you said, we got into the era of token maximizing. That was the leaderboard era where usage became the badge of AI maturity. And we all remember some of the conversations around Meta who tracked employee AI usage on an internal leaderboard. They used to call it tech and they used roughly between 60 to 74 trillion tokens in a single month. And according to the data that was published, the top individual user used 280 billion tokens. So to give a sense of how many this is, that is roughly 2.3 million

Starting point is 00:05:11 books worth of text. So if you want to try and imagine that, that's about 50 books every minute continuously for a month. So that was the meta story. And then Uber launched also an adoption leaderboard and burned through the entire 26 AI coding budget in about four months. And there was another company unnamed, but according to Tech Ranch, they ran up $500 million of Claude Bill with no usage limits in place. So then in that era, usage became the metric, and dashboard measured activity while claiming to measure value. Obviously, this was unsustainable, and I know you have some opinions around that, but I'll be curious to hear your points, but I also want to say that it actually got us to be, as always, the pendulum took us way too far to the era that I call token inches, which is where we are. But if you have anything to say in defense of leaderboards, I'm here to listen. Yeah. Look, the defense is less leaderboards are a great concept.

Starting point is 00:06:05 They come with, I think, a set of very predictable challenges. In fact, so predictable that I would say that the hand-wringing around the idea of people gaming them has always struck me as a little absurd. Of course, people are going to game systems if you put real stakes around them. But that's pretty predictable and also fairly, from first principles, you can figure out a lot of of ways to deal with that. So I think, one, it's overwrought all the sort of people who are freaking out about that. Secondly, my bigger point was that a company that wildly overspends right now, via a token leaderboard or anything else, I will bet any amount of money that they will be farther ahead than a company that underspends because they're overly concerned with proving out

Starting point is 00:06:48 ROI or whatever it is on a sort of year time scale. Now, the Goldilocks scenario, which I think is what you're going to get into is being able to experiment, being able to learn, being able to build, and being able to actually understand consumption while also not being afraid of it. But I agree. I think that the pendulum swung so aggressively, far too aggressively, back from token maxing and excitement to token anxious. And that's the paradigm that we've been living in in the recent few weeks, a couple months, whatever it is. Yeah. Good. So I think you love my model of how to use tokens wisely. But with regards to being token anxious, what I'm seeing in many companies now is that many employees are self-censoring themselves, basically trying to avoid costs. And even if we look at

Starting point is 00:07:32 the same companies that were token maxing, so meta went from the leaderboard to sending a memo that constrained the AI usage. And now the press is calling it token minimizing instead of token maxing and Uber caps employees at 1,500. So even the same companies where token maxing are now significantly shortening that. And when employees self-sensor, it gets them to feel like every prompt is an ROI conversation, and that's not something that we want to have. And I think that this is a very bad era to stay in, because I believe that the most expensive token is the one that your best person is afraid to spend. So this is where I want to direct all of us to be in, and I call it the token smart era, meaning that you need to spend wisely and not sparingly and understand what creates value and where

Starting point is 00:08:18 usage quietly leaks. That's the entire kind of theme of this episode. And what I wanted to go by in this era, in this context, is to talk about the four elements of what the token actually is, why tokens were not born equal, how to audit your own usage, and how to govern or do it better within your company. And I'm trying to make it relevant to anybody, whether you're the practitioner that needs to apply some cost engineering playbook and be smart about that, or the executive and admins that need to be proactive and avoid having a difficult conversation with the CFO without a proper response to let's just cut the bill without talking about the business implications, as you just said. So that's the plan for us today. So I want to start with introducing you to the

Starting point is 00:09:05 token because I feel that even though it's the most used term in AI, not too many people truly understand what it is because that's in the root of every bill, quota and rate limit with AI in tokens. So in a simple words, token is a chunk of a text that the model reads and writes. It's typically bigger than one character and it's usually smaller than a word. And if you have never ever seen a tokenizer or how tokens look in action, OpenAI has a very good page that is open to everyone that you can just take a look at how tokens actually look. So it looks something like that. You can paste the text and then you will see how words are being chung. So you can see that some words are staying as one token,

Starting point is 00:09:47 while others might be separated into multiple tokens. And interestingly, numbers often are being chopped in the middle and so on. So we'll put it in the show notes, but a very interesting experiment if you have never seen how your text looks. And by the way, if you paste a non-English or non-Latin language, you will see that typically the amount of tokens is much larger than an English language. So that's the Open Eye tokenizer. and a few things just to lend it home.

Starting point is 00:10:16 In general, like the ratio in English is around three quarters of a word to token, meaning that if you have a page of text, it's roughly 1,000 tokens. And some languages that are like Hindi, Thai, Greek, other languages like that, might get 2 to 5x more tokens for the same content. And because billing is per token, then some questions if you ask them in other languages might cost you much more. And that sometimes referred to as language tax with AI, with code, Also, it's different and it has its own way.

Starting point is 00:10:46 Indentation and brackets and white spaces, they all become tokens. There are some newer ways to tokenize text that are more code-friendly in order to do that. But still numbers is a huge problem. So you've seen the one, two, three, four, five being chopped in the middle. And by the way, that's also why whenever everybody's doing like the strawberry test for AI, and it very badly fails in trying to count how many hours are in the word strawberry. in many cases, that's just a tokenization feature rather than a failure. And the model just has never, ever seen the individual letters. It just saw the straw and the berry as separate

Starting point is 00:11:23 words. And that's why it's counting it off. So modern models have various workarounds, but many of the AI is so dumb memes are literally just tokenizer issues. So that's tokens. In terms of what everyday work costs, I think that's a good kind of mental model to have. So for example, drafting an email is around 500 to 700 tokens. A page of text, as noted, is about 1,000 tokens. You can see a longer text can be more than that. If you send a model or a tool to do like an AI web search, often it will add a few thousand more tokens for the results, sometimes much more. Images, interestingly, are in many cases, are not that large in terms of how many tokens. They are roughly around slightly more than 1,000 tokens. Interestingly, deep research

Starting point is 00:12:12 can very easily be 70,000 or hundreds of thousands of tokens, but just the other day, one of our learners in one of our courses had a yes-no question. And accidentally, instead of asking for a web search for the problem, he was asking for a do, like the agentic tool to do a deep research. The tool spawned about 100 sub-agents to do the research. And then his yes, no question cost over four million tokens just to answer this question. So it can very easily amount to much more than that. Specifically, a few additional places where you can find very token-heavy workloads will be data analysis that can easily get to one million or more tokens per task. And heavy coding can also be very aggressive, similarly with many agentic working flows. So just to give you

Starting point is 00:12:59 a sense of where it is, and to be a little bit more concrete here, the everyday stuff, as you've seen, like the emails and so on, is almost free. So it's around half a cent and nobody should rush emails, it's not where their money goes. Search and research can multiply very quietly. So that can be a place to look for efficiency. And the top of the letter, that's a completely different sport. So if you compare like email to a gentic coding task, it can be a factor of a thousand or even more. And another thing that you need to pay attention is that every conversation compound. So the model doesn't remember your previous messages. And as such, it sends all of the previous conversations within the same session back to the model. So,

Starting point is 00:13:40 by, let's say, turn number 10, it may be processing so much of the earlier exchange alongside your new message that the total grows much faster than the number of turns suggests. That even happened before the system prompt, and we'll talk about strategies later on, but this is one of the things that can very easily, just having very long sessions, can very easily amount to a ton of tokens being consumed. I think this is one of the reasons why this is such an important conversation is Another way to put this is that the more advanced and ultimately higher value use cases consume more tokens, which is intuitive that more intelligence is required for bigger challenges. But the direction of use cases is proceeding this way. And so the reason that the token anxiety is going to create problems if not addressed is that it will incentivize people to stay swimming around less sophisticated use cases. So this is the trajectory. is clear in terms of less token consumption. You want, as a leader group, your people to be doing

Starting point is 00:14:42 more advanced, more useful things with AI. It's just how they do it well. So you want them to do deep research where deep research is required, but you don't want them to accidentally do a deep research on a yes or no question that they can Google in a second. Good. So speaking of the agentic or the more advanced capabilities, those can significantly grow the amount of tokens because agents work autonomously in loops, and as such, they consume by very widely cited industry, estimates five to 30 times the tokens of a simple chat and poorly designed agentic loops or agentic harnesses can be even worse than that, because a typical task involves between 10 to 20 model calls carrying instructions and history and tool definition and previous results.

Starting point is 00:15:27 And I think according to McKinsey, they estimate that roughly around 60% of an agentic task's cost is tied to the checking and refining and the regeneration of the answers after the first response. So the expensive part is often getting from the answer to the accepted results. So that's an interesting one. And now it gets even more complex because tokens were not born equal. So by the way, the point here is not to not use the agentic tool just to know that, as you said, intelligence cost. But it gets even more complex because tokens were not born equal.

Starting point is 00:15:59 And every model lab has its own tokenizer. You've just seen the Open AI, but different model labs have different tokenizers. So, for example, the Open AI current tokenizer has a vocabulary of about 200,000 tokens. Gemini has around 256,000. Lama by Meta has about half of that. And Claude is unpublished. And the reason why we all should care is that the price per million tokens is denominated in each lab's own tokens, and often we don't know them. And the same document can be 10 to 20 percent more tokens on one provider than another. and even more so for code and non-English text.

Starting point is 00:16:35 So the model behavior widens the gap. And one model may answer in a single pass, well, another reasons longer and writes more and takes more agentic steps or needs retries. And the tool around the model adds its own system and context and the loop design. So the two stacks doing the same task can have different token counts and different completion rates. And as a result, it's completely different bills. So the per token price is kind of the sticker, but the cost per accepted task

Starting point is 00:17:02 is the operating metric, because otherwise there is no way for you to compare between different providers and different tools. All right. An important story that also illustrates that. What happened when Opus 4.7 came on board? The tokenizer basically under the hood changed, and it was this April, and the price sheet was identical to the previous model, the same dollar per million token, but the model was shipped with a new tokenizer that produced by Anthropics on the documentation.

Starting point is 00:17:29 They didn't hide it. roughly 30% more tokens for the same text. So there were quite a few independent analysis of over a million requests that found native tokens. They grew and the accounts grew by about 32, all the way to 45%. And the real world, the bills grew by 12 to 27%. Because some of the difference was absorbed by caching. Even Simon Wilson, he measured one of his own prompts at around almost 1.5x more tokens.

Starting point is 00:17:57 So even though it was documented, the factor we paid. more for the same intelligence. And this is like a shrinkflation, right? The same sticker price, but a smaller candy bar. So nobody prints now 30% free awards per dollar, which is the case that happened there. So that's something that is constantly changing. Every lab tunes the tokenizer and often for good reasons. But the operator lessons here is that we have to talk about dollars per task and not dollar per token because the budget is like a moving denominator and it's not the way for you to try and understand how much it's going to cost. Let's talk about what tokens are used for by the AI tools.

Starting point is 00:18:33 And you have to understand that every AI request has three token layers and they are priced very differently. We have the input tokens. Those will be the prompts and the conversation history and the files and the tools definition and everything that is part of the input. This is what the model reads and this is the cheapest per token, but can accumulate fast because if the history is being recent or if a lot of context is being read, that can cost quite a lot. Then we have the reasoning tokens.

Starting point is 00:18:59 That's the second layer. These are the tokens being used for the model internal thinking before answering. For the most part, it's going to be invisible to you. But it's built at the output rates, meaning at the high rate of per token cost. And those can add between 4 to 20x costs per request. And finally, we have the output. That's the answer that you actually see. And this is typically 3 to 5x more expensive than the input price per token.

Starting point is 00:19:26 And I think the reasoning layer is the one. layer that catches everybody by surprise. Because you might have a 400 token answer, but under the hood, it carried like, I don't know, 4,000 thinking tokens underneath because the model was having an internal monologue and doing a lot of thinking in order to give you the answer. And if you want the analogy, it's like thinking about the part of the restaurant bill that is labeled the kitchen time. So you don't get to see it. It's not part of the dish, but you still have to pay a lot of it for that. And the models with the high reasoning effort are often the one with the 20x. amount of tokens being consumed versus the lower reasoning effort.

Starting point is 00:20:01 It can be the same question with a significantly different price tag. And sometimes spending a higher reasoning does not get you better results. So some metrics even showed it for a simple question, it's better to use lower reasoning because the overall cost per task will be significantly lower and the quality will be improved without having the model overthink everything. So it's not always that smarter or spending more time thinking gets you better results. One of the more interesting shifts in enterprise AI right now is how quickly the conversation is moving towards infrastructure and operations. As AI moves into core workflows, regulated data environments, and agentic systems,

Starting point is 00:20:41 enterprises need governed infrastructure and inference that can operate reliably day-to-day, with clear operational accountability built in from the start. As those systems scale, the operating model increasingly becomes part of the AI strategy itself. Rackspace technology is the operator of the full enterprise AI stack, from agents to infrastructure across private cloud, hybrid cloud, and edge environments. Rackspace builds and operates governed AI infrastructure inference and production AI systems for organizations where sovereignty, compliance, and uptime are non-negotiable. Therefore, deployed engineers stay embedded beyond deployment to help operationalize and run AI in live environments.

Starting point is 00:21:15 To learn more about where enterprise AI runs and outcome scale, go to rackspace.com. Every AI coding tool on the market does the same thing first. It starts writing code. Blitzy does the opposite. Before writing a single line, Blitzy spends days reverse engineering your entire codebase. Thousands of agents ingest millions of lines mapping every dependency, every undocumented constraint, every architectural decision made over the last decade. The result is a dynamic knowledge graph that understands your software the way a principal engineer would after 30 years in the building. Other tools guess at context with grep searches

Starting point is 00:21:46 and markdown files, Blitzy never guesses. It builds true understanding first, then delivers over 80% of entire software epics autonomously. Validated, end-to-end tested production-grade pull requests. That's why Fortune 500 engineering teams trust Blitzy with the codebases that matter most. See for yourself at blitzy.com. That's BLITZY.com. Here's a harsh truth. Your company is probably spending thousands or millions of dollars on AI tools that are being massively underutilized. Half of companies have AI tools, but only 12% use them for business value. Most employees are still using AI to summarize meeting notes. If you're the one responsible for AI adoption at your company, you need Section. Section is a platform.

Starting point is 00:22:26 that helps you manage AI transformation across your entire organization. It coaches employees on real use cases, tracks who's using AI for business impact, and shows you exactly where AI is and isn't creating value. The result, you go from rolling out tools to driving measurable AI value. Your employees move from meeting summaries to solving actual business problems,

Starting point is 00:22:45 and you can prove the ROI. Stop guessing if your AI investment is working. Check out section at sectionaI.com. That's SEC, T-I-O-N-AI-com. This episode of the AI Daily Brief is brought to you by HyperAgent, where you run fleets of agents your team can manage together. New users get $1,000 in inference. Forget local agents and chat workflows waiting on your laptop to be prompted. HyperAgent deploys always-on agents in the cloud, doing real work across the tools your team already uses.

Starting point is 00:23:14 Marketing's agent turns competitor moves into landing pages. Saleses agent enriches leads, drafts emails, and updates the CRM. Ops agent chases the paperwork and tracks the budget. Every agent has access to shared context and follows your rules about scope and approvals. It's time you add agents that feel like teammates. Hire yours at Hyperagent built by the team at Airtable. Claim your $1,000 in inference at hyperagent.com slash AI Daily Brief. All right. And I think that the gap between input and output pricing keeps widening at the frontier. If you look at the Fable 5, it's about $10 per million input tokens and $50 per

Starting point is 00:23:53 million with the GPT 5.6 solid's 6x ratio. So we're seeing the gap even widening. And those effort levels, that's also something that highly adds the complexity because these frontier models increasingly letting you dial the reasoning effort. With higher efforts, the reasoning tokens are significantly higher. And that's probably the dial that you should even be more mindful of, even beyond the models, because those can easily cost you 10 to 12x token increase between high or extra high effort to the lower medium. All right. One last thing here, a price tag that experiment that came from Databricks. because a smarter model might not always be more expensive than a less expensive model.

Starting point is 00:24:37 What Databricks did is they tested coding agents on real engineering tasks from its own codebase, and they were using Sonnet 5. It was 1.7 times cheaper per token than Opus 4.8. However, Sonnet cost around $2 per task or 2.09 per task versus 1.94 for Opus. So because Ronet needed more iterations and more reasoning had to spend way more tokens to get to the same results, overall Opus, which is significantly on paper, more expensive model, it was cheaper to operate, which means that we shouldn't just reach to the cheapest model possible. We need to reach to the right model for the task, and that's not easy to get, but something to be mindful. The other thing that matters to the bill is the tool itself.

Starting point is 00:25:25 So Databricks in their same experiment, ran the same model at the same thinking effort through different agent harnesses, and they saw that more than a 2x difference in cost per task with the same quality using different harnesses, just because primarily one tool was feeding the model roughly three times less context than the others, and thereby the overall cost was lower. So very difficult bill to read and very difficult build to navigate, and I'll try to help you as best I can. So bottom line, we're dealing with cost per task and not tokens because otherwise we will not be able to actually compare apples to apples. And the cost will include the rate rise, the review, the every like correction that needs

Starting point is 00:26:06 and every additional iteration. And then you need to divide by the number of accepted results. That's your cost per accepted task. That's the metric that you should aim for and optimize for. And this also brings the conversation much more into return on investment and business value, rather than just having a conversation around tokens that is very hard as hopefully by now you understand to meter. Good.

Starting point is 00:26:27 The practical task that you can do, you need to take between five to ten representative tasks of what you do, run them to a tool model or tool options in order to have a good understanding of, in your option space, what you should do, hold the input and the quality bar very constant and compare first past success, attempts, human correction and elapse time and the total cost.

Starting point is 00:26:49 The winner is the start. that gets your actual work done reliably. I know it sounds like a lot, but if you have a good taxonomy that was optimized for yourself and you know for the task that you do, which models overall get you better results potentially with fewer tokens, or if you can do that for your team or your company and you'll need to do that recurrently because things change quickly,

Starting point is 00:27:11 then at least you can teach folks that if you're doing that type of research, the recommended model to get you to the overall best quality and value per task, is the following and so on. That's the current reality that we live in. Okay, now I want to give you a language on how to look at your own tokens, and hopefully using that you will be able to distinguish between the tokens that add value to the ones that's not so much. Every token that you, all organization spends, in my opinion, is one of three kinds.

Starting point is 00:27:41 There are tokens that I call tokens that teach, and this is running in both directions, meaning that you're teaching yourself, as Nathaniel said before, we don't want to stop the experimentation. So the tokens that include the experimentation, the failed workflows, and let me try these three different ways so I will learn. And so these are tokens that are worth spending because you get learning out of them and you can look at them as tuition. And by the way, those also include what you are teaching AI about yourself.

Starting point is 00:28:11 So those will be the identity files and the curated context and the knowledge packs. The memory can be counted as those because teaching. your AI, who you are, what is your context, and learning what works for you in AI is very critical for you to continue moving forward. They look a little bit like a waste on a dashboard or a lot like a waste on a dashboard because no deliverable ship. But I claim that these are tokens that you need to defend fearlessly, because if you won't defend those, you will very quickly go back to just getting AI's help to draft emails and translate between languages rather than moving towards the workflows that matter, and especially if your company and yourself has a lot of

Starting point is 00:28:52 catch-up to do on where I is currently at. So I want the tokens that teach to be defended, because those are the things that will move the needle beyond the next category, which I call them the tokens that produce. So obviously, those are the most defensible ones, because those are the tokens that use to create work that chips. It can be the final proposal or the research or the code. obviously that's the thing that is much easier to show the ROI. But lastly, we also have the tokens that should be eliminated. And those are tokens that I call tokens that spin. Those can be machines talking to themselves or automations that nobody is looking at

Starting point is 00:29:30 their output or automations that are running too infrequently. Idle agents, bloated context, misused tools and context, using Fable to write an email. So using the wrong model, an optimized flows and so on. those will be activity without sufficient output. So the token smart move, if I need to summarize, is to kill the tokens that spin, to tune the production, to make sure that it is cost-effective

Starting point is 00:29:55 and protect the teaching. And that's the order. Like first, go and do the audit on your spin tokens and then do the rest. I have a very embarrassing tokens that spin story, which I will share in a minute. But I do want to also note with regards to tokens that teach that a failed experiment is as important

Starting point is 00:30:14 as a successful experiment. So you should definitely encourage your employees to fail to try because otherwise the tokens that produce will not yield as much value as possible. So it's embarrassing as an AI expert to talk about it, but my open claw was a chief of stuff, was because it's currently disabled, a chief of staff that I called Chloe, and it was using the entropic API.

Starting point is 00:30:37 And because it was using an API, it was like auto renewing all the time. And the bills were sent to a secondary inbox. I wasn't really monitoring them. And I was seeing that the charges seemed quite high, but because I was getting a ton of value and because I was not paying attention to how frequently I'm getting a new bill, I wasn't noticing. And then early June, I was traveling. So I was not using my open claw at all.

Starting point is 00:30:58 And still I exceeded the bill kept coming. So I was saying, like, why am I still getting some bills? So I opened the dashboard only to realize that I spent in two weeks $1,500 on an agent that I was not using. So I opened the dashboard, like double-click, and I realized, that I have almost 400 million tokens in and almost zero tokens out. So it was a ratio of almost 3,000 to one from input to output. And that's literally the definition of a machine talking to itself and billing me for like an internal monologue that it was running with itself.

Starting point is 00:31:29 Looking further, there was a bunch of crown jobs that the open clock created for itself. And it was like a compaction job that ran every 30 minutes on empty sessions. And even worse, like the trend was going up. So I, first of all, closed my open claw and only to optimize it differently. But if it happens to me, in this setup, it can happen to literally everybody. And especially when the credit card is owned by your company and not by yourself, often you will not pay attention because you're not sitting on the billing. Yeah.

Starting point is 00:31:59 And also the fact that you have a bunch of other things that are working well, that you might assume it's those things that are amounting for the cost. So one thing that I wanted to mention with Spin is that I think a lot of the framing of Spin, spin, if people were to pick this up, they might assume that it's only mistakes or errors that produce that spin. But that's not always going to be the case. Like, sure, this is sort of an in-between example where it wasn't exactly an error because it was doing something that it was meant to, but you weren't really paying attention, so it was doing more of it than it needed to. But I think a lot of times spin will also be just ill-defining the parameters for a job that you actually do

Starting point is 00:32:37 want. An example of this that I had is I had an open claw going for a while that was perpetually researching new data sources in AI that could help us figure out where the state of certain adoption metrics was. Every day, there's new studies that come out that measure this or measure that and that tell you about data readiness or systems integration or use cases or whatever. And it's too much to monitor for humans, but agents are really good at it. And so this open claw agent was a researcher that its only job was to on a set schedule based on its heartbeat, go out and check for new things. And it was never meant to stop. It was always, it was on a specific schedule, but it basically was this continuous research process that was crawling to the ends of

Starting point is 00:33:24 the internet every day. And it ended up just not being valuable enough for the cost, but it was doing what it was supposed to. And so I think part of the auditing spin is also just figuring out what things have accidentally become spin, even if they started in the right area. And I think that's why this idea of auditing, I think, is a good framework, because sometimes it's going to be about just updating or changing a process that was valuable as well as catching mistakes. I agree. And even more, I see many automations that people created because they think they will be useful. Like, oh, I can't read. I have too many Slack messages. Let me just create like a Slack miner that runs every hour and reads my entire set of channels. And that can

Starting point is 00:34:04 easily become $1,000 in tokens that literally do a job that moves the needle for nobody. Or the morning brief that you created wholeheartedly with the intention to read it every morning, but for some reason you don't find value and you don't read it. So these are the things that you should definitely audit and kill. And my rule of thumb is if you created an automation and for one or two weeks you have never ever used the output, you should definitely kill because it's a definition of a spin. Or if you are using this automation, but there is a very bad proportion between the value of summarizing all of your select channels to the bill at the end of the month, that's also

Starting point is 00:34:38 something that I consider to be a spin. And I think that now that everybody gets co-work or GPT work, that even more and more within companies, because it's so easy to build these automations and without sufficient literacy about how to effectively use the tokens, people create a ton of these automations that look good on paper, but don't look so great on the paper of the bill at the end of the month. All right. So let me give you a list for the suspects for silent token spenders. First of all, it's going to be your idle agents and the over-frequent jobs. These are going to be the things that run without any meaningful output or way, way too frequently.

Starting point is 00:35:13 We also, in many cases, see automations that nobody uses. So it can be like the weekly report or the dashboard that nobody ever goes to read. Additional thing can be what we refer to often as the pre-prompt tax. So anything, the model runs and reads before the very first prompt. So those include the always-on rules or instructions, the skill definition, the tool definition, on. And those can very easily, if not properly organized, amount to many thousands of tokens each run without you typing a single word. So those amount significantly. Many folks also hold the immortal conversation, meaning that they will continue an endless session that keeps carrying

Starting point is 00:35:52 old history and old context, often if even creating a poorer quality. We also, in many cases, see users never filtered data retrieval. So instead of just getting 20 rows from a database, will pull 500 rows or they will process the entire inbox to look for a specific mail that they know what was the subject line and so on. Many other folks will have their context all over the place. So the agent will have to read through a ton of documentation just to understand what are they talking about and what's the truth here, as well as rework loops. So any time that your agent or your skill or your just day-to-day usage gets you to do

Starting point is 00:36:30 more iterations just to get the same result, this is the, like just more tokens being spent on nothing. So these are the like immediate suspects. And I want to show you how you can try to potentially identify whether your system is in a spin situation or that the spin to production ratio of tokens is not well formulated. And the thing here is that not everybody can detect in the same way. Some folks have concrete meter. Those will be people who are using the API version of the models.

Starting point is 00:37:03 They have the API console that they can use. or if they are cloud code or cursor users, they have usage view. And of course, people with admin privileges, they have an admin dashboard. So if you are one of those, you can do the following things. One thing that you should definitely do is the weekend test, meaning that if you didn't do anything with AI, but you look at your bill and you see that your bill keeps compounding, you know that there are things that are adding to your value without,

Starting point is 00:37:29 to your bill without any value. That was what happening to me. also look for very extreme input to output ratio. So agentic work legitimately runs with high ratio, but if you get to a point where it's many thousands to one between input and output, in many cases, that's empty loops. In my open claw case, it was 2,600 to 1, which is ridiculous. And if you see that your spend keep rising while the work or the value that you do stays flat,

Starting point is 00:37:57 that's also potentially an indication that you are in a scenario of spin and you need to go and further understand what's the case. However, there are many folks that don't have direct meter because they're not using one of these tools or they don't have the admin privileges, which is probably most of the regular users. For them, you should probably use proxies. So just go directly to list all of your automation

Starting point is 00:38:19 and the scheduled jobs that you own and ask which one of them added business value last week. If you don't know, that's a suspect. Then also watch your quota. And if you're burning to your weekly, click quota extremely fast, especially if you compare to other people in your setup or in similar roles, that might be that you're doing something wrong there. And if you're in an enterprise plan, your admin do have the view, at least of how much you're consuming. And also the typically

Starting point is 00:38:45 the input output, you can just ask them. And there are many places that you can look. There are specific like slash context and slash usage in cloud code. There is also in application visualization, now both in cloud code and in cursor, that you can just click on the usage meter and try to to understand there. So regardless of what and how visible it is for you, you should definitely put some caps on how much you spend rather than letting the bill just extend all the time and put some alerts if there is some kind of a significant jump in how much you consume. That can be an indication that something is up in your system. So that's for identifying spin. And now the habits that we should all adopt to mind our tokens. These are several things that anybody can do immediately

Starting point is 00:39:30 that typically improves the token consumption without reducing the business value. New task is a new session. This one is an interesting one because we will talk in a minute about also model routers, but at least for now, for the most part, be intentional about which model you use for what task. Sometimes it's actually going up to like an opus or even Fabel class models because they will get the job done in one iteration and overall reduce the spend. In some other cases, it's not doing a web search with Fable,

Starting point is 00:40:00 rather going to the haiku or the lower cost of models. Right size your context. Tell your AI what it needs to know. This is a classical Goldilock, not too much, not too little, but sufficient such that it will not go into endless internal reasoning token loops just to try and understand what you're talking about. Builder usable capabilities often when we're just vibing with our model and trying to use it ad hoc rather than sitting down and creating the skills, creating the proper

Starting point is 00:40:29 automation, creating the proper agents, we're just wasting a ton of tokens to re-ask the tools to do something again and again. So seeing and building proper systems often is one of the best levers that you have to use their tokens wisely. And filter everything that you can. Tell it, in which rows of the table the data exist, in which parts of the project board the data accounts for, which select channels and so on. The more you point the model to the right place, the better the results that you will get. And lastly, in many cases, we start doing the work. we realize that the model is completely off. Maybe it's the wrong model.

Starting point is 00:41:02 Maybe it's missing something. Don't let it spin. Just kill the job early and start again while understanding what you do. And this is one of the cases where looking at the model reasoning will go a long way to understanding that it's completely off in the wrong direction. So I would recommend whenever you send the model to start doing something, especially if it's a significant portion of work, open the thinking to understand what the model is understanding from the task that you gave it. And if it seems to be off, stop and improve the instructions rather than letting it. So that's the habits for everyone. Two additional levers that you should consider, and some of them are very new.

Starting point is 00:41:35 So if you are a cloud code user, you can use the slash doctor command. This will basically check not only how much like a past installations take on your machine, but also how are your token divided, whether you have stale skills, stale tool configuration, whether your overall instructions are overly long or overlapping. So it's a very good command that will be created for us, that you can, go and execute if you are a cloud user. If you're not a cloud user, you can just have your AI tool investigate what the slash doctor command does and basically recreate it for your own tool, because it's not like a very complex thing to do. It just audits all of your system for

Starting point is 00:42:16 you and gives you a structured report with concrete recommendations of things that you can kill because you haven't run them for a while or things that are duplicated or stale or contradictory that you can potentially reduce significantly. And with regards to routing, a lot of the industry conversations sit right now around the model routing and you were just talking, I think, today or the other day around some interesting M&A around model routing. Picking the right model is still one of the highest return things that you can do, even if you are able to use like the Kerser automated router or some of the other solutions that are coming our way because it's not always going to be, even if you have like a router in the background,

Starting point is 00:42:58 it's not always going to be as precise as you knowing which model to use. And in many cases, you still don't have in your existing tool a good enough or even an existing router. Yeah. I think that we are very early in figuring out the right patterns around routing. Obviously, there are a million solutions coming to market. They're all taking slightly different approaches. You have independent experiments from enterprises who are building their systems that route between, you know, custom models that they've trained as well as, you know, the Premier model. Like, it is, there's no one clear approach yet. And even when there do start to be clear use cases and patterns, they may not fit everyone in every use case. I think it would be

Starting point is 00:43:41 entirely unsurprising to me, or I expect that routing norms around certain types of software engineering get solved first because it's more deterministic and clear. And you can kind of actually have, you know, more sort of verified success or not, I think when it comes to knowledge work tasks more broadly, it's going to be immensely more complicated, especially considering how much of our personal model routing that we do right now is about not what the benchmarks would say on a test, but how we like the particular nature of one type of response versus another for a particular context. So I continue to believe that understanding different model capabilities and having model preferences is still a very high leverage activity and is going to be for quite some time. I agree. And I think the ultimate test was

Starting point is 00:44:27 when GPT5 was automatically routing us and all super users or just like more than occasional users, we were all very frustrated by what we got from the auto mode. I think that's the original test that we want to control and we will probably even with a great router for many things will continue to be opinionated and rightfully so. Just for people who are also building their own, obviously they have additional levers, like you can create more caching and so on, but still for them, it's much the same physics. Like, the more control you have, the more you are able to be smart about the way you use the models. That's the additional levers. We talked about tokens that teach, and I think that up until now, we were very much focused on things that we can do to reduce the bill.

Starting point is 00:45:14 But here I want to fight, a good fight and say that we want to protect those tokens, because those are, in many cases, the tokens that you spend in order to get much better return. And it's not just about optimizing the bill to go downwards, but rather to improve also the return that we're getting. And often to improve the returns that teach. And we're talking about two ways, whether it's you teaching yourself, meaning that you run, the same task using three different models in order to get to this taste of which models you like for each task or you try the same task in three different ways until you learn which one works

Starting point is 00:45:49 best or you experiment with a new tool or you try a new skill or a new automation and it doesn't work and you try something else. So all of these typically gets you overall to much better results from AI. So those should be protected firstly. And also the other side of you teaching AI who you are, building the systems, adding more contexts such that you will get much more personalized results or much more organizational aware of results, those are almost always with direct correlation to how much value you get from AI. And so does data. I've seen a study of 20K developers that found that the heaviest AI users were roughly twice as productive in terms of the amount of production code that was shift. So in many cases, it's actually becoming much more like a smart

Starting point is 00:46:36 exploratory user will get you to better results. So we know. So, So to summarize, what you need to do in two sides. So for the individual users, these are the things that you should definitely do. Go and see whether you have tokens that spin. I'm sure that all of us have those idle automations or maybe some of us have like an even worse scenario of the amount of tokens being spinned without any business value. Practice those six habits. You can even put them on a post-it and just get yourself to work more effectively with the tokens that you have. Do spend the time to invest in reusable capabilities and improved context, that the model can be much more selective and discover the relevant context where it matters. I also want you to audit the things on a schedule,

Starting point is 00:47:22 meaning regularly go back to the system and see what is now stale, or maybe something that was working well has become a stale automation. Maybe you need to improve the context, the instructions. Maybe you can remove some of the instructions per new advice coming from entropic that the modern models need fewer instructions, not more. And make sure that you protect the learning budget. And as needed, go and negotiate that with the people responsible for the budget to make sure that you're not now being reduced to the amount of tokens that leaves you with very little room for exploration.

Starting point is 00:47:55 For the organizational side, make the usage visible and then teach the people. Because when managers and employees see their own, they are much smarter about how they use, but make sure that they are not being encouraged to spend as little as possible. possible, but to spend smartly and also make sure that the budget is by workload and by individuals. If someone is building skills and context and usable capabilities for their entire team, they need to get significantly higher budget than the person that just uses the tool as an extended Google. And all the time, we need to make sure that it's by, that you tear it up in some organization and some individuals get significantly higher, while others potentially less and not just one size

Starting point is 00:48:35 fits all for the entire organization and make sure that everybody listens to something like that or that you do an internal training that teaches people on how to be smart about tokens, but not how to spend as little as possible, but also how to be mindful about the ROI and aiming to use tokens for the things that move the needle for the company. So that's the concrete actions for you and the team. And if you want to be even more token smart, so beyond the audit of your own usage, we created for you a token gym that you can go and learn and flex your token smart muscles. And if you want to go even further and to learn how to build and work with AI and agents properly,

Starting point is 00:49:16 we do have our existing trainings and the next cohort start on early September. So we'd love to have you there in the executive catch up or the executive agent leadership that will bring you all the way to be very smart about AI or very smart about agents depending where you are. That's it. Awesome. I think that this, we're always at the beginning when we're talking about things on this show, but this one is, I think, particularly inflection pointy, let's say, to use a word that doesn't exist.

Starting point is 00:49:46 We are so clearly just at the beginning of figuring out how to organize the relationship between people and the compute and intelligence that they're going to consume. And it is going to be iterative and messy, which is why I think so many of the, ideas that you presented are shared as frameworks, you know, patterns to explore, right? It's a set of steps that you can take to try to get a handle in these problems, but every organization at the beginning is going to solve them or not in different ways. So thank you for sharing some starting points and, you know, we'll continue to evolve this conversation as the tools around us change too.

The AI Daily Brief: Artificial Intelligence News and Analysis - Everything You Need to Know About AI Tokens

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.