Technology, Connected - AI Inference, Wasted GPUs And The Cost Of Every Token

Episode Date: July 14, 2026

AI companies are splashing billions on more GPUs before they use the ones they already have. Many sit idle, others use a fraction of their potential. For the hyperscalers, cloud providers and AI companies, buying more GPUs does not solve much if the GPUs already in the rack spend too much time waiting for the rest of the system.The next battle in the AI infrastructure wars is efficiency: GPU utilization, NICs, memory movement, networking, CPUs and software that keeps expensive hardware doing useful work.Moshe Tanach runs NeuReality AI, a company building infrastructure for AI inference. In this conversation, he explains why the next wave of AI infrastructure depends on GPU utilization, NICs, memory movement, networking and software efficiency, not just buying more chips.This is a technical conversation about AI inference, infrastructure efficiency and the hardware layer beneath ChatGPT, Claude, Gemini and the AI tools now being built into business software.Please enjoy the show.Thinking on Paper is a technology podcast about AI, Space, quantum computing, science, and the systems shaping your life. 🏠 ⁠Buy us a beer on Substack⁠🫵 C⁠hoose your own technology adventure ⁠📺  ⁠Watch our beautiful faces on YouTube ⁠🎧 R⁠emember Steve Jobs on APPLE⁠📺 ⁠Get clips and exclusive videos on Instagram ⁠--Chapters(00:00) The GPU challenge(01:27) Training Vs Inference(05:45) Memory & Keeping The GPU Busy(07:31) Deep Seek, Blackwell & Ruben(09:03) How Ripe Is CPU For Reinvention?(11:03) AI Agents Run On CPUs(12:51) Do We Need All The Computation?(14:35) GPU Utilization Rates(16:50) Tokens In A Context Window(20:59) Data, Knowledge & Wisdom(23:56) Why Are Your GPUs Busy? (A Truck Analogy)(29:07) Hyperscalers(31:59) The Decode Phase(36:47) What More Efficient GPUs Means For The User

But still, most companies are focusing on the top line. What to do with AI? How to increase the top line. Not enough are talking about efficiency and how to improve. So why do I run after more GPUs if I am a 25% utilization? Now, hire a software company that didn't or change your network and boost it up to 50% or 75%. This is double or triple the supply you currently have on the same hard on.

the same hard work. Is AI efficient? Energy isn't infinite. GPUs aren't infinite. Grid capacity isn't infinite. And yet in fear it's demand, the daily use of AI is increasing. More users are coming online every day. More businesses are integrating it into their workflow every day. Demand is going only one way and we can't just keep throwing more compute and more GPUs at the problem. So we're talking about efficiency. Can we find room to make what we have the infrastructure, the GPUs, the CPUs, the data centers we have, more efficient our guest today. Tell us that is Moshe Tanak, the CEO of New Reality AI.

Moshe, welcome to the show. Thank you for thinking on paper with us today. Thank you, guys. Thanks for having me. I'm a pleasure to be here. Talk to me about how people should think about the difference between training a model and the difference between inference and how that has. how one is easier to solve than the other and why?

What is a model? A model is a very big mathematical equation. It has different architectures. Gen AI, LLM, CNN, RN. It's just a very big mathematic problem. And when you train it, you want to fine-tune the coefficients that are multiplying every step in this equation.

You want to fine-tune it to a specific purpose. So you're taking a big data set with tagging. It's a very long process. And you can leverage as many machines as you can get just to shorten the time of the training. You just need to split the long work of going through all the data, fine-tuning all these coefficients

until you have good accuracy in your model. This costs a lot. Usually it is paid by R&D spending or TAPEX spending because once it's free, frozen. Once I have a model, I can start using it. Using it is inference. So you're loading it, sometimes on one machine, sometimes you split it on three machines with 24 GPUs just to shorten the time it takes to compute it. Because we have some interactive. Everybody talks about

the gentic and agents. Agents use models multiple times to do a job. Unlike the traditional use that you send a query or you send an image and you get an inference response, agent will keep asking more questions, going from one step to the other, learn from the question and fine-tune what he asked. So you want the agent to work fast, but it's going to touch an LLM 100 times. So you have to shorten the time of the inference computation, and for that, you spread it on multiple servers. So now you're not just relying on a GPU, you're relying on on a set of GPUs that communicate between each other because they're sharing data.

All these 24 GPUs doesn't even reside on the same server. They're going through the network, either scale up network or scale out network between Rex to compute that. So in training, very long job, huge data, R&D spending, inference a use of a frozen model coming out of your cogs, out of your,

cost of sell. So if for every product you sell, for every usage, someone, let's say you're SAP and you're giving a CRM solution. And now behind the scene for every request, I'm adding an inference or 10 time inference or 100 times. So you have the training. There's a bunch of data. There's some compute that's crunching that data and working. Not much going on on the networking side unless it's like one or two machines kind of doing the job. But if we translate that to inference, the requests are coming in and they're trying to be redirected and their, so their networking becomes a little bit more challenging. The communication between devices, response times, latency, all of that stuff. Am I thinking about that, correct?

Training actually has more overhead on the network. More. You bring a lot of data from storage and you share much more data between all these nodes that are fine-tuning different coefficients. sometimes in the model you have an all-reduced function. You need all the results to every one of the nodes. At this point, the network is overloaded. But inference is also impacted by the network, especially when you want more interactivity, you want a faster response.

So the network challenges in training, just the large amount of stuff that has to go through the pipe. And then the challenges on the inference side is the amount of requests for stuff going through the pipe, kind of. Yeah, and the time sensitivity to finish it fast. And you're working on a solution for inference rather than training. New reality is that inference from day one, yeah. Efficiency is important in training as well.

Where are the big inefficiencies in training? I think you look into the job that each GPU does, and you want to keep the GPU busy at all time. As long as it does just computation, then it's either how much multiply accumulate arrays it has and how much memory bandwidth it has. Sometimes the memory prevent the GPU

from consuming all its multiply accumulates at every cycle, so you're losing utilization. That's the first thing that everybody is focusing on And that's why you see Nvidia buying GROC. Why do they buy GROC? In high level, every LLM has the pre-fill step and the decode step. The pre-fill is where I take the query. I do all the calculation of the model and I load the KV-Cash to a state that's represent that request.

And then I'm starting to generate tokens. Generating more and more tokens is the decode phase. The decode is very, very memory bound. You need a lot of memory bandwidth. And GPUs has the HBM. It's limited in capacity, but it's the highest performance you can get. Now you see different startups like ROC

that relied mainly on S-ROM, on internal memory in the chip. It's very small, but much faster than HBM. Cerebras is being the same. And now we have new companies, startups like D-Matrix and Majestic, that are trying to remove the memory barrier. So that's item number one to focus on,

and the industry has been focusing on that for the last 10 years. The secondary one is network. If you look on Dipsic article when they released it, they balanced the compute and the network to a perfection. GPU never wait. It always has the next data coming in, so it continued the next stage of computation. But this was only possible because they used an inferior

GPU because Nvidia couldn't sell the H-200 to China. So they used an inferior one. If they would use Blackwell or Ruben, they would be at 25, maybe even less percent utilization. So they will lose 75 percent of the GPU they bought just because they're waiting on network. That's the second inefficiency. And that's why you're seeing Ruben, the new wreck from Nvidia, with two nix per

GPU when it used to be just one. Nick is the component that connects the server to the network. So you have GPU, you have a CPU that manage it, and you have network interface controllers, NICs, that connected to the network. And now we're seeing Nvidia putting two

on every GPU, and AMD even went and put three, and Google is doing optical network between the TPU. Network is a critical component in disaggregation, in distributed compute. If you just run on a single processor, you don't care.

But if you spread the work between 24 or 240 GPUs, network is everything. That's how remote the data is from one to the other. So you're mentioning GPUs. You're mentioning NICS. You mentioned CPUs as well as a function in this whole thing. How ripe is that arena for reinvention related to, you know, increasing efficiency in the whole stack?

This is a good question. As an industry, we've been trying to reinvent the CPU for the last 25 years. But still, the single thread, multi-threaded version of X-806 or Arm or Risk V now. Seems to be the best because you can do Python, you can do C++, Java. And the pace of innovation, especially driven by AI writing code, is just accelerating. and it has to run on a CPU that understand that was built for Python. So we're hearing all this.

So our first generation was AI CPU. But it wasn't reinventing the CPU itself. It was reinventing the heterogeneous compute nature of that device. We put four types of processors on one die. We had the arm cores, we had DSP engines that has wide vector of multiply accumulate and very programmable.

We had video decoders and audio codecs. And the idea was every data that coming from the network, from clients or storage, into the GPU for neural net processing, need to be manipulated and prepared for the GPU. And the CPU doesn't keep up. But then the thesis got changed a bit. With LLMs, the amount of models, each GPU process, is not that big.

A model is now spread on many GPUs. So the CPU is not that busy. I think CPU is quite well, but those who thought that we're going to sell less CPUs finding themselves wrong right now. We need more CPUs, especially with agents. Agents are not neural network or AI form of solutions.

They are just software that is decision-based. Some is using AI, small model for AI, but it's mainly a controller above the model, above the AI, the frontier model. And those agents are in Python or other language, and they need to run on CPUs. So now, Nvidia and Arm are showing racks full of just CPUs, no GPUs for agents.

and we also have the CPUs that are attached to the GPU. So I don't know how much it's ripe for a revolution, but it's definitely needed. The more GPUs will deploy, the more CPUs will deploy close to them and even more networking switch and nick components. The agent revolution is only just beginning as well from what you've just said, as the use of agents becomes easier, as more and more people use them,

the CPU demand is going to increase incredibly. Definitely. I think we're just at the beginning of it, and it's hard to predict where it's going, because if you look on the $700 billion that all the hypersquiters are investing in AI this year, it's a ridiculous number, and it's there only because TSM, Samsung, and Intel

can't manufacture more. If they could manufacture more, maybe pricing would go a bit down, and more companies would buy more GPUs. Where is it going? Do we really need all that computation in the world? Why? Is it to go to space? Is it to go to...

It's really mind-boggling to see what will happen in the next 10 years. Are we going to continue this race that sometimes, It doesn't make sense. You know, I'm driving the race, okay? So, and I'm joining the race, but. You're in it. But come in it. But sometimes it doesn't make sense, the amount of hardware building.

And at the end of the day, you started saying that more are integrating AI and all. But still, most companies are focusing on the top line. What to do with AI, how to increase the top line. Not enough are talking about efficiency and how to improve. So why do I wrong. after more GPUs if I am at 25% utilization. You know, hire a software company that they improve or change your network and boost it up to 50% or 75%.

This is double or triple the supply you currently have on the same hardware. But the rate is on and people are afraid that next year they will not have GPU so they pre-order the next orders. Yeah, this is kind of where I want to go. This is a really fascinating topic because when you're interrelated, a race is defined by a destination and a line of competitors and you're trying to get to the destination before the competitors or at least before half of them or at least before a third of them, right? So there's this intensity.

There's just like I have to have to have to. Sometimes it's zero sum game and you become hyper focused on the race. How valid or how well documented are the utilization rates of GPS, are the utilization rates of GP? use because I hear a lot about it that a lot of them are sitting idle, especially in the inference phase. How well documented is that? Not well documented because it doesn't serve the people. You know, it's all about interest, right? It's all about the people.

So think about the enterprise that, you know, a big bank in the U.S. chasing technology. You're in the infrastructure group, software, hardware, you're building data centers. You've been focusing on transformers, Bert, Roberta, three years ago, four years ago. You had 150 small models deployed, trying to experiment, which will serve. Suddenly, LLM comes. The executives are asking, what is this? Open AI, JetGPT.

So you're stopping everything and you're starting to learn what is LLM. How do I deploy LLM? You dropped all your investment from before. This is reality. This is what we experience with our first product, working with these banks, bringing them this inference-serving platform that is very, very efficient. but they dropped it and just moved to the next buzz, to the next challenge.

So to me, actually, when I look at the race, I think it's mostly a race between the hyperscalers and the countries, the regions, that are afraid and must have their own solution, whether it's South Korea, Middle East, China and U.S., of course, Europe. No one wants to stay behind. Everybody has to order GPUs or they don't have it. I want to see the race between the small companies.

That's when I'll know that AI is mature. Those small companies that are adopting, that are fine-tuning an application that is enhancing our security on the road or health or other things. Right now, prices are very high because hyperscadlers are pre-ordering and paying the neocloud to build for them.

You know, a lot of the neoclod or building data centers for the hyperscale. Right. We've spoken about the training that speak about inference. And just for me, just so I can visualize this, I use, Claude, I have my single context window for thinking on paper. I have three, well, I've got a couple of contexts. Unless I've got one, I've got 300 episodes in that context window.

I'm using it. I'm asking questions. I'm finding links between different episodes of thinking on paper. You mentioned earlier that the, the, tokens are being divided between GPUs in one-cruces. Even for tokens within one single context window, could you just help me as a layman understand how that works and then how we make these inference systems more efficient?

Think of it that your data is one thing in the solution you're getting. The wisdom is inside the model that cloud built. And that wisdom is one trillion parameters, separated into mixture of experts, so groups within it. And every request of yours will extract some of the data you had based on wisdom.

So the wisdom will say, oh, this is the prompt from Arc and I need to touch that data, I need to collect that data, then go back into the wisdom with this extracted data. And so there's a couple of iterations that will go and query

that big model to give it some of the wisdom of what's inside. First of all, what did you ask? Then what's the data that is relevant? Then with all this accumulated, aggregated prompt, data plus what you request, let's ask another question to the LLB based on. So there's a couple of computation that are running, and in order to run the one trillion parameter or two trillion parameter model,

we're using multiple GPUs because if we would use one we would do part of the model then flash the memory, bring the next piece of the model compute. It would take a long time and very inefficient because the

GPU will wait to flush and reload. Flash and reload. Instead, I'm spreading the model between many, many GPUs. And every query of yours will just hit all of them. They will work together. each one will do a different part of the model.

And then the summary will come back to you. So what have we touched? We touch storage with your data. Everything ran on network, very fast network. Many GPUs shared intermediate results between themselves through the network. CPUs were running the Pytorch or the Tritone framework that manage all this,

all these, orchestra works for every request that you requested. And you know what's funny? Right now, they're subsidizing it for you. Thank you. Thank you. How long would it take?

Will it last? You loaded all your episodes. Someone loaded 800 videos. Maybe someone just a little bit text. Why should all of us pay the same? You're consuming more storage and currently they're probably not charging it from you. The same with a CRM company or Salesforce or you're buying AI enhanced features,

but the token consumption is growing behind the scenes and it's creating a problem, an economic problem. And at the end of the day, the customer will pay for all these problems. That's really interesting to think about because right now it's just kind of like, hey, let's get everyone using this stuff, let's open the gates, jump in, do your thing, but eventually we're going to get used to these very awesome tools. And then they're going to be like, here is the new customer agreement. And we're going to be like, whoa.

Yeah. I have them. So what you said earlier, I want to kind of go back. You used wisdom. And I thought that was an interesting, interesting placement of that. Because I think about this a lot. What would you say the difference between data, information? or let's say what's the difference

how do you relate these three? Data, knowledge, and wisdom. How do you compare and contrast those? Oh, man. Well, first of all, data has everything inside, but it doesn't let you see it. If you want all the connections between all the data, you need to read it all and find the connections.

I think the wisdom that LLM gives you after it consumes all this with some guidance, is all these connections. Suddenly he talks about, you know, he took some psychology books of how people act and some real proof of interaction on social media

between rivals and applied it on a question you made on Formula One race between Verstaff and someone. So there's a lot of connection inside data that teaches us a lot of things. and those huge equations carry that wisdom of connections. So would you say knowledge is more on the wisdom side?

Is knowledge a stage two wisdom? Is it like a step to get to wisdom? Or is it kind of the same thing? I think knowledge and intuition are the wisdom. And I think everybody says that we're still a long way from having, from the machine doing intuition the way, the way we do. What about experience?

Experience. Yeah, but what do you get from experience? Some conclusions on situations and on data and on whatever, right? Those conclusions become our intuition. I'm managing for the last 20 years. I learned a lot about working with people. And I have intuition. And if you read thinking fast and slow,

you know that intuition is right 50% of the time. But I still rely on my intuition, just as you do. Which that answers the question of can AI is wise if you don't need the experience to get to the wisdom. We're on the precipice of diving into a consciousness discussion, which I want to pause. I love that. But I want to point back we're in efficiency. And in order to care about efficiency, I think you have to have investment in something that is put to use. So I was thinking about this as an analogy.

All right. So let's compare, let's compare GPUs to trucks in a logistic company, right? So I own a logistic company. I have trucks and I invest in these trucks, millions of dollars in trucks. And if they're on the road delivering packages, I'm getting a return on that investment. But if they're not on the road delivering packages, I'm not getting my return on investment. How could we take that analogy and look at GPS? from the basis of companies actually using AI, because there's a disconnect. There's an additional layer. Companies don't really deploy their own GPUs-ish yet. They rely on another layer. So I want to cast that. I don't know that there's a question in there yet,

but a thinking space. Let's explore that together. Is that analogy makes sense to you? Definitely. Very, very good analysis. What would determine how busy your trucks are. First of all, do you aggregate

carriage from multiple places to multiple places? And are they local? You know, where you pick up and where you drop. You want the truck to be full. That's one. Then timing. Is your truck getting to

to the destination, when the destination is open. Is it taking the right routes? Is it fueled on time? How much do you lose on sleep time for the driver? Do you have two drivers? Does it make sense? What cost more? The driver or the truck?

When I look on all this, and the roads, of course, the roads that it takes, how much time on the highway, how much time off-road. I think when you look on GPUs, you should think about first of all how busy they are is the CPU above them giving them a big enough problem

if you gave them a small problem and you're not even using all the different multiply accumulate arrays then you're wasting GPU if you gave it a task that is too memory bound and it waits on memory so it used all the multiply accumulate in some cycles

but on other cycles it waits on data if you didn't route a good network between them and they're waiting on data they will they will wait it's just like you send some data on the road but the road is off road instead of highway and then you have the software above it that keeps managing it and communicating with when we say that the GPU

will share data with other GPU, he doesn't even know that he shares data with other GPU. He knows how to calculate and when to take the data of the result and put it in a queue. And that specific queue is not a queue going to the CPU, it's a queue going

to the nick. Who tells the nick what to do with that queue? It's the CPU above it. It's software. So the software is orchestrating everything. and telling, okay, now you do compute. When you end, I need the nick to know to wait on data. The minute the data comes to the queue, take it out, send it. And then we have another orchestration layer that makes all the GPUs work together.

And when you swap between users, you swap security keys. So you must work on the software and optimize it. You must work on the network between the GPUs to optimize it. and you must have a good multi-threaded CPU above it to run all these processes. Just so the GPU is not waiting, GPU is $40,000 today. CPU is only around 5 to 10. Nix are 3,000. The most expensive piece is the GPU.

We want to keep it busy. So it's a very good analogy. I use a pizza place, you know, with multiple ovens and, but they're under, under stuff because you don't have workers, you don't have orders, you don't. But that's a good one, the truck. So there's two different schools of thought, I think. I mean, if we simplify it a lot, there's the hardware side and then there's the software side as we approach efficiency. My understanding is you're kind of focused on the hardware side.

There are other people focused on the software side. Let's talk about what you're doing first at New Reality and why you believe in it. And then also what you think about the software side of the fence. So actually, in your reality, focus on the networking side. it includes hardware and software. But the most important thing is the NIC hardware. And when you look today, we're building wrecks. Nvidia, AMD, Google TPU, others are building racks full with GPUs or XPUs,

a replacement to a GPU that has a different architecture. And you need to connect all of them. Inside the rack, there's the school of Nvidia and the alike. that does scale up network. If you heard about NVLink, NVLink, is a very high bandwidth, low latency, dedicated switch component that connects all these GPUs within the rec.

And then you need to connect multiple wrecks. Now, some people think, oh, I can fit a very big model into one rack. But when you talk to the hypers, you understand that it's naive. You can fit a huge model into one rack. But after a while, GPU fails. What do you do?

You reboot that. And at that time, they don't participate in the computation. So you commute that microservice to a different wreck until it wakes back up. At this point, you're relying on the scale-out network, the connectivity between the recs and not on the scale-up network inside the wreck. So I always tell everyone, scale-up is not more important to scale-out, and scale-out is not more important to scale up. Just like asking what's more important,

the connectivity out of the chip or the connectivity inside the chip. All the connectivity is important, especially when you go to Frontier Models and huge AI pipelines and a genetic world. So we focus on the scale-out nick. We have technology for scale-up as well that can be integrated into XPUs,

but we're building a nick, and what's unique about it is first of all, it was purpose-built for AI. So it doesn't carry all the, all the fat of a general purpose, Nick, that also access storage and virtualization or all kind of network functions. We just focus on connecting GPUs between themselves to an internet network. The second thing that is important is that the output, the outcome of focusing on one

problem, that is a huge problem. In two years, we're going to have much more scale-out ports, nicks, than, frontend mix connecting to storage and in CPUs. Latency, we're less than half of the best nick in the world, and latency is important. You're sharing small chunks of data. You're always paying the time it takes you to transmit the data plus the latency between source and destination. So if I can cut down latency in half, it's easier for me to keep the GPU busy.

the last thing is power power we're roughly half of the best in class again because of the purpose built because we rely on in in in on chip memory and not the host memory and then besides just having the better better quantities or or or kPI's for the nick comes heterogeneous compute. How much do you know about heterogeneous computer? Are you talking about between classical and quantum, or is that something different? No, between types of GPUs. Like in about GROC. So the LPX from GROC will focus on the decode phase while the GPU or maybe part of the decode phase, while the GPU will focus on the pre-fill and other parts of the decode.

Or you have DPU is one technology. and Vida GPU are different, AMD different and now the hypers are finding themselves in a situation that again, scarcity, not enough, how can we optimize

and lower the cost of tokens? Run, the solution is to run the piece that need more memory on a type of GPU that has more memory and the piece

that need more compute on the type. So now pieces of the model run on D. different technologies, and guess what? Each technology build their own network. Nvidia use Infinity Band or Rocky.

The Maya in Microsoft use a different protocol. TPU use Falcon or a different proprietary. How do you connect between them? That's another special thing we have in our Nick, a programmable transport engine that can be reprogrammed to run any protocol besides the standard one that will try Internet and the Rocky. So now our nick can connect between those clusters in a very efficient way, low latency, low power. And for the builds coming in 2027-8 line, everybody will do at Regenus compute,

and you have to make sure that those protocols between those clusters are converted in an efficient way. And today, there's no solution. You have to go to a network appliance, that the CPU will convert the protocol, or go through the CPU on the GPU side, it's very complex and slow. This is another word that crops up on thinking on paper all the time. You're speaking about interoperability. If you go back to Jeremy's truck analogy, so Jeremy lives in America where the roads are very straight and very flat and very good.

I live in the UK a lot of the time where the roads are basically a traffic jam full of potholes. So if we're going to take that analogy of like the world has different road infrastructures and your trucks are rolling on different roads, how do we make it so my trucks can be as efficient on those UK roads as they are on a French road or a US road?

Can we expand this analogy here? Of course. The world is investing in ultra-eternet right now. Although Rocky, Rocky is a transport vehicle running on Ethernet that Nvidia drove and it's very efficient, but it's limited to point to point.

So if I want to communicate from one GPU to a lot of GPUs, I have to send it to all of them. Ultra Ethernet came to solve it because in AI, in this all-connected or all-reduced functions, you need to communicate between all-to-all.

You can save and send data to all, and you can also send through multipath in the network. because this path is congested. I'll use a different path. So one thing is to make sure that all the roads in the world are the same. But this is long and tedious work, and the geography doesn't fit everywhere.

So the other way is to convert your truck when you move from one place to the other. So your Volvo truck, when it is shipped to the U.S., or when it cross a bridge to a different country with a different, it just convert itself, to a different connectivity with the road. That's what we're trying to do. At the end of the day, we're sending a buffer, a message from one chip to the other.

TCP protocol is doing reliability in a certain way. Rocky do it in a different way. Can we convert between them somehow in a very efficient way, leaving the control plane that control all this in CPU, but offload the data pass to hardware? The answer is yes. but there was a misconception that no one need to invest in that because I'll build my own XPU

and I'll run all the model in a world, in a perfect world it would be great, but where you're scarce with GPUs and you want to improve your efficiency, you want to do better so you split the work between different clusters, different countries, and you need to convert your truck when they communicate with each other.

What people really want to know is how does this affect me? Does this make my use of AI cheaper? Does this make Claude better? Does this change how I interact with it? What happens? What changes for me? First of all, cost go down and availability goes up. I think the ultimate goal for everyone should be that AI will be,

will have the narrative of zero marginal growth and not the way it is today. You know how SaaS companies are relying that I'm building an infrastructure. It costs me a lot to get to a point of serving the first million users, but then the cost of the second million users and the third is negligible. In AI, it's not the case. The more users, the more tokens, the more spending, and you're stuck with gross margin that will never evolve

until cost of AI goes down or you're starting to do smarter thing. than using cloud. Maybe use an open model, Lama 70B, fine tune it with your data. It takes more work, but at least once it's there, you stop paying the premium.

And just to give you an audience, an understanding, if hardware costs $1, buying API of an open model on that hardware, open model like Dipsic, GPT, OSS, Lama, Kuen, For every dollar in hardware, you pay $2, maybe $2.5 on open model.

And you pay $15 to $20 on a frontier model from opening higher cloud. So there's a lot of way to save and to get to this point. And our job as an industry, as humanity, is to get there. But currently, I think it conflicts, it contradicts with the race. everybody are afraid of what will happen if I don't have enough GPUs. And then we have GPU sitting idle. We didn't even talk about GPU sitting idle because governments bought them and we don't have the skill set to use them.

They don't know what to do. How many of those are there? How many is seeing idle because politicians don't know what, governments don't know what to do with them? How many? Yeah. It's hard to tell, but I think the Middle East built you data centers. I don't think they're in the utilization that you see in the US or China.

far from me. Do you think, based on what you just said, related to costs, differences between standing up your own hardware and using open source models, do you see a trend towards companies building that kind of infrastructure internally versus relying on APIs to the big boys?

Yeah, we're seeing some of that trend. Companies that are more mature with what they do with AI, start to do these calculations. Neocloud provided, another path to hardware that is much cheaper than hypers and take the burden off you

in building, finding real estate, finding energy, finding, you know, it's all about, we talked about GPU, we didn't talk about energy scarcity. This is scarce as well. So you see all these blockchain companies that bought energy for blockchain converted to be AI companies

because they're sitting on energy and AI gets you more revenue than blockchain. I think I think we'll see the pendulum coming back to small data centers, own data centers, maybe

Neo-Cloud-based. We're going to see more people going from just buying API to buying hardware, infrastructure as a service, IS, and buying software like we have to generate

their own tokens, and we're going to see a trend moving from closed models to open models, maybe not on the leading applications like the coding, but on other applications that companies can use their data and lower the cost of AI

dramatically. And open models are closing the gap on accuracy with closed models. So I'm looking forward for it to happen because then I see, I think it will become ubiquitous. Everybody would use AI and innovate with it. Open model is the tortoise

in the hair tortoise race slowly catching up. It's going to be interesting. Moshe, this has been a fantastic, fantastic conversation. I think there are probably four or five threads that we didn't explore that would have been fun. Maybe we set up a V2 of this. But I really enjoy the conversation. I love what you're doing. You broke it down very easily to digest. So thank you for that.