Semiconductor Insiders - Podcast EP363: The Impact of an AI Factory on Production Deployment with NeuReality’s Moshe Tanach

Episode Date: August 28, 2026

Daniel is joined by Moshe Tanach, co-founder and CEO of NeuReality. Prior to founding the company, he held senior engineering leadership roles at Marvell and Intel, where he led complex wireless and networking products from architecture through mass production. He also served as AVP of R&D at DesignArt Networks (later acquired… Read More

Hello, my name is Daniel Nenny, founder of Semaiwiki, the Open Forum for Semiconductor professionals. Welcome to the Semiconductor Insiders podcast series. The guest today is Moshe Tanak, co-founder and CEO of New Reality. Prior to founding the company, he held senior engineering leadership roles at Marvell and Intel, where he led complex wireless and networking products from architecture through mass production. He also served as AVP of R&D at Design Art Networks, which, was later acquired by Qualcomm, he led development of 4G base station technologies.

Welcome to the podcast, Moshe. Thank you for having it, Daniel. So first, let me ask, what brought you to co-found new reality? I think short, it's the change of a new technology that is infecting. Back in 2018, I was still in Marvell, heading the product definition and architecture for networking solutions. and we started learning AI and using AI to enhance our products. And it was obvious that it's going to shift the focus from the old system architecture to a new one.

As more and more compute moved to the GPU, CPU-centric architecture was about to become obsolete. Not CPU-obsolite, but the CPU-centric, everything is around the CPU. And, you know, fast forward eight years, is it? You obviously see the amount of silicon spent in GPU. So it changed everything and it was obvious that it's going to change. We're now coming from a strong system and semiconductor background, including software and system flows that connects multiple device across the network with networking technology, compute technology and storage. And if you want to utilize the hardware well, you have to think about the system flows, how data flow between how compute is partitioned between the different servers or the different type of processors. So I saw how the GPU is about to become stronger and stronger.

The system was not built to keep up with that. And actually, it still isn't. And me, it's Vika Iossey, we were looking on where inference and AI will really become realized. Not just research and training, but how will we embed them in everything we do? Just as regular compute is embedded in everything we were doing in 2018. And it was obvious that the system need to change. And we sat down with white papers and started drawing, how would you build if you're free of the chains of legacy?

How would you define things? And this is how we came up with the first product and now we have two more. So that's in short how I jumped to the AI world and founded Neurality. How interesting. Yeah, the industry has spent years focused on AI training, but now the spotlight is on shifting to inference at scale.

So what fundamentally changes when AI moves into production? The fundamental is the training is coming from your R&D spending. And inference is coming from the cost of set. We're using inference. And we've been in the deep tech industry for a long, long time. And R&D, if you have the right product and the right market, R&D spending is nothing. At the end of the day, you win it back

when you generate the revenue and the gross margin on your product. But cost of sale is everything. And we see in the industry how many companies are successful today, but suffering from gross margin. Look at the neoclods that are building and building and raising more money to build more hardware, but underutilizing them and living with a very small

gross margin and the pyramid is kind of is upside down instead of SAS doing 90% gross margin and hardware companies doing 50 or 60. NVIDIA takes 75 gross margin and even a bit more. And many Gen AI and SaaS companies are with zero or very low gross margin. So when you move to production, you need to focus on efficiency, interactivity on different SLOs for different applications, and this is where optimization is becoming critical. Right. So we're hearing the term AI factory more often at conferences and in the ecosystem. So what does that actually mean in practice? And why is it a useful way to think about AI infrastructure?

Yeah, well, when you look on an application that use AI, especially in the Gen AI and large language model era. There's a lot of application layer, business logic layer, whether you're a healthcare company or manufacturing or vibe coding. But under the hood, there's this big LLM with a lot of wisdom and a lot of capabilities. And it's the same for many market sectors. So you should think of it that when I'm taking a lot of hardware that was built for AI with a low-level software that allows me to deploy a model, run it, and serve requests from all these applications, it's actually a factory. The trans LLMs. It has a lot of wisdom, and each application needs a piece of that wisdom.

So instead of each one of us building AI factory from the ground up, you just build AI factory and serve all these applications. This is why we see hyperscalers beyond selling infrastructure as a service, you know, selling serverless API. Because the AI factory can generate tokens from any model you want, and you don't need to deal with it. need to deal with a layer above it as a customer, as a customer that want to build business using AI, you want to decouple yourself from dealing with the model. You just need a factory to generate tokens for you and hide all the complexity of deploying a model and getting the service from that factory running that model. Okay. So when companies try to scale influence into production, where do things start to break? Is it latency, throughput, costs, or

the overall orchestration? Well, it has several aspects. One of them is in production, you're more sensitive to hallucinations, to failures, and we still see a lot of models, including Claude and Gemini and GPP, but also open source models, fail. You do a bunch of things, especially in the genetic world where we're seeing now growing. Agents keep asking a lot of questions and iterations using LLMs. And at some point, the LLM break. So in production, you can't allow yourself to break and capabilities of the GPU,

sometimes maybe 60 or 70 percent. So how can you increase efficiency and optimization to leverage them? And when you dig into that, you find that the fact you to have multi-tenants and multi-user and different models deployed on your GPU fleet, you're losing a lot of active time when you switch between models or when we switch between tenants or when you try to get lower latency and better interactivity for real-time applications and you deploy models on multiple GPUs at the same time. So network delays the traffic between GPUs and GPUs are just sitting idle, waiting

on data. This is a critical thing that actually in your reality we focus on that both from the networking side and from the software side and we'll allow our customers to take something that is more polished, was built for multi-tenancy and multi-user, and we measure everything we do with the output time we can get from the GPU. What's the active time? How do we make sure that the customer that paid for 64 GPUs or 1,000 GPUs is leveraging them to its full potential and maximum output. And this comes from network optimization, from software optimization. And this is the second thing that is important as you go to production and you scale your inference consumption. This is the opportunity to develop all these layers that put reliability on top of this raw,

LLM served by the AI factory and it's a piece that needs to be integrated into the agent as you go to production and want to give high quality of service to your customers. Another thing is cost. Companies that are just at the beginning of usage of inference care less about cost because usually it's, you know, they're focusing on the top line, gross margin is less of an issue, the operational expense of their talent and other tools they have is probably equal or bigger than the AI consumption but when they get to 100 million ALL or north of that the OPEX of AI becomes critical and then you find yourself looking on okay am I using the

GPU that I'm paying for to its full potential and when you dig again, you find that you're not. You're actually using 50% of the capabilities of the GPU, sometimes maybe 60 or 70%. So how can you increase efficiency and optimization to leverage them? When you dig into that, you find that the fact you do have multi-tenants and multi-user

and different models deployed on your GPU fleet, you're losing a lot of, active time when you switch between models or when we switch between tenants or when you try to get lower latency and better interactivity for real-time applications and you deploy models on multiple GPUs at the same time so network delays the traffic between GPUs and GPUs are just sitting idle waiting on data. This is a critical thing that actually in reality we focus on that both from the networking side and from the software side and we'll allow our customers to take something that is more polished, was built for multi-tenancy and multi-user, we measure everything we do with the output time we can get from the GPU. What's the active time?

How do we make sure that the customer that paid for 64 GPUs or 1,000 GPUs is leveraging them to its full potential and maximum output? And this comes from network optimization, from software optimization and this is the second thing that is important as you go to production and you scale your inference consumption and so most real-world environments are highly heterogeneous you know the different accelerators CPUs network stacks why does that make inference at scale so difficult this is a very good point but by the way we're just starting to see that Regenious compute coming to life and Vibia buying

GROC and integrating the LPX into their system to be combined with Ruben and future GPUs. And we're going to see more and more of that. Hyperscalers develop their own XPUs. You have a lot of amazing technologies coming from startups. And I think in the next three years, we'll see more XPUs. And practically the AI computation is consisted of different stage. each one needs something else from the system. The decode phase needs very high memory bandwidth. The pre-fill need compute. When you run the FFN or other steps in the AI, you can build

dedicated accelerator that excel in that specific stage. Now, when you combine different technologies, and we're starting to see that, and it's going to increase in the next two years, as Nvidia, AMD, TPU from Google, Maya and Microsoft, all these will come to life and share data as they work on one big task. The network protocol between them becomes an issue. If you look into each one of those XPU technologies, they took a different angle. and V-V-Link and Infinity Band or Rocky and proprietary protocol inside Google with their Falcon or Maya and Microsoft that is using their proprietary Rocky-like protocol. So how do you combine between all them? How do you connect between them? For that, you need very efficient network cards, Nick or Super Nick, that connects all those GPUs and XPUs. together in one shared network while converting protocols.

There could be one solution, go to ultra-in-net, everyone, but sometimes for stickiness, a lot of the vendors like Nvidia, they want to keep using Infinity Band and NVLink and have this stickiness with the customer, but now they need to connect with other technologies. So networking is a big issue. And one of our products right now is an AI supernick that has a very efficient pipeline for rocky communication or ultra-eternet but also have a flexible transport engine that allows us to load through software and programmability any protocol from any company.

And this allows you to connect between different type of XPUs in. in heterogeneous clusters without losing a lot of latency on going through a bridge that will convert the protocol. So I think the next few years in order to support growth of usage of autogenous compute without losing a lot of the efficiency and the active time that we talked before about, it's critical to have products like that, that know how to convert protocol very fast, And hopefully the industry will also align to more unified, internet-based solution, because this is what's right for the world, what's right for power consumption and green technology. Usually when specific vendor goes to his own proprietary solutions,

it's the opposite of open source and sharing the power of the community. And I think you see projects like OCP, Meta-led and Microsoft that try to combine everyone, all vendors around one technology. And then you see stickiness approach like Nvidia did with EnviLink and Infinity Band that doesn't allow this open. And I think we're going to see more open technologies winning in the next few years because this is what the world need. and the scarcity with GPU, so we have to bring more efficiency to the market and more interoperability between all these technologies that are developed in different companies. Yeah, I agree completely. So how does new reality health organizations move from fragmented infrastructure to something

that actually behaves like a scalable token factory? So the software product we are selling today is to. targeting two segments, the inference consumers, the Gen AI, SASS enterprise companies that are buying serverless API or owning their own their own software stack and buying infrastructure as a service. And we give them one layer of inference operating system that we called NR Nexus. And what Nexus gives you is hiding the complexity of building and integrating these open source with a different vendor stacks, whether you choose Nvidia, AMD, or others.

The nexus layer is agnostic to the underlying hardware. It can run on this GPU or debt XPU, and it can leverage a networking solution that connects them from different vendors, whether from Broadcom and VIDIA or our own NIC technology that we will sell. By the way, our AI super Nick is going to be available next year with a 1.6 terabit. This is the next generation bandwidth that NICS will have. But we don't force anyone to buy our hardware. If he buys Nexus, he gets the flexibility and ease of use of one token factory or inference operating system. through a control panel, he can deploy models, he can set different tenants for different applications he use or different departments he has in the enterprise. And what Nexus gives him is to extract

the maximum output as it load balance the consumption that all his applications are using between the different GPUs. So one GPU may serve a GPDOSS at one hour in the day and a DIPSEC model in a different hour of the day, just based on consumption. Nexus will auto-scale models and load balance between them and really hide the need to deal with all these inference engines because it deployed the inference engines inside,

whether it's SGLang, or VLM or Dynamo from Nvidia. So we allow them, those SaaS companies to get a license for Nexus, buy the cheapest infrastructure as a service, on-prem or on cloud, load it there and get the look and feel and the ease of use of serverless API. On the other side, we go to Neo-Cloud. And Neo-Cloud today mostly sell infrastructure as a service. And we want to allow them to sell inference as a service.

This will help them get out of this impossible situation that their gross margin is very low, although their spending is ultra high. And right now, the market is high, there's GPU scarcity, so they can keep living. But when the market will settle, the ones who will survive will be the ones that has the right growth margin to survive. And this comes with selling software. And Nexus allows those neoclods to go to market and sell inference serving instead of just infrastructure as a service and go from high capital spending to a more zero marginal growth that Nexus allows them to have and sell more inference on the hardware that they, they bought and built.

Okay, and what are the new metrics that defines success in inference production? And what should teams be optimizing for? I think the KPIs are the obvious ones. You're buying a machine, any machine. You have a pizza shop. You want your oven to work as much as possible. So you're taking orders in the restaurant. You're taking all orders on the call and you have delivery people that takes the pizza to different house.

If you have one oven, you want it 24-7 working. You have two ovens, the same. That's the same with GPUs. We pay, especially today, we pay premium on GPU technology. And we need to make sure that it works around the clock and being 100% utilized. So one KPI is, are you using your GPUs to its maximum potential? And for that, you need a Nexus layer or an alternative that will give you multi-tenancy and multi-user. The way to use the pizza oven more is to get more orders.

And that's what Nexus does. It gets as much inference requests from as many applications as possible. and allows your cost of sale to be the lowest. Another KPI is when you use real-time applications or use agents that are smart and work and you don't want to wait for two days for a very complex job. So you want something very interactive. Sometimes you want to pay more.

You're willing to pay more as long as you get lower latency to your model. and for that you need to spread GPUs, spread models to run on multiple GPUs at the same time and share data between them. So you want to know that your network speed and the sharing of data is fast enough. So you want to track the speed of your network and again GPU utilization. And last, how much do you pay for your token? And this comes from the Kappa and the OPEX. And the two things you need to measure here is how much are you spending on every token and how many tokens are wasted because of this reliability layer that is dropping it because it has had hallucination. And this will, as you measure that, you'll find ways to move to a smaller model or to a fine-tuned model with your data. because this wisdom of the big models is coming with the churn.

It's coming with extra computation. If you go to smaller open source models and you fine-tune them with your data, you'll end up paying less per token. These are the main KPIs. Yeah, that makes sense. Final question, Moshe. How do customers normally engage with your company? Well, this varies. So a lot of the customers are just tracking publications we do and data that we release.

A lot of it is being done by business development. We do with the big companies, whether it's semiconductor companies, NeoCloud. The smaller customers will find us also in conferences. We have our website that invite you to see a demo of any of the products and contact us for allocation of hardware and POCs. Those are the main ways we meet customers, either actively going after them or getting inbound requests to our website

or emails or LinkedIn. Great. Thank you for your time, Moshe. I appreciate it, and hopefully we can connect again to get an update on your success. Thanks a lot, Danielle. Pleasure to be with you. That concludes our podcast.

Thank you all for listening and have a great day.