In The Arena by TechArena - Hedging AI Risk with Midokura CTO Dan Mihai Dumitriu

Episode Date: July 16, 2026

In this episode of The AI Hedge, Hedgehog Founder & CEO Marc Austin sits down with Dan Mihai Dumitriu, Founder & CTO of Midokura, to talk about sovereign AI infrastructure, data privacy, AI inference costs, open weights models, AI networking, SONiC, RoCEv2, Ultra Ethernet, and the future of AI infrastructure in Japan.

Hello, welcome to episode one of The Hedge, where we talk about hedging AI risk with leaders in the AI industry. I'm Mark Austin. I am founder and CEO of Hedgehog, and my guest for this episode is Dan Mahai, who is founder and CTO of Meidel Curie. And Mark, thanks for joining me. Thank you for having me. We talk about hedging AI risk, and it seems like there are two big. big risks in the market, particularly as we talk about enterprise adoption of AI, cost, and data privacy. Let's take the data privacy one first. What, Dan, should enterprises be thinking about in terms of data privacy and data sovereignty? All right. Thank you very much, Mark. What we're seeing in our core market in Japan is companies are concerned about their data being

used in various ways by AI companies, which they may access via API, rather than having that all staying on premise. And so being naturally conservative, they would like to build their own sovereign infrastructure to do both AI training and inference. So this is like the core idea. And I would say we should look at how things have evolved over the last few years. The frontier models, the ones that finally became very useful, shall we say, we're basically behind APIs, right?

As of two years ago, one year ago, you know, like think about the open AI, GPT3, GPD4, et cetera. But now open weights models are really quite capable. And so that really opens the door for, like you said, sovereign infrastructure as a viable option. So already not even talking about the cost aspect, that is something that is now a viable option. Okay, great.

And that's part of what you do at Mito Kerr, right, is make private or sovereign AI infrastructure possible for more businesses in Japan. That's right. That's right, exactly. Yeah, because you have the hardware aspect. We don't build hardware. We have plenty of good chips from AMD, Intel, and Vindia, et cetera,

and various other startups now doing all sorts of silicon and PUs and so on. But operationalizing all of that stuff is really far from trivial, to say the least. It's actually much harder than what we used to do with on-prem infrastructure, like just servers and VMs. It's still servers and VMs.

However, the workloads are so sensitive that tuning all this stuff is actually challenging. So we want to take all that out of the picture for the enterprise customer. We want to do all that so they don't have to do all that. Great. So let's just talk about like a fictional scenario for an enterprise in Japan,

what they might want to do with AI, what their data privacy concerns are, and then how does Meadowkura help them? Sure. So let's take, for example, a, manufacturing company of some sort, because maybe it's cars, maybe it's something else. And you can imagine in their operations,

they have a lot of data, they have their domain knowledge, they have a lot of data that they've collected, and they want to use that data to train or fine-tune AI models that may help them in their operations. That could be things like visual inspection for quality control, and it could be things like various types of robotics or what they call physical AI these days, right?

So they have their own data sets. They want to keep that as part of their competitive advantage somehow. And so they need to have their own on-prem infrastructure. Because if they use a public model and provide all of that proprietary, sensitive information to the public model, there's a risk that they lose their proprietary competitive advantage. Yeah, maybe. I don't know the terms of service in detail, to be honest, you know, probably sometimes they're more or less invasive. Yeah.

And it requires some level of trust in a third part. Certainly requires some level of trust. I think there's also the point that when you use, the third-party APIs, you can't really find tune in the model. You use what's there, right? So switching gears a little bit, looking at the LLMs that we use all the time in these APIs, the way that we use them tends to be the so-called zero-shot learning,

meaning that we put all of our specific context, all of our specific data into the context, and then prompt it, and then the model is always the same for everybody else. Because of the context, it will actually learn stuff in context. That doesn't always work, though. and it's not practical for a lot of other things. And it also doesn't really work very well if you want to make the model more efficient and smaller.

Then you really have to fine-tune it. And fine-tuning, it means running some kind of process to fine-tune it. You can't fine-tune open AIs closed-source front-to-your model. Maybe they would do it for you. I don't know. But you could also just have your own stuff on-prem, your own dataset, use some open-weights model. You fine-tune that, and you make sure it works well for your use case.

And then you deploy it. So then you do it. deploy it means you deploy inference. You deploy inference service, which requires another kind of tuning. And depending on what type of inference you're doing, how often you're doing it, you need to think about the cost. How much do you need it? How big does the model need to be? How, et cetera, et cetera, right-sizing the infrastructure, right-sizing the model for your problem. Okay. But there's at least the opportunity to not only protect your data, but also at least know what your total cost of ownership is going to be for running AI inference. As a

opposed to getting a bill shock with however many tokens everybody's using. It's as you were saying earlier, at least you know the upper bound on your TCO. With the CAPEX may be big or depending on your situation, but you know the upper bound, whereas you're using the public services, you don't know the upper bound. Okay. So is that all I have to do?

I just need to go to write Jensen a really big check for a whole bunch of GPUs and I'm good? Or is there something else that Mito Kura does that enables me to start doing this? Yeah. So we do various things. The software stack is based on various open source projects, the best of breed. For example, OpenStack, Kubernetes, software-defined storage, and, of course, Hedgehog for the network layer.

We are also working to enable more different kinds of compute accelerators. So, of course, we have CPUs. They're getting better and better all the time. We have GPUs, notably Nvidia. That's the thing that we started with. That's our primary thing. And then we're also enabling AMD GPUs. and we're also starting to partner with other silicon companies

that are making AI-focused inference accelerators. And so what do you mean by enable is all of these things, in theory, work out of the box, but they don't necessarily all work well together. And to be concrete, probably people have heard of this pre-filled decode disaggregation in LLM inferencing because the inferencing has these two phases, the pre-fill phase and the decode phase. They have very different profiles in terms of computer memory use. And so to really be most optimal, you want to use one type of device accelerator for the first phase,

a different type for the second phase, and then they have to communicate with each other in an efficient manner. And that doesn't work out of the box, particularly if it's heterogeneous. So that's something that we're working on. And so over time, we want to see how things evolve, and we will try to provide the best-of-breed solutions for various use cases. Okay. So I'm going to try to summarize all that.

So Medocura provides an open source software stack that combines a number of different open source projects that otherwise are difficult to run together. And you run that on multi-vender AI acceleration infrastructure. And you're matching the accelerator to the specific use case. Correct. Great. Okay.

So if I'm using Medo cura and I'm an AI developer, what's my developer experience? What do I get to do? So the developer experience, depends on what you're doing. Again, you might be doing training and fine tuning. You might be doing just inference services. So let's take the most simple example.

You get the stack, hardware, and the software installed. Then you deploy a cluster, a virtual cluster, which may be bare metal or virtual machines, depending on your use case. And why would you do virtual machines? It depends, you know, depending on your organizational structure and so on. And then on top of that, you can deploy Kubernetes as a service, managed service.

Then on top of that, you deploy, let's say, an inference service using an open weights model. So all of this you can do now from basically a point-and-click GUI API, more or less all these different layers. You can do it all automatic. And that's the Metocura software stack. That's the experience. So you can script all that and just do it all in one shot.

For example, after that, you're using the inference API. You have to monitor it so you have some observability built in, scaling options, and so on. This is the basic experience, so we say. Okay. Great. And all of that requires a network. Correct.

Okay. And you have some experience in networking. We do. We do. Mitokura, when we started in the very beginning, which was like 16 years ago, our objective actually was to do networking for virtual machines, for on-prem infrastructure. And so I guess you've been around long enough to remember OpenStack from the beginning, 15, 15 years ago. That's what we also started using in the early days. And we built a network virtualization overlay that was an open stack plugin, essentially.

And that was called MedoNet, right? Correct. And that was providing virtual network segments for virtual machines. Not only the network segments, but also other layer three, layer four services. That was all software. And it was all virtual switch based running on host switches, on host servers. So the overlay, meaning that we really didn't care or touch the underlay network in between, that was like not our problem.

Obviously, things have changed in the meantime. Yeah. So that was going to be my next question. So given the fact that you've been doing open networking for 15, 16 years, you're an expert at it. Why make the decision to use Hedgehog? Okay, good question. Yeah. So things have evolved quite a bit in the meantime.

And just to be clear, I and we weren't doing networking and server infrastructure the entire time. We also had a little bit of a parenthesis doing IoT and an edge AI in the meantime. Which is relevant for AI inference, right? A lot of your inference data is. going to be coming from IoT to us. That's kind of how we, let's say, the AI and the infrastructure came together and here we are again in the data center.

So in the past, like I said, we were doing all these software-based overlays for virtual machines. Now we have a more complex picture. We also have bare metal servers, and we have in particular these AI workloads. So just talking about bare metal servers, if you have a bare metal server instance, we can manage it using OpenStack and all that

through the BMC interface and all that. But essentially, there is no place, There's no hypervisor. So there's no place to run a virtual switch, which is in a different administrative domain from the server, it's from the workload itself, right? And so we have to do the isolation at the next hop, which is the switch, the Ethernet switch.

Okay. Servers connected to one or more ports on the switch, probably many ports these days. Yep. And so from our control plane, yeah, exactly. Maybe like 10 next to know, right? Yes. So we, from our management plan,

or bussack essentially, we need to then essentially, let's say we allocate a bare metal instance to a tenant. We need to say, okay, now those ports that are of that server that are plugged into this switch, those are now assigned to this particular tenant, which may be a virtual switch, right? A virtual switch that this server is plugged into. So we need to manage the whole network fabric now, right? Not just the overlay. And long story short, we learned about Hedgehog and we really like it as a piece of software,

and we like the data model and how it operates and is working well for us. So we've integrated the Hedgehog API with OpenStack and in our operating, model and so everything is automated. So that's one thing, one thing that's changed, like I said, is now we have to manage the whole network fabric, underlay and overlay, essentially in one, and we do that with Hedgehog. And if you spend all your time trying to figure that out with this new, more complex AI fabric, you're not spending time on where you can add value, which is making it easier for developers to use. Correct.

Open weights models, fine-tune them. Exactly. And have lower cost, higher-cost, higher- quality inference for private enterprise data. Exactly. It's the same calculus that our customers would make by getting our infrastructure product we make by using Hedgehog because we need to spend our time where we can add most of that.

Makes sense. So anything in particular about Hedgehog or the Hedgehog API, other than just, okay, you handle all this network stuff while I can focus on my core competency that your team liked and anything you didn't like that we should improve on. Okay. Let's start with what we like. So like I said, the model fits what we believe is the right kind of scalable network infrastructure. To be concrete, that means that we have this BGP-based underlay, layer 3 network,

and then we run the NIs over that, switch to switch, port, et cetera. All of that is configurable from the single controller. And the shows a logical data model, having switch, VPCs, etc. All of that fits how we like to think about the network. There's that. The second thing is also that Hedgehog does have specific. features that are relevant to AI and notably configuring the QOS for Rocky V2, which is what we must use for host-to-host GPU to GPU communication to redo RDMA in these AI workloads.

So the fact that all works out of the box is very nice. And of course, Hedgehog is built to work with Sonic, which is the switch operating system, right? I don't need to tell you that, but. You might want to tell the audience. I'm not sure everybody listening. to this podcast is going to know what Sonic is. Start up. So Sonic is essentially an operating system and a few pieces of control software that run on an individual switch, right, on the CPU of a switch. So as some people may know, an Ethernet switch has two parts, essentially. It has the ASIC, the Ethernet ASIC that does all the forwarding, which is a high-speed programmable

thing. And it has a CPU, which is a regular old CPU that runs Linux and a bunch of control plane software and some of that software talks regularly talks to the ASIC in order to program the forwarding rules. So Sonic is the distribution of Linux plus this other control plane software that runs on the CPU of a switch. Right. Yeah. And so that then is the individual unit of the network, the switch. Then the Hedgehog controller talks to a bunch of switches to program the layer three routing. So for all the BGP connectivity between them, the underlay. And also, the layer two or three VNI VTEP overlays, I think are VX line, right,

that run from switch to switch as well, that then make the virtual connections between ports and switches. So your domain expert, you've just given us all a pretty good primer on an open network operating system. What's the risk of just running Sonic on your own and not using Hedgehog? Again, Sonic is just a one switch solution. Yeah, you can get Sonic, install it, but you get, you probably get a distribution from your switch event.

wherever that might be, they have their own supported distribution of the Sonic OS. But that does not give you a fabric. That gives you a single switch. You still need to create the fabric. So you could go and program switch by switch, the BGP, and you could program the installation of VNIs and all that. But that would be error prone and definitely requires domain expertise as well. So the underlay part, let's say it doesn't change that often, right?

Because it's only changes when you add or remove a physical switch or servers. But the overlay actually changes all the time. That's all based on the provisioning of the workloads. So it's actually impractical to do that by hand. You can't do that by hand. You have to script or automate that somehow. It's not as easy as it sounds because it is stateful. It's, let's say, intent-based and in a sense, right? So you have to maintain that state somewhere. And so you need a controller of some sort. Okay. So we started this on what do you like about hedgehog. You mentioned Sonic. And Sonic gives you sort of device choice. Is that what you like about Sonic.

That's right. Exactly. Okay. So we already worked with several different vendors for the switch side and the Sonic distribution on top. I think they're all Broadcom, to be fair, right? They're all Broadcom ASIC. So Broadcom Sonic, some various flavor of Broadcom Sonic is running on all of them.

Correct. In theory, we could have more ASIC vendors. In practice, we don't, but we'll see what happens. But yeah, definitely the device choice is important. Anything that you'd like to see us improve on? Improve on. I think we're very happy with the product now. Maybe we could talk about what we see going forward and, you know,

how might that interact with the product of the hedgehog the product? So one thing that we see in these AI workloads is reaching the limits of Rocky V2, which I'm sure you've talked with your CTO at like, right, is essentially RDMA over converge Ethernet, but essentially it's RDMA over IP, over UDP. You might want to give everyone a little bit of a background on RDMA and maybe in the van. Some people have probably heard about it in FinnaVam. So RDMA stands for remote direct memory access.

So the idea is that the CPU on one host can access the RAM on another host directly, quote-unquote. Of course, it's still going over the network somehow, but it looks like you're accessing memory on the other host. That's what it originally meant. What's relevant nowadays to the GPU is this thing that NVIDIA calls GPU direct, which is that a GPU can communicate with another GPU without going. going through the CPU. You can go from GPU to your NIC network interface card device over PCA over to another host. And again, from that Nick to the GPU. So you can access memory from a remote GPU from your GPU. And that's a big deal because the GPUs have a lot of stuff to

transfer when they're doing training and even when they're doing inference now with this pre-filled decode the segregation. So getting the CPU out of the way is often very handy. So that's RDMA. Now, Rocky V2 is a particular implementation of RDMA. There was a, there was a, there, there, There still is another technology called Inciniband, which is it's a full-stack solution that is meaning all the way down to the physical layer that does RDMA now working. But for various commercial reasons and scalability reasons, that's going away. And so RDMA over Converge Ethernet is winning, but there are some issues, right? One of the issues is that when you're running these large AI workloads, let's say,

distribute a training job, sometimes in different phases, you have to transfer quite a lot of data from one GPU to another, or maybe from all GPUs to all other GPUs, and they tend to be highly synchronized in time. Your network utilization just spikes quite often. And each individual flow that is like a flow of data from one GPU to another tends to be quite large, right? Yeah. Elephant flows.

They're like almost all elephant flows now when you're doing a big training. And so that was not the way that the network was envisioned to work. RDMA sends all the entire flow over a single path through the network. Yeah, Ethernet in particular wasn't designed for that kind of scenario. Exactly. And in the past, I think when we built these networks, like this Klaus type network, for virtual machines, let's say, we used to imagine, okay, we have a bunch of east-oess traffic, north-south traffic, etc., right? But we were always counting on statistical and multiplexing that is randomness to make sure that we don't overwhelm any part of the network. That doesn't hold up anymore because of the workload.

Now, there is a very high chance that elephant flows will compete for the same bandwidth on, the same link. It's not statistically random. It's actually like highly correlated. Yeah. Yeah. So we need new solutions for the transport, instead of Rocky V2. And there are solutions.

So we're not developing that stuff, but there are things in the pipeline that are coming now. For example, the ultra-ethernet transport, do you guys know about? Which is essentially a reimagining of a transport protocol for this type of traffic. And so one of the things that it does that's really interesting is that instead of setting the entire flow over a single path, it brings. breaks it up and sends parts of the flow over different paths in the network, maybe even packet by packet. What then it becomes challenging is that you have to reassemble those packets in the right order at the receiving side.

So practically what this means is that you must implement this in a very high speed manner on the nick itself. So the industry is now tending to call these things hardware transports, like as in transport layers implemented hardware rather than the OS, right? And Rocky V2 is also that. It's a hardware transport. So these things are coming. Google has their own solution, which is really interesting. is called Falcon that they presented last year at SACOM I went to this conference. This was like part of a big, a big theme there. So we believe we're sure that these things are going to come. You know, I mean, commercially.

You can't get it really right now practically, but it's coming. Yeah, there's an ultra-ethernet consortium. That's right. And the point is, your point, is that, hey, look, Hedgehog, you better be supporting ultra-Ethernet. Whatever supporting means, it's got to work with it. Yeah. What do you think the timeframe for that is?

Like, when you think ultra-ethernet's going to be commercially viable? Yeah. But in theory, it's commercially viable now, in theory. In theory, in theory, AMD is supposed to have their Pensando, whatever, Nick, that supports that, but I don't think that's truly available widely yet. Yeah. You would know, you're out buying this infrastructure. Yeah, I don't think we're quite buying that yet, to be honest, right?

Yeah. It's not yet mainstream enough. So it would be very niche. I would say that stuff is really for the hypersalcalers initially. So I would give it, honestly, a couple of years to be truly relevant for the medium to large enterprise. Yeah. And you mentioned Nix. You mentioned Ponsondo, which is an AMD company that does a smart Nick, and NVIDIA has Bluefield. That's right. And I know at Hedgehog, we're already doing a lot of

work on Connect X Nix for the gateway part of our product. But tell me, what do you see in the future for smart NICs and they're sort of their role in these AI fabrics or high performance networking fabrics? Yeah. They already have a huge role right now. Yeah. Just because they implement the direct GPU direct and RDMA, a transport, even if that's a Rock EP2. So it's a very important role that they play in the infrastructure today. Also, the storage direct, so accessing storage directly from the GPU is also an important point. I guess the question is, if you have to use these things, do you also have to manage them all from the controller? You know what I mean?

There's a lot of moving parts, right? A lot of moving parts. Any given GPU server, you got HDPUs, you've had the 8 ConnectX Nix, you got a couple of smart Knicks as well. There's a lot of stuff. Exactly, exactly. So does Hedgearm manage these nicks currently? Yeah, so we do host-based networking where we're configuring the NIC. That's part of the way that we're able to get really good nickel performance on an OCP reference architecture network, which is what we're here to announce that. What you're using is now the OCP reference architecture for AI networking. Thank you.

Great. Thank you. Thank you. Thank you for helping us develop that with the rest of the community. That's right. Yeah. And yeah, there's a, I don't know if there's an official name for it that's different, but I just call it the Neo-Cloud Working Group, OCP, which has a number of players that are in this infrastructure or platform as a service for enterprises who are wanting to adopt AI. So that's a really

great community, too, where I think you're going to have all kinds of things that you're going to be able to contribute and collaborate with the community. I think we'll look forward to joining that. Yeah. We haven't been much involved with OCP, but we will be. Yeah, which is ironic, considering you're starting with the whole tenant of, hey, let's do things as open as possible. That's true. That's true. I think it's just because of the way that the company evolves. And also most of our focus being on the Japan market industry. Okay. So let's talk about Japan for a little bit. How do you see the Japan market evolving? What do you see is Mitokura's role in AI for Japan and sovereign Japanese AI infrastructure?

So Japan is a paradox in so many ways. But in terms of this AI infrastructure, they really like the sovereignty aspect. They have a lot of challenges in actually building that out. Notably in terms of power. So it's not the most power-friendly country in the world, although it does have the nuclear power infrastructure. It's got some geothermal stuff too, man. Got some geothermal. I don't really know much about that. It's got a lot of spare nuclear because they shut it all off in 2011.

And so there's actually a good reason to bring it back on, shall we say, right, to power some more of this AI data. So there are several government initiatives now to build AI infrastructure in different regions in Japan, like, in Hokkaido and also in the south in Kyush. All of these things need software. They need automation. They need the expertise of scaling and operationalizing all these things and we're going to help. Yeah. And it's not easy to do, right? Not easy to do. Just having an in-country expert who can help them out with that is huge. That's right. Great. All right, Dan, thank you so much. We talked about a ton of stuff. We went way deep. And we talked about a lot of different risks in AI infrastructure and how you hedge them at the AI developer level, the enterprise DevOps level, as well as focusing on your

core product and trusting us to take care of a lot of details in the network. So thank you for your continued partnership. I appreciate being your guest. Thank you very much. All right. Thanks a lot. Thank you.