Grey Beards on Systems - 133: GreyBeards talk trillion row databases/data lakes with Ocient CEO & Co-founder, Chris Gladwin

Episode Date: June 10, 2022

We saw a recent article in Blocks and Files (Storage facing trillion-row db apocalypse), about a couple of companies which were trying to deal with trillion row database queries without taking weeks to respond. One of those companies was Ocient (@Ocient), a Chicago startup, whose CEO and Co-Founder, Chris Gladwin, was an old friend from … Continue reading "133: GreyBeards talk trillion row databases/data lakes with Ocient CEO & Co-founder, Chris Gladwin"

Hey everybody, Ray Lucchese here with Matt Lieb. Welcome to the next episode of Greybeards on Storage podcast, a show where we get Greybeards Storage bloggers to talk with system vendors and other experts to discuss upcoming products, technologies,founder and CEO of Oceant. So Chris, why don't you tell us a little bit about yourself and what Oceant is all about? Hey, it's great to reconnect. I know we worked together over the years and it's fun to kind of catch back up on what we've both been up to. But on my side, you know, I spent a lot of time in my career kind of building new companies,

particularly with Cleversafe, which was a company that ended up dominating the market for on-prem object storage. IBM bought that in 2015. And since that time, I've been working on building a new company called Oceant, which is building hyperscale data analytics solutions. Basically, we focus on helping organizations do the largest data analysis systems in the world. So I see a lot of discussion here about trillion row data structure, databases, whatever. So trillion rows seems like an awful lot of data to be messing with here, Chris. I mean, can you guys really work with that sort of data load? Yeah. So trillion is a number that I think one of the interesting things about it

is it's kind of the first number that a human brain really can't comprehend you know like maybe at a million you can start to wrap your head around these things maybe at a billion but at a trillion there's just no way if you were to take a spreadsheet with a trillion rows and print it out it circles the earth 73 times yeah you know human cannot look at something like that so one of the interesting things about trillion scale is like you can't cheat and just go look at the file look at the data you could in billion scale kind, kind of sometimes, but not in trillion scale. So that's one of the challenges is now we really have to rely on these data analysis tools because we can't look at the data anymore. And where trillions come from, you know, it's never typing.

You know, humans can't type that much. It's always machine generated. And so there's these machines that we've, that we've created that not only generate, you know, tons of data every second, all the routers that you have, they make records every time they do anything. Anytime they connect anything to anything, source IP address, destination IP address, location, packet type, how much data flowed, you know, location, all that. That's a very valuable thing to analyze for all kinds of reasons if you're a telco but you make across all

the routers in your network you make those things you know trillions every second you know so that's one example if you have 100 million cars you know as those cars are more and more digital and you're like analyzing a fleet of cars because you're manufacturing cars that's going to be trillion scale if you got a billion mobile phones, they're going to be a trillion scale. So there's, there are these, these data sets that are created that are definitely a trillion scale, if not a hundred trillion scale. Yeah. But you can't do like an SQL query on a trillion row database. I mean, it seems like a never ending story, right? Not, Not unless you want to wait forever for the responses to come back, right?

So where do you guys fit into that sort of question? I mean, can you really do a, let's call it a trillion-roll SQL query in a second or two? I don't even know what the number is. Yes, is the short answer. And that's the whole reason we started OCEAN. And the way it started is at CleverSafe, we spent 10 years of our life selling to the largest bit storing organizations in the world. And, you know, we really focused on the top 500. And, you know, this is interesting

because if we would have had this conversation in 2005, you know, in the second year of CleverSave, I remember calculating, making a spreadsheet that estimated how many organizations in the world at that time had a petabyte or more of data that they were storing. Which, by the way, is a corner of a server now. Yeah. by the way, is a corner of a server now. And at that time, my estimate was 14, that there were 14 organizations that had a petabyte of data. And I remember going to the engineering team that year and saying, because every year I had a theme for CleverSafe. My theme that year was petabyte excellence. We're going to be really good at petabytes. And I remember they all thought I was crazy. That's ridiculous. So we're going to be really good at petabytes. And I remember they all thought I was crazy. Like that's just, that's ridiculous. So we're kind of that way with Trillion. Although we estimate right now, the number, the number of people that are storing Trillion scale data sets

is in the hundreds right now. Those who are analyzing. So there's a difference between like, I just dumped 10 petabytes of data in Hadoop versus I just ran a query, a complex query on all 10 petabytes. I actually did that. You know, like there's like storing and actually analyzing are two different worlds. Yeah. Yeah. I'm presumption being someday you might actually have the resources to go do something with it. Right. Right. So there's a lot of that. Just dump it there for now and we can't really analyze it. We'll figure out later. So that was the whole premise. So when we got to know the largest data storing organizations in the world at CleverSafe, a bunch of them were actually who asked us to form OCEAN, the largest of the large, one of the trillion dollar tech companies,

the biggest data analyzing and that you know the biggest data analyzing and storing organization in the u.s government the top two and one of the might major telcos they were the ones that asked us if we could because they had seen us come into storage and figure it out make it limitless scale its cost doesn't go up as it gets bigger and it's still reliable and all that stuff they said could you do that not just for storage but analysis and so we thought about it for a long you know like thinking about it in that case meant like a whole team of people analyzing it you know for for a year and we convinced ourself we could actually do what you guys were just saying which is run queries that actually do things a trillion

times a second, you know, and above and do it on cost efficient hardware. You know, I mean, yeah, of course you could build a billion dollar supercomputer to do this, but that's not feasible. So yeah, we figured it out. It took us, you know, one of the things we realized early on was, well, first, to do this, though, you'd really have to build a whole new database architecture. And most every other new database company, we're not aware of anybody else that has taken that on. Because normally you can use Postgres or one of the big open source engines, and you could modify it a lot and then make it into some new company. But in this case, it wouldn't work. And, you know, the reason the reasons come down to all the old. There's really only three other database architectures.

There's end memory, which is you load everything in memory. It's super awesome. Can't do that with a trillion database, though, right though right right because memory is five dollars a gigabyte and if you have you know an exabyte that's a little tricky um you know then there's loaded on spinning disk which is um you know like the whole hudup ecosystem at the end of the day lives on spinning disks somewhere that was their architecture and it was great you know but the performance is like you know, like the whole Hadoop ecosystem at the end of the day lives on spinning disks somewhere. That was their architecture. And it was great, you know, but the performance is like, you know, hours to days to run queries at that kind of scale. And you can put those architectures on solid state and they'll go like 10 times faster, the Hadoop based ones, but they should go a

thousand times faster. But to do that, you got to re-architect the whole thing from scratch. And then there's the data warehouses, which try to get the best of both worlds where they like store the data in S3, you know, pull it across the network into compute tier. Those are great for elastic workloads or if what you're analyzing is relatively small. But if you just want to hammer a thousand times an hour, trillions of things, it's not going to work. It's going to overflow your compute tier, which means it's going to thrash back and forth. And the cost of that happening, even if you're not, then if you want to have a giant compute tier, it'll cost like a million dollars a week to actually run this thing. They don't make sense. Right. So we build a new architecture where you actually, like our specialty is queries that are going to hit trillions of things. And, you know, the cache is not going to help you because it's not, you know, you can't do this out of cache. You separate compute and storage in two tiers. It's a problem. Like you can't pre-aggregate the results because it's some kind of ad hoc query or something like that.

And it's just physically going to have to look at 10 trillion things and you want an answer in a second or like 10 seconds. Like that is what we do. And the way we do it is it's crazy parallel, I guess. It's impossible. How can you possibly do such a thing? It's impossible to do 10 guess. It's impossible. It's impossible. How can you possibly do such a thing? It's impossible to do 10 trillion questions. You can't do it. I have to go back though, Ray.

You're saying you're running this on some level of commodity hardware? Yeah. Yeah, so the thing that also happened since we started, and we knew this was going to happen because the hardware manufacturers were telling us the biggest change in hardware in computing was about to happen which was a new thing up until now up until you know a few years ago all the hardware innovations were like bigger faster cheaper of what was there before so like you know the you can get twice as much memory in a memory chip or the CPU is this much higher clock speed. More cores, that sort of stuff. And then GPUs were a big deal.

GPUs were a new thing. Yep. But then the new thing that came along was solid state. And the first generation of solid state didn't count because what they did is they stuck it behind a SATA interface and the S stands for serial, which makes sense for a spinning disk because a spinning disk physically wants you to feed things serially and extract things serially. They're terrible at random. You get a hundred or a thousand times the performance if you do things sequentially. So there was a lot of brilliant engineering, particularly in databases, to make a query that really wants to have all these random reads physically look to the spinning disk like sequential reads. And the problem with spinning

disk is how fast it can deliver random reads, which is really at the end of the day is going to determine how fast your database is going to go was stuck around 500 per second and it had been that way for decades because spinning discs are like record players and it's a physical thing it's how fast is the platter spinning and how fast does the read write head move and like it's you can't just moore's law that for 50 years mechanical issues are a problem not to mention that a 20 000 rpm disc can't go any faster because it will physically break through the walls of the enclosure yeah exactly if you kept spinning discs at moore's law it'd be like it'd be like a nuclear power reactor inside of that thing it would blow up. So on a Moore's law adjusted basis, spinning disks have been getting slower and slower. And one way to think about that is

if you looked at, actually Tom's hardware did this amazing chart about a decade ago, how long would it take to read or write an entire spinning disk? And if you go back to the 80s, it was like 90 seconds. You could read or write the whole disc. But the eighties is like a gigabyte, a hundred gig. I don't know, something like that. Yeah. But you could rewrite the whole disc in like minutes. And then, you know, I remember like kind of at the end of CleverSafe, the last, you know, in 2015, when I, the last time I was thinking a lot about spinning discs it was like three days it had it had moved and then now it's probably you know because those discs are giant now it's probably something like 10 days and anyway so that's one of the

constraints that you always had either could spend five dollars a gig for dram or you just have these painfully slow, you know, random reads. Solid state, and once the NVMe interface, the non-volatile memory express interface got stuck on the front of solid state, it's a game changer. And so in terms of volume, you know, this is one of the amazing things you sometimes see in computing that we get to be a part of, like these capabilities that were previously like unimaginable supercomputer all of a sudden it's in your phone so your phone has an fme solid state drive and so does your laptop it that's what it has and the whole it's a great standard because everybody's using it it truly is interoperable and it's gone from you know at oceant um obviously we buy a lot of these drives we need

them for our development lab and other things three years ago and we have friends in the industry it's not like this is our first rodeo we could barely get our hands on these drives like they like all the fabs like intel's fab it was like the the r and d fab that they build well the first time they do something and so like you know you had to like have friends to get like six of these um and it went from that to it's everywhere like it's in billion a year scale and if you want 10 million not a problem yeah yeah yeah not a problem and Yeah, yeah. Like at the end of the day, how fast your database could go is, you know, on a big query where you're looking at a lot of stuff is going to be determined by the physical number of random 4K reads per second that you got to get out of the drive. If you architect your software perfectly, you're going to go 90% of that speed.

That's the end of the line. And so you can think of it like with a spinning disc, you had one brick layer because that's really what it is. A brick is like a 4K block and you're pulling it off the pile and you're putting it on the wall. And then, you know, you can, it's like, that's like a, if you think of that as a, as a right, you know, read as you pull it off the block, you know, the wall and you put it in the pile, like how fast can the brick layer go and so you had one brick layer with a spinning disc because it's serial that's it and it could do 500 a second or something like that yeah yeah yeah like 500 a second that's it that's what you get period with a nvme solid state drive

you get 256 brick layers and each one is six times as fast. Like that's today. And they're on the Moore's law curve. So the next version, you'll get 512 bricklayers. The version you'll get a thousand, the version that this is on their roadmaps. So like in the time that the spinning disc could make a wall after a day of

work, like you build a neighborhood, you know, it's just an entirely different ballgame but to do it like then think about like the rest of your system has like before your database just had to make the pile of bricks as fast as the brick layer could one brick layer going pretty slow 500 a second could build a wall well now it's like no no no you need like 256 piles of bricks and you got

to do all this flow control. And there's six times as fast each. And so instead of like this one truck that went back and forth, you got to build a whole transportation infrastructure to get all the brick and on and on and on. So that's what I mean by like, you have to just change the whole architecture. It's not like it's this one super fast brick layer. It's a different world. So it's the PCIE as much as the as the disk itself that makes the difference right we're not going through a sata or god forbid a sas right yeah yeah it is the closer proximity to the processor that gives the and and the the the greater bandwidth of data to disk that gives the greater capacity to to gain this sort of quantum shift in power right yeah so a lot of

times people ask me like oh my god you're building this giant data warehouse engine you know the fastest biggest ever you must use gpus all the time and no that's the answer we would we would if it made price performance sense because it's not what you find in these massive data analysis queries is it's not how much density of CPU you can get per terabyte, which is sometimes typically been the limiting factor in an older, smaller scale database. But it's all about flow, which is what you guys are talking about, which is you've got, like it's in a big system, we're talking about 256 parallel tasks, brick layers per drive going on 512 going on a thousand per drive. You can get 20 of these drives in a U. You get like, you know, a couple few, you know, a hallway, you know, full of racks full of these things.

Next thing you know, you have a million literal parallel tasks in flight. And that's at the coming on and off the IO of the drives. Well, now you better have your PCI lanes organized in such a way that they can handle all that. And then your bandwidth of flow from the PCI lanes in and out of memory on your CPUs, you know, into the L3 cache, you know, L2 cache all the way into the CPU and back out.

And by the way, it's not like there's one god giant CPU. There's no god box in here. It is, you've got tons of CPUs, you've got tons of PCI lanes. And so it's just this level of parallelization. You know, five years ago, you probably had two cores in your CPU back when you're architecting some of these other things or 10 or 20 years ago. Well, now you got 50 going on 100 going on 200.

So one of the biggest differences in the architecture is it's just parallel everywhere. And you have to make these flows. Like one of the things we had to build was like a flow control system. If anywhere in the system, something like slows down, I mean, it's full, everything, you know, like we can have completely empty memory and, you know, and a ton of memory in a server. And if you had to garbage collect for a second, oh, you're in trouble. Cause we're like,

there's so much data at the front door that you got to like cat, you know, queue up somewhere. So, for example, one of the decisions we made, which you have to make, and I think you know where I'm going, is you have to write this in C++. You can't write this in Java, you know. And, you know, back in the, you know, like a lot of other data warehouses, you can write in Java. That's fine. But like a second of garbage collection is like a giant, huge problem. Now, Chris, these SSD drives with flash translation and all this stuff, there's garbage collection going on in the drives occasionally.

Inside of the drives, as long as they don't expose it to us, they do all. I mean, these drives are they're full on computers. They got big old CPUs, and they're moving stuff around between. They're managing the load so that they don't wear out a sector and all that stuff. They do all that for us. We don't see any of that. What we see is… You see more or less the right I.O. and read I.O. without… We want to see 256 parallel tasks, and I think it's like 6,000 times a second is what they do.

That's their job, and they do it. They really do it. disseminating or dividing responsibility for these tasks is really the correct and organized approach. I understand that. Yep. And that's one of the things we pioneered at CleverSafe as well. So when we entered the industry, like us as the on-prem vendor, AWS S3 is the cloud vendor, and then Google as the giant customer. The three of us all had the same idea, which was instead of buying really expensive custom hardware for a reliable enterprise storage system at hyperscale, which had been the way for decades like the whole way that whole industry had ever been all three of us pioneered the idea that we could use the most price performant at that time desktop hard drives in like the cheapest chassis actually we collaborated on there was a manufacturer in china that like made these, that bent the metal for us for this chassis that we are all using, like the most price performant, everything you can imagine in the box. And yeah, it's going to fail 10 times more often than in an enterprise thing. But you know what?

If you put a layer of really smart software up at the top, it's going to make a level of price performance that you just can't touch. It's going to 10X it down from where the whole industry was. And so the same thing's true, you know, in what we're doing in the database world, which is this is the domain that's traditionally been like custom, crazy, expensive supercomputing. But thanks to PCIe drives being in your phone, thanks to high core count CPUs being like, that's an average middle of the road processor. That's not some fancy thing. Now you can do this and you can do this at a level of price performance, which is at least one 10th of what it was before. In some cases, more like a hundredth. In order to do this sort of, excuse me, parallelizing this trillion read query kind of thing.

I mean, how do you decide what goes on? So I'm assuming you've got a 1U kind of server with 20 NVMe drives. And if you need more, you build a rack. If you need more, you build multiple racks of these things. So how do you decide what goes where in this sort of environment? Well, you know, one of the things, you know, you still have to do at this scale and it'll be especially now because each customer generally is setting some kind of world record, you know, because it's the first time you've ever loaded data this fast. It's the first time you've ever loaded this kind of data this fast. The first time you ever run a query like that, this on this much data or whatever, it'll change over time as it becomes more mainstream. But right now it's still

pretty, um, kind of high end niche-y stuff. And so we have to, we, you know, we, we find is basically we go into a vertical market at a time and in each vertical market, we identify the use case. It's a perfect fit for us. So we look, we're the new company. We're not going to get a customer for 30% better. Forget it. No one's going to, you got to be three to 10 to 50 times better to break in. So we only focus on use cases where we know that's going to be true. So then, you know, what we do is we then have to find a lighthouse customer or two of the first ones that are going to do it because it's going to be a lot of work for them. And one of the things they they have to do is they have to tell us in some level of detail what they're doing. And it may seem easy, but it's not like we need like, you know, we really want to see the queries.

We're going to need to see the data, you know. Yeah. Yeah. And, you know, you have to find the right kind of customer and they have to have the data, you know, yeah, yeah. And, and, you know, you have to find the right kind of customer and they have to have the expertise. I mean, it's just not, these are not simple questions and, you know, we'll, that, that really helps us understand. So like early on, you really have to get a lot of expertise from your customers to help make this work. Well, these are all, like you say, world record types of scenarios, right? These have never been done before. A trillion row queries in seconds is insane, Chris. It's insane. But totally cool. Yeah, it's cool. And I understand how it can be

done. I understand that parallelization will help. But I mean, if you're, you know, the challenge might be if you're trying to structure how your data warehouse operates based on every specific vertical, it's going to be a support challenge, wouldn't it? Basically, we find we only focus on verticals where at least 10 customers need exactly the same thing. And those are the capabilities we're going to add. Um, you know, anything above that, that would be one off, you know, that we would often find a partner to do the professional services. We'll do a little bit of professional services, but we're not a professional services company. And, you know, those things exist. And, you know, this always happens. Like every time there's any kind of step function and functionality, whether it's networking or CPU or storage or whatever, you always have these characteristics. You often have these characteristics up front where it just takes a little more time for the first kind of implementers to kind of work through all those details, but eventually it just becomes mainstream. You know, there was a time when, I don't know, I don't know, geospatial analysis was just not that easy to do, but then it became pretty straightforward.

And so we support not only full-on ANSI compliant SQL, including all the really hard stuff that people either skip or avoid at hyperscale, like joins and count distincts and all the stuff that is hard. But then we also are now doing geospatial at hyperscale, which has traditionally been impossible. And then we're working our way through machine learning at hyperscale. So you could have a model that looks at 10 petabytes of data and you train it every 15 seconds. I'll give you an example of that.

Yeah, I'm curious. Give me your example. It's got to be Tesla or something. Telcos have policy models that will look at things like when to hand off and what areas of the network to avoid and things like that. When do they time out? And so those policy models are made with giant machine learning models every day. So at the end of day, there's a new model published. Maybe they added, I don't know, a router upgrade over here. And there's a new connection over there. And so the model updates. It takes them about eight hours to run the model and they update it every day.

Well, things happen in the world more frequently than once a day. There's a traffic accident. So there's a giant amount of cars in a new place or there's a public service emergency or some over eager backhoe operator cuts a my favorite cable somewhere yeah be much more useful if they could retrain that model every 15 seconds you know and then it would just like all the things would reroute the second there's a problem but to do it they have it's a giant you know machine learning model they have to run and you know that's a perfect example like eight hours versus 15 seconds would be pretty useful yeah yeah yeah but i'm assuming that got all the gpus they could possibly want in a telco type of environment so that's not the issue it's

getting the data to them and getting the data loaded that fast and then, you know, getting it to analyze. So one of the other things we do at Oceant that we didn't originally plan to do, actually, which is not only do we have to build this whole new database engine from scratch, but we found that we had to build a data transformation and loading engine that had these same properties. Because if you're, let's say that telco example, if a trillion new data points are coming in every 10 seconds, you need to get that data, and it's never clean data that comes in. So you gotta get it transformed, you gotta get it indexed, you gotta get a secondary indexed,

you gotta get it compressed, you got to get it indexed, you got to get a secondary index, you got to get it compressed, you got to get it reliable. And then you got to get it inside relational schema and do all that within seconds of real time of time equals zero, which we can do. While at the same time, you know, loading that in while regular queries are still going on at hyperscale without, you know, completely tanking their performance, because you're going to load data every second of every day. So we had to build that as well, the transformation and loading engine that has the same, you know, stateless, scalable, all that kind of stuff. You mentioned compression in there. And I saw on your website something about, you know, I'll call it space efficient copies that you guys are capable of doing. You want to talk a little bit about that, Chris?

Yeah. So at Cleversafe, we, you know, one of the things that we did was we brought the idea to the industry that a more effective way to make reliability instead of making copies was to use math. And math, thanks to Moore's's law does get cheaper every year so you could use an accelerating amount of math to make reliability year after year without having to make copies and so you know the cost savings would be tremendous i think the whole industry at this point has switched over to that because it's so compelling so we looked at that that same thing. It's a little different in the database realm as opposed to the storage realm. So here we're really just using parity, actually, two-dimensional parity like RAID. Works actually quite well. And so calculating two-dimensional parity and then doing that inside the database. So if you have 10 servers that are your database servers,

and if any two fail, you're fine. That you can do with parity, with RAID. And so we're doing it inside the database. And we're not aware of any other commercial database that's ever done that. And the advantage is that you get reliability as though you're making many, many copies, but the overhead, and we've also done stuff not only for the reliability overhead, we don't have to make copies, but we're very efficient in our creation of indexes and stats and all the other stuff you have in a database so that we can really have a complete expansion of less than 30%, sometimes less than 20% of original data versus reliable indexed, you know, stats available, all that kind of stuff physically when it's stored. And it's a giant cost saver. Oh yeah. Yeah. You would kind of imagine, imagine. Yeah. And then on top of that, if you do, then you do compression, you make it actually even smaller than the data started with. You mentioned you also, besides, I'm not exactly what terms are, but the data warehousing capabilities, you're also doing data transformation capabilities on these trillion row data structures and stuff like that. You want to talk a little bit about, so that, you know, when I see that sort of stuff, you're cleaning up the data for machine learning or you're, you're doing some processing or pre-processing the data for some geospatial types of workloads

and stuff, pipelines kind of things. Is that how this world plays out? Yeah. Kind of all of the above. I'm almost data never in these giant data sets, you know, they're either coming from routers or phones or cars. I mean, there's certain places these giant data sets come from. They never come to you in perfect relational schema. It's always some kind of semi-structured documents like an ad tech. For example, if you're, you know, an ad tech, there's 10 million digital auctions or so every second of every day. So every time anybody does anything on their browser or their phone, there's auctions that occur. Every time you click on a page, a bunch of little auctions are going to get triggered on who gets to place the ad. That person's going to see in about 500 milliseconds or 300 milliseconds. So there's ads and there's 10 million of these a second. And then there's all this data that goes with it, like information about the people that are clicking and who I want to buy and stuff like that. So those,

those auctions occur on these ad exchanges, these giant ad exchanges. And so there's all kinds of people that want to analyze that data. They, they're going to run a campaign. They're going to look at the last six months of ad exchange data. They're going to run back testing of their model, their campaign forecasting model, just like they do in FinTech with trading financial things. So they're doing all that stuff in ad tech. The way that data physically arrives from the exchanges are these like semi-structured JSON documents, you know, like 100,000 of them a second. So you've got to take 100,000 JSON documents a second, which are kind

of a mess, you know, from a database point of view, they're a mess. And so you got to like rip them apart and then get them inside a relational scheme. And by the way, a lot of the data structures you're using in the relational database are like these giant arrays, you know, so you got to get them in these really complex arrays. And then at the same time, like I said, you know, you're compressing, you're making an index, you're making a secondary index, you're encrypting, all this stuff is happening. And then we're able to do it to where it's within, you know, 100,000 JSON documents a second are coming within two, three, four seconds. That data is actually showing up in queries and all this stuff happened in the meantime. That's hard to do. And so then you go back to some of the stuff we were saying earlier,

well, how do you do that? Well, you have to crazy parallelize everything. So we have, basically, the loaders themselves are basically stateless, which is wonderful. We don't have to lock anything and you can just like spread data over any number of servers that you need. So you basically just look at like, how big is the flow? You think of it like a network, actually, you know, how many gigabits per second of flow is coming in or terabits per second of flow is coming in? How much can each of these servers handle? And then you just divide. And then that determines how many of those servers you need. And then inside the server design, we're doing the exact same sorts of things. Like here's how fast data is coming in off that network

card here's how fast it's going to move up into the cpu like you know here's how fast we got it you know it's just this pipeline and then our job as software engineers software architects and designers is if we've designed our stuff right we're going to go you know we call it entitlement so whenever we do designs we instrument up of the servers when we do testing and we see like how fast is data flowing through the L3 cache? How fast, how full are the PCI lanes? You know, how fast is the IO at the drive? And, you know, entitlement to us is like something like 85%. So if we're running at like 85% of how fast that physical thing can go,

all right, then we've done our job in software. Yeah, yeah, yeah. Not an easy task, I might add. And probably spending years doing this. You know, I noticed you guys are on like version 19 of your product. How long have you been shipping? That's about five days. No, it's not. Yeah, and the truth is we did have 18 other versions we

just talked about v20 this morning it's looking good it's coming out soon and the reality is like we we had to make that many versions the first one that went production i think was v18 and so we spent five years which we knew going in with a giant, expensive, talented dev team building this thing. But we knew we would not have, you know, version one in production until, you know, year five or so, which, you know, we were right on schedule. And version 18 really was the first. Maybe it was version 19. I can't remember. We did some pilot.

I mean, the first customers that went into production went into production April of last year. And I think we were on version 18 or 17 when they went into production. I also noticed you guys are, you have solutions that run in the cloud as well as on-prem. And you have your own cloud, if I realize this correctly. Yeah. So we're a software company. Like I've been talking about, we're really good at using the most price-performing hardware on the planet and squeezing every last ounce of capability out of every single part of that hardware. And we can run on-prem, so like HP, Supermicro, Dell hardware on-prem.

And then if the customer wants to buy their own hardware. And then we can run on cloud. So we run on, we just announced our partnership with Google. So we run on the Google Cloud platform. And so there's a couple different configurations of one of their boxes that's full of NVMe drives that we can run on. And then we also run on AWS as well. And we'd run on, you know, some giant customer set,

or you really did run on Azure, we'd do that or whatever other platform. So we're pretty portable. And then in addition, we do also offer it as a cloud. So if you want us to go buy the boxes and rent the data center, load our software up and then sell it as a service, we'll do that too. So as a service kind of thing. So, and that's sort of a solution, would you, God, how would you build such a monster? And this is a monster environment. I mean, you're talking. Well, for on-prem and on cloud, we charge by the core.

So our minimum pack is a 500 core pack. And that, a lot of that has to do with like, we're focused on kind of large systems and people that are just hammering away every day all day. So that's for them core base pricing is much more efficient. They're not paying by usage. You know, moving forward, that's like two servers. Maybe. Well, no, because what's the other side to that?

And over time, it will be. Over time, a trillion will be two servers. But you know what's going to happen over time is be quadrillion. 10 trillion, 100 trillion, quadrillion. The undeniable force driving this whole thing forward is the world generates an accelerating amount of data and it will get analyzed. You know, if you're the IT person at whatever company, no matter what, your data is always going like crazy and you don't ever get to say, you know, I'm just going to need an accelerating amount of budget in order to do this

work because it just keeps getting bigger. Like that never happens. And I'll give you guys a challenge and I've given this before. And so far no one's ever met the challenge is I challenge you to name something where the new version creates less data than the old version, because I can name everything else, whether it's a building or a phone or a plane or a car, I can name everything else. A telescope, a microscope, a cat scanner, a oil pressure monitor, a thermostat. Chris, Chris, Chris.

All right. You got one? It's only because of supply chain requirements, right? So they're, you know, and I'm not sure what vendor. It might be a car, okay? So a car because in the old days it was able to access, you know, LiDAR, ultrasound and camera and multiple CPUs. The only thing I can think of that might've gone less is because of supply

chain issues, they've reduced some of the sensor packages in the cars, but very unusual. And quite frankly, if it wasn't for supply chain, I'm sure it would have increased. So you're absolutely right. Yeah. And even if it has, that's temporary. Yeah. But I would still, even if they reduce the number of sensors, the ones that are there are making more data. So it'd be interesting if the net is that it went down. I've asked for this counterexample for like 10 years and so far no one's named one. Maybe that's the one.

No, that's the bloat that is this industry. Yeah, but there's 10,000 examples of everything else and there's no end in sight. And so Trillion Scale today is this top 500, top 1,000 data analyzing organizations in the world. It's just a matter of when that becomes mainstream. Well, this has been great. Matt, any last questions for Chris? No, I am a little blown away by what you've done here and I'd love to see it in action.

Don't know where I'd put it, but wow. Chris, anything you'd like to say to our listening audience before we close? Cool. It was fun. You know, Matt, your question makes me think like we do this. Oceant does a podcast called Big Bites with Oceant. Like we'll interview our customers. And, you know, if you're like a tech geek like me and you and others, like it's really cool what they're doing.

And it would be fun, you know, because even though it's a podcast, we have the video. It'd be fun to maybe do an actual demo. Because some of this stuff is just amazing that goes on. So I'll try and make that happen. Well, this has been great, Chris. Thank you very much for being on our show today. I really enjoyed it. It was great to catch up with you.

And that's it for now. Bye, Chris. See you. Bye, Matt. Bye, Ray. Until next time. Next time, we will talk to the system storage technology person. Any questions you want us to ask, please let us know.

And if you enjoy our podcast, tell your friends about it. Please review us on Apple Podcasts, Google Play, and Spotify, as this will help get the word out.