Signals and Threads - Building a data warehouse from scratch with Jacob Baskin

Episode Date: June 24, 2026

In university Jacob Baskin studied at the intersection of computer science and economics, thinking about systems that incentivize people to express their true preferences. He put those ideas into practice at Google, where he worked on ad serving, before joining Jane Street’s database infrastructure team. In this episode, Ron and Jacob discuss Superstore, a distributed columnar database now central to Jane Street’s tech stack that Jacob began building practically the day he started. How do you support wide-ranging analytical queries while transactional writes stream in at the speed of trading systems? And what’s it like when your first design doc leads to an eight-figure hardware purchase? After building Superstore Jacob has since gone back to his roots, thinking about schemes for bidding on compute time as he works to optimize usage of the Hive, Jane Street’s massive compute cluster for research. You can find the transcript for this episode on our website. Some links to topics that came up in the discussion: Mechanism design, second-price auction MapReduce, BigTable, Google File System Vertica Apache Parquet CockroachDB Paxos BitTorrent

Well, it's my pleasure to have Jacob Baskin here on Signals and Threads. Jacob's a software engineer here who has worked on a bunch of large-scale compute and data systems that we've built here, and we'll talk about a few of those. But first of all, just welcome. Thanks for joining me. Happy to be here. And I always like to hear a little bit about how people got started. Can you tell me a little bit more about how you got into computer science in the first place?

Well, so I've been coding since I was a little kid. My dad taught me basic back when I was seven. But then in college, I was actually not going to be a computer science major. I initially planned to be a psychology major, but I couldn't get away. I enjoyed coding too much, and there was too much cool stuff to work on. So I went to Brown University, and I ended up doing a bunch of stuff for my thesis around mechanism design and sort of the connection between economics and algorithms, which I found super interesting and had no particular belief that I'd ever end up using this in my career.

There's some really cool stuff where incorporating the economics makes finding efficient algorithms and finding good ways to actually do the computations within a reasonable amount of time much more difficult. I just say like a bit of jargon, like mechanism design is like weird econ jargon for designing markets, right? Yeah, well, markets are one kind of mechanism. I guess if you want to be technical about it, there's a lot of mechanisms that don't look like markets. Usually when people talk about mechanism design, they're talking about something that looks like an auction, where people have some number that's denominated in like dollars or whatever, that they're willing to pay for a certain thing or a certain collection of things.

And your job as a mechanism designer is getting everyone to save their amount of dollars without having to be strategic and underhanded in a way that lets you figure out a good, a socially good, so like something that will result in the most like actual value for real human beings, allocation of resources to the people who are contending for them. And then how does mechanism design connect into software engineering? Well, often the thing that you're trying to do is hard not just from a mechanism perspective. It's not just that different people want things in different amounts and you have to figure out who wants stuff more and who wants stuff less.

But actually figuring out the allocation is computationally difficult. So some classic problems in mechanism design, are things like building networks. If you want to connect a bunch of different people's houses to the internet, you have to literally go in the world and dig trenches and put fiber in trenches. And figuring out what are the right trenches to dig and what's the right fiber to put, even if you're the king of the world and get to dig whatever trenches you want and command the resources of everyone in the economy,

it still involves some algorithms and computation. but in a world, you actually have to, like, find contractors to do this and pay them, and, like, you know, Verizon is out there and AT&T is out there, and they want to do this in different, you know, in different places

and are willing to charge you different amounts, right? Then you get this combination of this algorithmically challenging problem, this economically challenging problem, and you have to kind of solve both at once, and they have interesting effects on each other. Sure, although I would like to point out that I asked you, how does it connect to software engineering,

and then you told me how it connected. Computer Science. The digging trenches. Yes. Fair enough. Okay. It connects to software because a lot of the problems

that we're trying to build software to solve are problems that are economically important where the ways that people interact with software are driven by their incentives and by their economic desires. Right. And obviously, you can totally see this stuff

playing out in design of exchanges, right? And markets and stuff like that. And there you see like a rich intertwining of what you do on the mechanism design and how people behave and how that affects the technology. So like, you know, despite you imagine, I'm imagining this wouldn't come up much in life, like here you are at a trading firm.

Right, here I am. It turns out that my interests are what they are and are somewhat consistent over the course of my life. But in fact, the way that I ended up dealing with this first at work involved advertising. My first job out of school is at Google. And I wound up working on Google's very large and very interesting advertising business, helping Google do a better job of showing ads on other people's websites. The system that I was working on interestingly enough was called ad exchange. So yet another market. Yet another market.

And this one was, in fact, a two-sided market, much like financial markets, because you had people with space who wanted to show ads. And then you had people who had ads and wanted to find space for them. And our job was to connect them. And we did, in fact, have to do some mechanism design to make the system work well. So, I mean, Google obviously, like, selling advertising is, like, very deep in that business. And in fact, like, early on it was, like, kind of the only way they made any money at all. Like where was this in the kind of evolution of Google's ad business? Like what had been happening in their business at the time?

So in 2008, which was in fact the year that I started at Google, they bought a company called DoubleClick, which is one of the original companies for doing banner ads. Back in the olden days in the internet, advertisements were much more about, like, you go to a website and there's like an image, and that image is an image that advertises something, and you have to pick which image to load when someone visits the website. And like that particular problem of loading the right image when a person visits the website and then also like keeping track of all of these, that was double-click spread and butter.

And Google was very good at showing ads on Google search, but not so good at showing ads on other people's websites. I mean, that was Google's had that business for a long time too. But particularly when it came to like sort of larger and fancier websites, double-click was the industry leader. So Google acquired them. And then they had this problem with, well, you know, can we do both? can we take the stuff that we're so good at when it comes to showing ads on Google.com, which was an auction forever. They wrote very interesting papers back when they were building the system

about how they did this auction mechanism for showing ads on Google. Can we combine that kind of auction mechanism with something that takes into account the needs and the desires of large web publishers? You know, people like newspapers or, you know, large, news operations, Huffington Post, you know, BuzzFeed.

This was the, you know, aughts, so those were large companies at the time. Right? So how do we adapt this auctione world

to one where there's all these publishers, but also where we don't know what all the ads are in the world, where in fact, there's all of these other companies, whether they're ad agencies or whether they're middlemen,

who understand the advertisers and what they want and want to integrate with our system in some reasonable way. Can you say a little bit more about like what the kind two sides of this, you know, two company combo was. Like, I can sort of get, I can sort of guess at what Google was good at here. Like, Google is really good at building infrastructure. And I bet

did a lot of good work on, like, you know, setting up the, the auctions who had like made sense and had coherent incentives and all of that. So what was double click bringing into the table? What was double click good at? Double click understood enterprise customers. I don't want to over generalize about Google in the mid-2000s. But there is certainly an ethos of we don't know why anyone would rather talk to a salesperson than use a WebUI. I mean, I do sympathize. I sympathize too. I would almost always rather use a WebUY than talk to a salesperson.

But there are definitely cases, especially when it's one company talking to another company, where first of all, having the attitude as a business that we really want to understand what it is that our customers are trying to do and why, rather than think that we should know better than them and tell them what we should try to do. And then also, like, have a much more managed, relationship with these people that's not just like, here you go,

enjoy your advertising system. So that was sort of where Double Click was coming in. They had built relationships with the very largest web publishers and the very largest advertisers. And, you know, obviously Google appreciated having those relationships,

but they also really appreciated having the DNA of, well, here's what it takes to serve large customers, which is just not something that Google did very much of at all in the early days. Got it. So then what was the kind of technical work that you were doing? Like you have these two companies that you need to integrate their businesses together. How does that turn into a technical problem? Well, so we were building all of this on Google infrastructure. So double-clicking Google had completely separate technical infrastructure. And pretty quickly the decision that, and this was in fact before I started, Google made the decision we were to throw out all the double-click stuff and build it all again on Google infrastructure. Because of course they did. Right.

So the challenge became sort of trying to take the insights that Double Click had about their business and combine them with Google's, you know, it's worth saying that at the time Google's infrastructure felt like it came from the future. Right. A lot of the stuff that Google built, right, and these days, you know, we just sort of live with it in the cloud. So we don't recognize after the fact how revolutionary it was back. in the mid-2000s. But things like MapReduce and Bigtable, which were Google's sort of distributed systems for storing and processing large amounts of data, right? They have this very large distributed file system called GFS that stored like many petabytes of data. You know, you could just

run applications over thousands of computers at once just by like typing in the right numbers into a config file. These things were not table stakes across the industry. Yeah, and I remember when those papers came out, they made a huge splash because it was, you know, in some sense, like the underlying techniques were not super new or surprising exactly, but they had sort implemented it at scale and gotten, you know, and there's a lot of hard things to go from, like, taking an idea that kind of makes sense to doing it really effectively at scale and had learned a lot from that experience. And like, it was actually great that they published those papers and I think it influenced the industry broadly a bunch. Yeah. So really the technical work, or one very hard

piece of technical work, was taking this Google infrastructure to the, that was, you know, again, very horizontally scalable, and combining it with these double-click insights about, you know, what is the feature set that you need to serve these large publishers and large advertisers? For ad exchange specifically, there's another very interesting technical piece, which is called real-time bidding. To integrate with the systems on the advertisers side, the things that actually know what ads people want to show,

rather than have them give us all of their ad inventory in advance and be able to, like, pick out of a database, we actually would query these systems across the internet for every single ad impression that they wanted to show and let them submit a bit in real time. Oh, wait. How does that actually work practically? Like, they have ads. Those ads have, like, relatively heavy assets, like images

and stuff that need to get delivered. And is it, like, literally just in time, like someone loads a webpage and it needs to find an ad, and it goes all the way end-to-end to the actual advertisers who's providing the ads and gets the image at the last moment? Is it really just in time the whole way through? Well, so it's actually just in time twice. So the first thing that happens is you say like,

hey, I want to show an ad here. And then you need people to bid. So that goes through Google systems to look at all of the people who sort of have bids. They have some like filter criteria, right? We don't call everyone for every single time Google shows them out on the internet. But there's some filter criteria.

We choose which advertisers call. We say like, hey, here's information about this advertising spot on this website. You know, what's your bid? and then they return us a bid, and then if they win, we inject a little HTML snippet into the ad unit on the web,

on the web page. That goes to their system and fetches the image. Got it. So then the final asset is not actually flowing through Google systems. That's just being like directly loaded from the final ad ad.

Exactly. With a lot of asterisks around, this is all very much integrated with all of the crazy ad tracking pixels and cookies stuff that got, built in the internet around that time. So really the load on pages for loading ads, I mean, these days as well, but also in those days, was incredibly heavy just in terms of all of the

various tracking cookies and pixels and different domains that provided all of this different stuff to do metrics and monitoring for the ad. That ended up being much worse than the image itself. Got it. And then also this whole infrastructure is also probably like kind of a security disaster, although maybe in a way that wasn't as obvious back then. Definitely all of the tracking and cookie stuff is absolutely a security and privacy mess. But we did, in fact, think about this. And one of the things that we built was a way for advertisers to link what they knew about a given user with a particular ad impression without us being able to extend their reach to that user across the internet. We would anonymize users' identifiers in a way that didn't give advertisers

access to them, but it still let them know if they had seen that user before. Got it. So I guess that sounds like part of the security problem. I guess the other security problem is around the like delivery of weird malware through JavaScript nonsense. So yeah, once, yes. Yes. So Google also had had pretty reasonable controls for that. That can be very difficult. Like that was a real arms race with malware actors. But at least we tried very hard. But yes, certainly once you get to put arbitrary HTML inside a user's browser, it's a pretty fraught scenario. And certainly at the time, browsers didn't have any particularly good sandboxing abilities for managing this well. Right. Luckily, when this all started, it was like less important. And then we had some time to spend time making it better as it got more central to the world. Yeah, I think that's fair. It's certainly very important to the publishers at the time. Like, this was, you know, if you remember back in the day, the New York Times didn't have a paywall. Oh, right. Right. You just so much journalism on the internet and just content more generally on the internet was ad-support. And this is one of the exciting things about working in that industry at the time is we were like on a mission to make the ad supported internet actually work for people. Now we failed, but we tried really hard.

So you studied something about mechanism design in school and then here you were like right in the middle of designing these mechanisms. Like how well did the lessons that you learned in school like pan out? Like what do you feel like you learned from the whole process of trying to do this stuff with like a real system? So the main message that you get, that I got at the time from the sort of academic community around mechanism design, was that building incentive compatible mechanisms was extremely valuable. That if you build the mechanism right, it benefits people to tell you the truth. And if you build mechanisms wrong, then people are going to try and game the system, and this will end up resulting in a worse situation than you otherwise would have. And I think the main thing that I learned in industry was that that's definitely a valuable thing that's nice to do, but that is a much smaller part of the problem than I believed it was back when I was in school. So what's the rest of the problem? So two big things that you have to focus on that are not covered in that view of mechanism design.

First of all, people need to understand the mechanism. It's no good if you can give them a complicated proof that the mechanism is incentive. compatible if they don't have any idea what the heck is going on. So being able to point to, well, here's why, you know, this good thing happened to you. Here's why this bad thing happened to you. Here's why we charge you this amount, right, without having to like get a whiteboard out is really important when you're actually trying to sell something to people who are not mechanism design PhDs.

Can you give an example of like a part of the technical design that was done to achieve that goal of making the results more explainable to people? Okay, well, so a great example is budgets. You know, if I'm an advertiser and I want to go and show ads on websites, probably I don't want to spend infinite amounts of money, even if I can get really good prices all day. Sure. Probably I want to spend, you know, a thousand bucks, maybe a thousand bucks a day.

You want some risk limits. Yeah, I want some risk limits. Fantastic. Right. So let's say that I go and I start spending my budget and I spend it really quickly, and then I have like five bucks left, and it's getting towards the end of the day. What should I do to give myself better outcomes?

I should probably lower my bid, right? Because the chances are that I have been spending too fast and I could get more of the good stuff that I want for less money if I just bid less for it. Okay. And this is not really an incentive-compatible result. Because another way of saying that is if I have a budget, then I should bid less than my marginal value of each thing.

in order to get the best results for me. Right. So budgets break sort of a very naively, memoryless incentive-compatible auction all the time. And yet, very few people try and model this for their market participants. In almost all the cases, what you'll see is that people offer, maybe they have some kind of second-price auction, maybe not even, but nobody tries to take into account people's budgets

when figuring out how to allocate them stuff. What do you mean, like, in the academic literature, nobody thinks about budgets? Nope. Or do you mean in real world systems? In real world systems, people let you manage your budget. They don't try and manage your budget for you. And they will even maybe say like, hey, you should bid less so that your budget less all day. Right?

And rather than trying to model budgets as part of the mechanism, they just let it be this other thing that people have to manage on their own. Because when we try and incorporate people's budgets for them in some way, they just get results that they. don't understand. And in fact, people have a lot of constraints that they're not telling you, that you're not really in a good place to know, and you just kind of have to let them do their thing and be sort of incentive-compatible in a narrower sense, if you can, without really trying to be fully incentive-compatible in some theoretical sense. So, like, simplicity is a value. Simplicity is a value. So Google was obviously an interesting place to learn about, you know, all of this kind of market

stuff around ads. But there's also a lot of interesting technical stuff happening there, as you were mentioning. What do you feel like you learned from the experience of being a kind of systems engineer and systems designer in that world? Just so much. I feel like I'm still mining a lot of my early Google experience for, you know, software engineering insights. One of the main things that I learned was kind of how to think about distributed systems in ways that let you design them well. The Google approach to building distributed systems is to try and make the hard part be as contained as possible. You take some existing system that does a very hard distributed system thing, rather that's distributed consensus, or whether that's like building a large

stateful key value store, and then you build your system in terms of that existing thing. And often this is sort of a nested process where like Big Table, which is this great distributed key value store, is built on top of Chubby, which is this distributed consensus mechanism. and like most engineers at Google are not building big table or chubby, they're building something on top of both of them. And in particular, building large stateless systems is much easier than building large stateful systems. So you really want to contain the state of your system in as few different components as you can.

Right. Although I find it's always kind of funny when people talk about building stateless systems. Like often there is state there. It's just like not being managed by you, right? And so it's not so much about the system you're building. being stateless, but like the component that you're adding on, not being responsible for managing this data. Almost every system is like stateful in some sense, or what is it even doing?

Right. Like maybe you're building, you know, a data center of a million pocket calculators, but otherwise you're pretty stable. But yeah, again, the goal is having as few of the components be stateful as you can, right? So a very Google approach to this kind of problem is like, well, we'll have this, you know, stateful core that like goes and gets the data and loads it into memory and some extremely efficient form. And then we'll just have a bazillion web servers that I'll go talk to it. And we'll push as much of the work out to these bazillion web servers as we can.

And then they'll make what are hopefully very efficient queries to the thing maintaining the state. And that's what the system will look like. This does not look a lot like the way that trading systems work, where you end up keeping the state very close to the edge because you prioritize latency at levels that Google just didn't. Right. Like, our latency target for ad serving was 250 milliseconds. Right, which, you know, by some measures is kind of fast. Yeah, it's fast enough that you're not going to go and get a cup of coffee waiting for the website to load, which is really the important thing. Right.

But it's, like, very, very different from the kind of latencies that we worry about when we are, when we were a building system. And so you can do a lot of things with a 250 millisecond latency budget that you can't, if your latency budget is in the microseconds. Right. Yeah. And people often talk about technical organizations pointing out that like the technical, the technical organizations are often kind of shaped like their initial application. Yes. And Google's early infrastructure in many ways was built around search.

And ours was built very much around trading systems. And that's led to some very different choices. And in ads at Google, we were 100% using components that were initially developed for Google search. Right. Google looked at advertising. This was before my time. Google looked at advertising as just a search problem. You know, our problem is like we have to index all of the ads in the world and find

the best one to show for a given website. Yeah, if you squint hard enough, sure. Okay, so you learned a bunch of this fun stuff at Google. How did you end up here? So from Google, I ended up going and doing a startup focused on helping cities manage their curbs. What do you mean manage their curbs? So if you walk down the sidewalk, certainly in New York, but also in really just about any city in the world, there will be all kinds of arcane paint markings,

and signs with icons on them and so forth, denoting if someone pulls a vehicle over to the side of this particular street, what are they allowed to do and what is going to get them a ticket? Maybe you're allowed to park, but do you have to pay? Can you park for four hours or can you only park for one hour? Maybe you can't park, but if you're a truck, you can load and unload goods from the truck. Maybe you can't unload goods, but you can unload passengers, but only for three minutes. All of these rules have sort of been built up over time and accrued in the

these cities and almost nobody has a good record of what they are and why. So we were trying to solve this problem and build a platform for cities to manage both the actual physical control infrastructure that causes these rules to take effect on a given street and then also better be able to understand and analyze what the rules even were. How many parking spots are there in New York? This is a harder question to answer than one might think, especially because it changes day a day and hour to hour. Wait, it changes data,

how does it change from hour to hour? Well, there's a sign. This is no stopping 4 p.m. to 6 p.m. Monday to Friday. Okay. And then how is, and how is this company going to solve those problems? One of the interesting things about parking signs

is that exactly where they are matters to like plus or minus one parking spot, which is a little tighter than you can do reliably in urban areas with GPS. Okay. So one piece of infrastructure we had was an app that used, sort of AR-inspired visual odometry techniques to like figure out exactly where along the street different things like signs were. Wait, AR augmented reality.

Yes. Like, what? If you play like Pokemon Go, you know, you lift up your phone and like there on the table is sitting a Pokemon and you move your phone around and the Pokemon stays where they are relative to the things in the scene, even if your phone moves. Okay.

So to do that, you have to build a map of the scene inside your phone. Sure. If you can build a map of a scene inside your phone, and you can build a map of a block inside your phone. Okay. And then we can use those to position the parking signs along the block with extremely high accuracy. Nice. Okay. Fun. So like that data collection problem is actually really fun to solve.

And then once you have the data of like, well, where are the parking signs? Well, then you have to figure out what they say, right, which is a transcription problem. We didn't have, you know, clawed back in the day. We had to figure out how to do this, you know, using slightly less general techniques. Sure. And then even once you know what, they say you have to know what that means. Right. And, you know, parking signs have this language that's sort of all their own.

I would think what advantage is like the vocabulary is relatively small? The vocabulary is relatively small. We ended up essentially hand-building parsers for this. But it's actually quite tricky because this is designed to be sort of easily looked at by humans who know that particular city very well. So the vocabulary varies city to city. and also things like the position of different numbers and words on the sign looks very different than like the linear reading of a sentence of text on a computer.

You'll have like there's a big two over here. And then somewhere else on the sign it'll say maximum hours parking. Right. And do you connect those? Well, like, yes, as a human, we're very good at using spatial context to put these things together. But figuring out a way to model this in a computer can be quite challenging. So this sounds like a big pile of like fun, grotty engineering problems and a bunch of like weird domain specific knowledge that you need to make

this thing work. How did this work out as a business? Not so well. Not so well, Ron. I love this problem to death. I still wish this data existed in the world, but we were not really set up very well to turn this into a business. Okay. That didn't so much work out. And then I was saying, well, okay, what do I do next? I knew I didn't want to go back to Google. Partly just kind of been there, done that. Partly the company had gotten much, much bigger in the time that I was there and in the time since I was there. And I was looking for something that I could do in New York. I love living in New York. I didn't want to move. Back to the being enough of an urbanist to want to do a startup around cities. Exactly. And, you know, I loved this kind of economic thinking that I was doing back in the,

you know, earlier in my career. And I knew, in fact, a bunch of people who were doing finance-y-type stuff in New York. And so I thought, well, where do I do this? You know, what's the right place for me to, like, put all these things together? Jane Street was a super obvious choice for me, partly because almost all the people I knew who were working in the financial sphere were, in fact, at James Ford already. These are people who I'd worked with at university. These are people I knew from Google. These were, like, random people I knew socially.

It was a bunch of people who I thought were extremely smart, who I really respected, and they all managed to wind up at this place. And you're kind of like, hmm, I wonder how that happened. I mean, also I had done some functional programming in college. And like even back in 2008, already in the back of my head, it was, oh, Jane Street's a place you can go and write O'Camel. Like, oh, that sounds pretty fun. So our odd taste in programming languages strikes again. Absolutely.

And I had in fact written some O'Cammell to do some like combinatorial optimization problem solving for a class back in university. Not that that came in that handy when I started here. Like the O'Camel that Jane Street write, This actually looks pretty different than the camel that I wrote, you know, 15 years ago. But, yeah, I'd known about the place, and it seemed really cool. And so I thought, you know, want to I interview with them and see what there is. Cool.

Okay. So when you got here, like, what did you find yourself doing? Like, what part of the organization did you land in? So one of the things that Jane Street does before you start is there's this team match process where you go and meet with people, you know, who, you know, seem like teams that you might want to work on. and you chat with them. So one of the teams that I did a team match with was this team called database infrastructure.

And I talked to this guy, Sam, who talked about, well, Jane Street runs all these Postgres databases. Some of them are really big, right? And we're having trouble managing them. And like, we just sort of let people put whatever data in there and it doesn't always go very well. So we really want to make this work better.

And I'm like, okay, well, that definitely sounds like a hard problem. But why are you trying to do this? Why don't you have some kind of, you know, distributed analytical database? that can query over all of this different data without it having to live on a single computer that runs Postgres. And I was like, yeah, you know,

that kind of does seem like something we've thought about. We haven't built it yet, but it seems like that would be a good direction to at least explore. And I'm like, okay, I'm very happy to work on these database problems, but just so you know, if I come and work on your team, I'm going to try and build this data warehouse analytics thing.

So can we like dig in a little bit more to the like, the kind of like state ex ante? you like, what is wrong with you, like, throw a lot of data into a Postgres database? Like, isn't that what databases are for, is throwing a lot of data into them? Like, how are you using them? And what, like, what was weird about that use from your perspective? So Postgres is a great database. But it has a lot of different goals that it's trying to meet.

One of the things that Postgres is trying to do is be able to do really fast, you know, single row or few row transactions. Think about, like, running a store. You know, you want to sell, someone clicks a checkout button on your website. And they go and buy a widget. and you want to like both record this order that someone bought this widget, you want to decrease the number of widgets in your inventory by one, and you want this to all happen while the page is loading in a very short amount of time,

and you want it to all succeed autonomically. Postgres has to solve that problem. There's this other set of problems, which we kind of call sort of analytical processing, which are about, I have a lot of data, and I want to answer questions about all of this data altogether in aggregate in some way, that could involve not just single rows, but like many millions of rows of data. Postgres can do this for you as well, but it's much slower at it, because it's constrained by needing to store this data in a way that lets it do these

atomic transactions over very few rows. It's like, for example, Postgres maintains indexes, right? You have all this data in a table, and Postgres can look up this data for you by potentially a lot of different things, a lot of different potential keys into this piece of data. But that means that all these indexes have to live on. disk. And also, Postgres can very efficiently update individual rows. And if you think about it, what this means is that Postgres can't really compress across rows. Because if I have to be able to change that one bit on that one row, if that bit is compressed into like some, you know,

50 kilobite or one megabyte chunk, well, then every time I change a row, I got to write that 50 kilobytes or that one megabyte of data back to disk. And that's just going to be a non-starter for these transactional use cases. So Postgres does as good of a job as it can. And given the constraints that it has. But ultimately, when you're doing these analytical processing use cases, you really want to lay out the actual data at rest in a very, very different way. Right.

And this kind of goes back to this is like classic distinction between row-oriented databases and column-oriented databases, which has always struck me as super weird. Like, the idea is like, I would like to get a piece of software for managing my data. It's like, wait, do you want a row-oriented database or a column-oriented database? And I'm like, I mean, I have to pick my data. structure now? Like, why can't I pick it on a table by table basis? Why can't I, like, take my data and represent it multiple different ways so that I can do both this kind of access

and that kind of access efficiently, kind of depending on the particular kind of thing I want to do? And a lot of people try and build systems that do both. But what we found is that the abstractions that make columnar databases efficient kind of leak through the interface layer. like you can go out and find right now a lot of SQL compliant, whatever that means, different things to different people, databases that have columnar storage. In fact, at Jane Street, we were using one at the time that I came and made my bold claim, which was called Vertica, which is an existing columnar database. It's in fact very efficient for a lot of these sorts of analytical problems.

What we found with Vertica is that because it tried to give you these full SQL transactional semantics, it was very easy to break. A lot of the things that you could do with SQL that Postgres can support very efficiently, you know, trying to build that transactional layer over underlying columnar data just gets rid of a lot of the advantages that you're trying to get in the first place. And the way that this gets implemented with a lot of these systems

just dramatically slows down the queries and can even make the database unusable. Now, what most... SQL is like the blessing and curse of the database industry. Exactly. Right? There's this language defined like what, in the 70s, I think, that somehow we're all still using more or less the same language today.

And, like, I guess the problem you're pointing at here is like a kind of over-expressiveness problem. Like, SQL can do, like, a lot of different things. But the implementation you put under it, maybe we'll do some of those things really well and some of those things really badly. And when you give the full API surface to people, they'll maybe try and do all the things that you tell them that you can do. And it's going to be a bad time. Now, the way that I'd put that is that SQL does such a good job of being what it is, that it doesn't feel like programming. If someone told you like, oh, C++ is way too expressive,

it lets you like do whatever you want to do. And like sometimes people do bad things with C++ plus and that means it's a mistake. Like, no. God knows, C++ is plenty of faults. But you're not going to say, oh, it's too expressive. It lets you do too many different things. If people treated the SQL that they're right as if they were like writing a software system when they were constructing their queries,

I think we wouldn't mind that SQL can do all these different things and it has leak abstractions and et cetera, et cetera. This is all stuff that as software engineers we take for granted. But SQL makes it so natural to write queries that you just feel like you're asking a question of the system. You don't feel like you're writing a program. Even though what a database's query planner actually is is a compiler that generates code to run your SQL query as a program, it doesn't feel like that's what it's doing to us because SQL is such a good abstraction. But I think it also goes back to like almost like the mechanism design point you're making before of like,

sometimes it's more important to be explainable. And like the key thing that SQL trades away is explainability, right? Which is like you do put in a SQL query and like, and then it's optimized. And then like a miracle happens or maybe the miracle doesn't happen, right? And like depending on, you know, not just like the shape of the tables you run it on, but like the statistics that were gathered from that table, it will like optimize in different ways. It can be very hard to predict what the behavior is.

Right. So you're not sort of not used to as a user who's sitting there right. Whereas we can all simulate our. compilers in our head and understand exactly what they do to our O'Camel code. I mean, I think a lot of the last 10 years or 15 years even maybe of CPU design has been designed to give people the illusion that they still understand what the CPU is doing to their code when in fact they have no frigate idea what makes it fast. I mean, I think what you're saying is obviously true in the sense that like if you want to understand what's actually happening in a program

that you write in OCam or C or whatever, like, and you really want to sound it in a very low, level of detail, God help you, right? Like, there's an enormous amount of complex things happening at many different levels and lots of optimizations that are in there that try to make it fast for you, even when you haven't, like, done all the careful work for yourself. But I think there's a difference of degree, right? Like, there are deep algorithmic, like big O notation changes to the execution of a SQL query, which makes it dramatically harder to predict what it's going to do just from the query alone, then you can from looking at a program in almost any ordinary language. Right. This is true. Very few compilers will, like, change the big O complexity of your code

up from a... Yeah. Right. Whereas, like, the decisions of the query optimizer for SQL query engine can often be extremely load-bearing. That's 100% true. When you say, like, we were, like, throwing lots of data and creating lots of queries and stuff, like, what for, like, what was the use case that you were running into that we were all kind of running into and how we were using databases? Also, the funny thing is that this was, in many ways, just kind of emergent from lots of different people's connected use of these databases. It's like a great example. It's like a list of all the trades that Jane Street has done every day. Like you can totally imagine why that's valuable.

Like, okay, let's take some information that I have in my trading. Let's like combine that with all the trades that Jane Street is doing and like derive some insight from this. Right. Super valuable, right? Let's understand, you know, things about the fees that we're paying exchanges or like the positions that we keep in our different. different accounts. Or the symbology. Or the the different symbols. Yeah. Like, yeah, like, what markets does this stock trade on, what currency is it

denominated and stuff like that? Right. And part of what gets surfaced here is like, part of the value here is like database as ecosystem, right? The set of things that you can join together, like every new thing that you add to that increases the value of the platform. And like the key thing about joining the data behavior is literally join, like SQL joins. Exactly. The essential data operation for bringing this data together. And many companies, when faced with these problems with transactional databases, end up essentially getting rid of a lot of the flexibility and a lot of the usability of these systems in order to maintain the scale. They will have like a schema council that like,

before you're allowed to add a table to the database, you have to like have these people sit in judgment on you and like find a schema, right? You'll end up with databases where the only queries you're allowed to run are ones that have been code reviewed. Right. And in fact, often what you'll have is you'll have code that sits in front of the database and like, you know, you can't just like use the database directly. You go to the program that like you, you know, type something into a GUI and it goes off and constructs the queries. And so you don't have any kind of like free reign and just like running whatever queries that you want. But we are getting huge value from traders, not just traders, but among other people, traders, being able to do more or less whatever they want on these databases.

And we were putting in a ton of work on maintaining the database in such ways to give traders the illusion that they could do whatever they want. Right? There's, like, don't get me wrong. Postgres is a very good database, but there's a lot of things that you can do once you have hundreds of users on a Postgres database that will slow down and kill the database for everyone. Long-running transactions being the best example.

So rather than just like not let people run transactions on the shared database, we had like a long-running transaction killer that would go around and kill long-running transactions if they got too long. So can you see more about this? Like, what is wrong with it? Like, why are long-running transactions bad? The way that Postgres does animicity is essentially every time you have a transaction that writes to a table, it sort of forks the table and, like, keeps it in two states, the state like before your transaction and the state after your transaction. And then once you commit, it then has to like do essentially almost like a version control merge.

So that on the face of it sounds insanely expensive. When you say like forks the table, do you mean like makes a whole copy of the table? That would be bad. It's all copy on right, right? So like it only forks the pieces of the table that you touch. So it's both copy on write and it's copy on right in a kind of segmented way. Exactly. Got it.

Okay, but so. So then why is that bad? Like, you know, we, like a nice analogy for this is like version control when you like get or whatever, you can have lots of forks. And actually it's like kind of okay to have lots of for you. In fact, have lots of long running forks. And that doesn't necessarily cause other things to fall apart. Like, why do you think, why is it, why is it problematic to have these long running transactions

in the database context? The way the transaction semantics that databases give people aren't like, okay, you just get to, you like begin transaction starts a fork and you work in that fork and then later on you have to merge. So the way that the semantics of transactions in SQL and thus in Postgres is that while a transaction is in flight, it's effectively locking all the rows that it's changing. Not necessarily every row that it's reading, but every row that is writing to, it gets effectively a lock. this is to stop, you know, merge health from happening when transactions commit, which turns out to not be what most people want out of their databases. Right? So instead of letting you do whatever you want and then merging, we assume that if you're touching something,

you want to lock it until your transaction is done and then the next person gets to run on that. But what that means is that long-running transactions can accrue a large amount of locks and a large amount of data that we don't know if it's the future or not. And this can end up effective. costing a lot of storage and a lot of compute time on the database for having to take this into account and do all the bookkeeping for this transaction while it's open. Once a transaction gets closed, you just can very quickly garbage collect all of the past stuff that happened before this transaction. Is the main issue there that if you're acquiring a bunch of blocks,

that you're basically like running the issues of performance isolation? Like this long-running transaction blocks other transactions from completing because maybe somebody else wanted to write to that, And now they have to be pushed later in time. That's definitely, that's certainly the brunt of the problem. There's other ways that this interacts with like replication, for instance, and like the ability to do what Postgres calls vacuuming, which is like cleaning up old data in a table that were long-running transactions make that hard too. But yeah, certainly this ability of this chain of long-running transactions

is one of the things that makes us difficult for Postgres to manage performably. Okay. So the state of the world is we have this like incredibly valuable, not just a single database, but a set of databases that are being used in this kind of like somewhat wild and undisciplined way where like lots of people are pretty freely throwing in new kinds of data,

throwing new SQL transactions at it, and like relatively routinely running into exciting performance problems that you have a bunch of people who are doing a lot of operational work to try and keep this Frankenstein monstrosity like running efficiently. And just another big part of the problem is that Postgres very inherently has to keep all of your data on a single. machine. So we had, I think at the time, it was like the largest computer that Jane Street owned. I was running Traynor DV. So what are our requirements when we're trying to replace the system?

Like, what are the things that we want out of the new system? Well, we still want it to be the Wild West. We still want people to feel like they can do anything. We absolutely want this to be able to store more data that you can fit on a computer. And ideally, we want this to be, you know, much more compressible and much better at doing sort of large reads over arbitrarily large amounts of data than post-rests. Right. And essentially you want to move from something that is focused on transactions to things that are focused on like big read queries that get lots of data and figure out the results of complex computations.

Exactly. And we also want it to be much harder for people to break isolation. We want it to be much harder for me to go and do some crazy thing on the system that will impact your ability to use it. Right. Both from a correctness perspective and also from a performance perspective. Like, it should be much harder for a single user to bork the whole system for everyone. Do you have problems? I would have thought you'd have lots of performance isolation problems in something like Postgres and very few correctness problems.

Well, you had deadlocks, was sort of the main ways. I see. Is it a deadlock a performance problem? Well, in a sense. It's a very, very bad performance problem. Got it. And inherently, when you're grabbing lots of locks. Actually, how do you deal with deadlocks in the Postgres world?

So what we had is we had a deadlock detector. And if he's got deadlock, we would just kill them. makes sense. So this is what we wanted. There's no free lunch, right? As I said, we're not going to just like do Postgres but better because we think Postgres is pretty good. So what can we give up?

Well, it would be really nice if we could give up transactions. Sure, although transactions are really nice. Actions are really nice. It is nice to build a reason consistently about the data in your database. It sure is. So what if we couldn't, is there a way that we could avoid giving up transactions entirely? while still avoiding the kinds of problems that transactions caused on Postgres.

So this ended up being a lot of the meat of what we tried to build when we were specking out our data warehouse. And we looked at a lot of the existing systems around this. A lot of them either give up transactions entirely. You know, you have something like Kafka, say, which Kafka isn't a database. Yeah, I know you don't think of Kafka as a database.

But it is a system that stores a lot of data, right? For sure. If you look at Kafka, like, it doesn't, it doesn't like have read modify right. Right. To the degree that it has any notion of consistency, it's the ordering guarantees that are provided. And those are very narrowly scoped. Yes. Right.

And, like, you know, there's like the amount of, you know, data that you can fit inside of, like, a single partition of a single topic. Like, that's the scope or which you have well-defined ordering. Yeah. Or you have something like Redis, for instance, which has, like, very, which has no guarantees, essentially across keys. Okay. Yeah. another way of limiting the scope. Exactly.

Which is like totally not good enough. If you're trying to do some like complex join across multiple tables where you need consistency across those things, it's like you definitely need to cross keys. Yeah. And so on the flip side, you have something like Vertica, which tries to provide full transactional semantics. But like if you use it just right, you can make it efficient. Which again, we still want this to be the Wild West. So we want something that you don't need to use just right and it'll still work well. Okay.

So what we ended up landing on is something in the middle. In particular, we wanted to make it possible to do consistent rights to a single table because this is something that comes up a lot, is I want to have some, like, log of a system's behavior. Like, or going back to the list of all the trades that we do. You really want to not miss any trades. You really want to not double count any trades. But you don't necessarily need to insert new trades transactionally with other pieces of data, because like either the trade happens or it doesn't, and you have some sort of.

of truth for that. So you can always just kind of, you know, pull in new trades from the source of truth to keep this up to date. That's right. And also like the, to the degree that you're making changes, most of the changes are appends. Exactly. You're mostly adding new data. And so a lot of the cross table stuff that happens is like, it's not quite consistent for free, but it's like mostly consistent in a sense that like you might be missing data, but the data is mostly not being changed under your feet. Yeah. So we wanted to have data that you could. could atomically write to a single table, you could preserve, get order preserving rights to a single table, you could essentially get all of the guarantees that you want for that table, but not between tables. Another key thing that we gave up, so one of the big problems with columnar databases is that rewriting stuff is really annoying. So the way that columner databases

store their data is every column becomes its own separate stream of data. And the reason you do that is because a single column of data turns out to be extremely easy to compress and manage. The other reason you do this is because most queries don't hit all the columns in a table, especially analytical queries. You have these very wide tables. And maybe a given query needs like seven out of 100 columns. And you just can ignore the other 93. And so this makes adding new columns to a table nearly free at query time, which is super cool.

Sure. So given this layout, doing like an update of a single row, something that is in PostScript, extremely easy and efficient, becomes very difficult. Because first of all, all these columns are their own separate things. And second of all, they're all compressed. But again, we didn't want to give up the ability to, like, update and delete data. Because it turns out that being able to do that is really important. Right.

Even though it's by far not the ordinary thing. Most of you stream it in. Sometimes you get it wrong. Exactly. You would like to go and edit the table. So we want to have these be able to happen, but we're sort of okay with them being slow as long as they don't slow down other stuff that's going on in the database. So what we decided to do, and this is kind of a weird decision, is we decided to make all rights in this database

asynchronous. Now, you write to this table, and your right gets committed, right, which is the database says, yes, I acknowledge you're right, this is going to go in in the correct order, this is going to go in atomically if you ask it to be atomic and so forth. But readers aren't necessarily going be able to see it just yet. Right. And this lets us take the time in between when you do the right and when it becomes readable to make sure that the data that you're writing gets readable efficiently, right? That we like put it in the right place in the call in your data store, that we redo the compression, if we have to read, do the compression, and so forth and so on. And we're much happier doing that neither when someone's waiting for their read to complete,

nor when someone's waiting for their right to commit, but like somewhere in the middle. right? And this is very different from doing updates to a database via SQL, where SQL is a kind of part of the language and you're kind of like just mixing together the reads and the rights into one set of SQL expressions and they kind of all are supposed to operate in this kind of like integrated way. And here you're kind of like breaking out the rights essentially as a totally separate API that has different semantics and where the kind of results of those rights happen asynchronously. Exactly. And part of, you will not see very many commercial or open source systems that make this decision, just because, I think mostly because you have to educate your users every single time. But this is one of the cool things about being at Jane Street where all of our users are in house is we just get to tell them how it works and why. And like we can even build the first few examples ourselves to give them examples of how this works really nicely. And also very importantly, we can have. have, even if SQL is like the first-class read interface to this system, it doesn't need to be the first-class right interface. Because one thing that we noticed looking at TraderDB, for instance, is that even though a lot of the read queries were extremely ad hoc and just like written by hand by a human being, most of the rights were happening in much better defined ways through, like, actual software systems. Right. Maybe this goes back to like its role as an analytical database.

the place where the innovation is happening is on the query side, on the read side. And there's data loading. You got to do the data loading. But the data loading is kind of done once per data source. And that's kind of that. Often. Now, we didn't want to stop people from writing ad hoc data to the database if they wanted to.

Like, a very common case is like I have this shared data that's very big and important and I want to query it. But I want to join it with some other thing that nobody else has ever heard of. So you want to make it easy to upload your like little join table to the database, to join with the big existing table. Right? We want to have ad hoc data. We don't necessarily need to have that have like full SQL semantics.

You just like upload the whole table at once. And like if that's fast enough, then if you want to change it, I'll just re-upload it. Got it. So you sort of avoid the whole like writing an individual row by just like not having that. Exactly. Like not a thing in the system. So another thing that's interesting about this system, it's called Super Sure, is that another thing that.

Another thing that we, there's various things that we have, like, not given people who wanted from it. In particular, just to like say a little more about the underlying design, like the underlying data is actually stored in parquet files. Parquet is like lovely open source format for storing columnar data. And a request that people have put out a lot, which has always been denied, is like, oh, can I just like access the parquet files directly? Right. You have all these nice parquet files distributed in this nice distributed system. can I just go and read them directly and do something whatever I want to them? And like the answer from the Superstrip team has been no.

I'm curious like, why is that? So in some sense it seems like a very natural decision to make because, you know, if you look at, again, Postgres, can you just go and read the files on disk that Postgres uses to storage tables? No, that would be crazy. You have a database for that. So, right, if you think about it as like, well, we're building a software system and offering it to the firm, it's a very natural decision. But the funny thing is how little software that Jane Street develops actually

works like that. People are very smart and good at programming and love looking under the hood and understanding what's going on. And so once you tell them like, hey, this is a bunch of parquet files on a disk, they're like, oh, so I can read them, right? So why don't we let them read? Because we get to manage all of the reads and all of the rights, we get to build a lot of features that we can reason about. So one of the best examples is just access control. If we let people access our parquet files on the disk, then we'd be stuck with like Unix file system. access controls because they were reading files off of a disk and we don't have any code that's running in between the disk and them. But in fact, since they're reading them through our system,

we get to build an access control area and have accels and so forth through these datasets. Another thing that we can do is we can do usage logging. So this came out of a real problem with TraderDB and with other systems at Jane Street, which is they become nearly impossible to deprecate or to do scheme evolution on. But imagine you have a table in a database, and there are like 500 different people querying this database every day. And you want to delete this table. Well, which of these 500 people are querying this table? How do you find out?

Like, I can go, like, maybe, okay, so I write down every SQL query that everyone is running, and I like, grep for the name of my table. Well, okay, maybe this works, and I find, and I grep, and I don't see any matches to my table. Well, except there's a view that references my table. And now I have to get for the view too. Well, maybe there's a view on that view. Right. So, like, it ends up being very difficult to understand who is using this table and who's not.

Right. And then, like, even if you do understand it, you know, why are they using it? And, like, what data are they reading from it and, like, how much you care? Like, maybe you can keep this, this data, but you have, like, you only retain the last month instead of going back to all time. Like, can you do that efficiently? Who knows? So being able to have. have like full read and write access logs to Superstore turned out to be very important.

And to just even just like build a taxonomy of like, well, here's the list of all of the data sets that people are storing in our system. Now again, Postgres has that. But like this is a system that's many orders of magnitude larger than Postgres. I guess this is another important thing to say. It's like single computer databases only scale so far. Right. Right.

If you want more than, you know, a few thousand tables, you're going to need something that scales horizontally. Right. And all of this is happening in the context of like a huge explosion in the amount of data that we were using and the kind of intensity with it was using that data and many different new use cases for all of this data. And like somewhere along the way in the middle of this, a kind of revolution in the kind of machine learning driven approaches that we're using also arrived and also put a lot more pressure on this system. And it just turns out that once you have a system that's able to query, you know, over many terabytes of data, efficiently, you keep finding new use cases. You know, and again, we were building this, you know, this database we keep talking about is called TraderDB. We were building this mostly for traders and researchers, but we keep finding use cases

across the firm. Cybersecurity wants to have access logs for, like, our network. Oh, why don't you just put that in Superstore? Tools and compilers wants to have access for like all the history of the code review that people have done for all sorts of ad hoc queries that people want to do. Right. Every build that's ever run at Jane Street. Yep.

you just kind of put stuff in and then from the trading side, you know, you end up wanting like a record of the thing that a trading system thought at the time that it made a given order is like a very good example of a kind of data that has very high cardinality. You want to write down rows like that across the firm like many billions of times a day.

And we just couldn't do that really, except that now we can. So you end up doing it a lot. So one thing I wonder, about this design is like, you sort of pointed at a few different kinds of systems that, like, where there are kind of clear distinctions in approach. But, like,

data warehouses aren't like a new idea. Lots of people have written data warehouses. And, you know, there's lots of commercial products for data warehouses. Like, how did you think about the question of, should we just, like, I don't know, sign up for, like, ye old software as a service data warehouse thing versus, no, actually, we're going to go and design

our own thing from scratch? So software as a service is, a very interesting question here, because really most of the data warehouses that are at the cutting edge in the world that we see are cloud software as a service offerings, Snowflake and Data Bricks being kind of the two primary examples that it makes sense to point to. And these are perfectly good software. There are a few disadvantages there. One is they're extremely expensive, right? And in particular, whenever I talk to people who work at companies that use these systems, they end up thinking very hard about what queries they want to run or not

just because the cost of running a lot of queries in dollars is quite high. And we just sort of don't want to think about that. The other problem is that our data isn't in the cloud. Like we have a lot of systems producing data and managing data on premises and requiring people to copy all of their data to the cloud in order to run queries over them just seemed like not a great idea. Sure. But even still, once you're on premises, there's still many offerings.

There's commercial offerings. There's open source offerings. So why don't we just use one of those? well one of the things that we've learned and really I say we but this was really not me who was pushing this because I kind of came in here like oh this Jane's here is so much not invented here syndrome like we never use open source software we just like write everything to scratch probably it's because this wrong guy makes everyone write everything in O'Connell that is a fair complaint but I do think I guess now I've been here for long enough that I kind of see some

of the method and the madness in particular the finance use case. does not get very well understood by the open source community. Sure. Because we keep pretty quiet about exactly what we're doing and why, because we think it's a competitive advantage. Sure. And our competitors think the same thing. So, like, you'll have a lot of open source people

who can spout off, like, what's sort of the tech company reason to use an analytical? They're like, okay, people visit your website. You want to record all their clicks and all their visits and sessions and stuff, and you need a big database for this because you have millions of people visiting your website. That's a very well-understood use case for data warehouses. And so a lot of the open source products are very good at solving that, right? I think very few people who are developing these systems can spout off some of the finance-related use cases quite as well. And there's just a huge advantage for us in being able to do the specific thing that we want to do very well.

And because these systems aren't necessarily built for these use cases from the top, we don't have any particular reason to believe that they're going to prioritize the things that we care the most about. Honestly, even in the ones that do think more about finance use cases, there are time series databases where a lot of the people who are using them are finance companies. But James Street's a weird finance company. So being able to just sort of go into our system and make this thing work is extremely valuable. Right, like the exact choice of tradeoffs is like an enormously load-bearing thing. and the fact that we get to make the choice of like

which of these features matters and which doesn't and where can we give up on something and where are we going to demand that we really get really perfected. Like that's just an extremely powerful place to be. And we have use cases that like if we were to share them with people building these pieces of software commercially, we just make them shudder and like walk out of the room. You know, honestly, sometimes they make me shudder and walk out of the room.

And like one impulse that I've had to really stifle in myself is like asking people who are using tools like Superstore, like, have you tried not doing that? Like, I ask it more than no times, you know? It's definitely good to push on people and see if there's a better way for them to meet their use case and like the first thing they have to have come up with. But you really want to not push back on the use case itself. Because our traders are very smart and they've thought very hard about what is the thing

that they want to understand. So being able to say yes in many more cases than a vendor would perhaps say yes to us is a huge advantage. So one thing I wonder about the story in general that you're telling is that like, like you kind of joined Jane Street and were kind of immediately plopped into a team and started to help lead a project that was to replace a pretty central piece of infrastructure that, you know, the day it falls over like it's really bad, right? Like a pretty important central piece.

And like, here you are like a person pretty new to the organization. And like, how did that all happen? So a lot of this is not on me. And I had many collaborators doing this. Many of them had been at Jane Street much longer than me. I talked briefly about Sam, who was my boss when I started at Jane Street. He was very much bought into this general project and did a lot of the work of, you know, giving me talking tos when I, like, proposed something that just would not work at Jane Street

or that is like a bad idea for reasons that he understood and that I didn't understand yet. Right? into sort of transmitting to me what are the things that make software valuable at Jane Street and Y. And also doing a lot of the work with the prospective users of this system to explain to them what we're building and why and make sure that they were ready to use it

once something existed. So these were both incredibly important. I was involved with them, but I don't think that they worked, I don't think I deserve the credit for them going well. I think really this, involved a lot of very impressive organizational flexibility from across Jane Street. One kind of conversation that I had a lot that I found very impressive was, you know, I would go and talk to a team who wanted, who we thought might one day want to use Superstore.

This is before it existed. Right. And they would say like, hmm, this seems pretty hard. Like, are you sure you're going to be able to do it? And we'd say like, well, we're going to try really hard to do it. You know, we're going to prioritize like this exact thing that you're talking about working for like the following reasons. But like you just have to kind of trust us that we really think that this is achievable because we've like run some tests and it seems to look good. And they'd be like, okay, this sounds great. We will use it when it's available.

And I was like, okay, wow, this was a very sort of ego-free conversation with a lot of trust between dreams and like trust of me who had not particularly done anything at that point. And, you know, once we did launch the thing, we got a lot of. of usage very quickly, I think in many ways, it turned out to work quite well, which is nice. But people were willing to give it a try pretty easily and early on, which I think really helped contribute to the system's success. Yeah, maybe there's like two key cultural things going on there. One is this kind of extension of grace, right? The trusting of someone else is trying to do something and it seems like a reasonable thing to try and do. And I guess the other thing is just like the willingness to make

bets, right? Like, it's a trading firm. We make bets all the time. And the idea that like some of the software products we're doing are also bets. And I think, you know, I'm fairly confident that like the various people that you talk to who said, oh, this seems hard. Like some of them thought, yeah, this probably won't work. But like, it's okay, right? We're allowed to make bets that, like, you know, there's some risk and sometimes they work and sometimes they don't. Well, and one of the fun conversations on that note that I got to have pretty early on in my tenure at Jane Street was like buying the storage appliances that we're going to back superstore. So there's a few questions here.

One is like, when do you buy these? And the second one is how do you size them? Right. Now, this is in 2022 when like supply chain dislocations from COVID were still going on, right? And lead times were quite long. Thank God there are no more supply chain dislocations. It's all good now. All fixed.

So, right, so like, we didn't have a running system yet, right? So like, we sat down with a relatively small group of people. Okay, well, how, you know, what's the right amount of storage for us to buy? Like, I don't know how, but we ended up landing on, a number of like 10 petabytes. I was like, what petabytes really? Like that seems like pretty big. And they're like, yeah, but you know, you don't really want to go smaller than that.

Like, well, how much is this going to cost? Right. And it was like some, you know, eight figure amount of money. To be clear, I had just come from a startup where like we had never seen an eight figure amount of money in like the whole company's existence. So like, before that, you were at Google where they hadn't seen a storage number as small as eight petabytes. Right.

But where decisions were made in a much larger and more centralized. way. It's like the fact that, you know, a few people could get in the room and decide to write this check, you know, most, you know, on the strength of a design doc was pretty, I mean, it made me nervous that day, but I think it's also a very impressive thing to be able to do as a company of Jane Street size. So, Superstore, like, had a bunch of embedded bets in it of, like, different design choices, different things it was trying to achieve. And, like, on the whole, it has panned out very well. Are there any aspects of the initial approach or design or things you did that you kind of, in retrospect, think didn't pan out particularly well, decisions that you kind of regret? Yes. The main decision that I regret from Superstore is the way that we handled the metadata.

So in all of these columnar databases, the metadata ends up being a pretty key piece. Metadata being just like, what are the tables and like what data, you know, what are the pointers to the data that's currently in them? And I guess also like what are the permissions and. who has access to what? Is that also part of the metadata? In our case, no, but in many other systems, yes, permissions live in those metadata stores. Sure.

So we thought, well, what do we want to use for the metadata store? Like, this pretty clearly has to be some sort of distributed globally consistent storage system because you certainly don't want your data to be consistent, or not just you don't want. It's impossible to have a system where your data is consistent, but your metadata is not if the metadata is like the pointers to the data. Sure.

It's like, well, what should we use for? this. We decided to use CockroachDB. It's like a well-known, well-regarded, open-source. Transactional data. And again, you want the metadata store to be transactional, because very often doing these transactional, even lightly transactional operations over the underlying data require transactional metadata operations. Right. And like some of it, like, some, like you do a transaction, sometimes every time like you add a new chunk of data, you're doing a transaction. And like, it really looks

more like the Postgres use case. And that like, instead of operating on whole columns, are operating on rows of like, oh, there's a new chunk of data and I want to like register that chunk of data. And so the ordinary use case is like the dual. It's the opposite thing. Exactly. And in many ways, CockroachDB was a great fit, but it turned out that we've had more and problems with it as we've been scaling.

I think one of the key areas where we've been having problems with CockbridgeDB comes to, like, right latency. So we have a lot of the data that's in Superstore. I would say, you know, probably by bytes the majority of the data that's in Superstore. is data that comes in streaming. You know, some system is writing some stream of data, and then we want to take that stream of data and materialize it as a superstore table as fast as we can. As fast as we can is ideally, like, within less than a second.

Sure. Which is like, again, for a training system, not fast, but for, you know, for an analytical database pretty fast. Right. The fast enough that you could, like, build stuff for, like, analyzing trading on the day, and it would be fine, fast enough that you could build monitoring systems based on this. Yeah. Not fast enough that you would want your path from, you know, quote to order or something to go through the system.

And this particular style of access ends up being pretty bad for CockroachDB. Because every time you need to write to a row, you end up doing a transactional update, which for Cockcler'sDB being distributed means like distributed consensus round. And CockrochGyb gets its scale by just dividing all of the rows in the world into many different Paxos group. or sorry, not Paxos they use raft, many different consensus groups that are independent of each other, and like having a bunch of stuff on top of that to make that mutually consistent. But if you're touching the same row over and over and over and over and over again, in effect, you're serialized. There's no way for them to do those transactions in parallel.

And the like the wall time that a single consensus round takes ends up being quite high. Got it. And just to like unpack a little bit of the distributed systems terminology here, consensus is like the name for the algorithm that does like the simplest like get a bunch of servers in different places to agree on like anything. It could be a single bit. Exactly. I don't want to, right, I don't want to have there be a single computer that's the only place that stores like what's the latest data in this table because then if that computer gets unplugged, you know, I can't access my table. So what CockbridgeDB does and what many distributed systems do is they write this to a bunch of computers.

And then essentially they vote. If the majority of the computers agree on where the data is, then congratulations you get to write your table. And this gives you very nice fault tolerance guarantees. But the way that these computers have to exchange information ends up being a really limiting factor in the design of the system.

And just to feel like if you had to, like you were saying that the bad thing is having to go to like a single place over and over to make a decision. But actually, if you just had to go to a single place over and over to make a decision, it would be fine.

Yeah, it turns out CPUs are pretty fast. CPUs are fast. And they can make a lot of decisions. And it's this, all of this like network communications and distributed synchronization that ends up making this slow. And one of the things we're doing now is like ripping out all of the cockroach TV

and replacing it with ARIA. Yes. Right? And Aria is this very like finance-style state machine replication system that basically shoves all of the data. It gets consistency by shoving all the data through a single core, right?

And you just like distribute that data out by reliable ordered multicast and it gets to all the people. And like the thing that's weird about the system from a kind of, you know, big systems person kind of perspective is there is a single point of failure, which is the sequencer, the thing in the middle that gets all the transactions and brought that's super-frapped. There is a plug that if someone unplugs, the entire ARIA system will go down.

And once Superstore is built on ARIA, there is a single plug that you can pull to disable an entire Superstore cluster. Although it's worth saying it disables it in that kind of temporary way. Yes. And that you can actually recover in a sense in that, like all of the information that goes to the sequencer is also replicated and captured by other components of the system. Effectively, you're in read-only mode for a while. That's right.

And then there's like a careful, in the end, involving some human intervention step to, like, make sure the original sequencer is good and dead to flip over to a backup sequence. So like, when it's a single point of failure, you're not like totally dead, but you are paused in a way that's like not awesome. Although I think one of the points about this design that I think is interesting is like, it's true that it's not great to have single points of failure. but if you have like a single, single point of failure, it's not as bad.

Like a well-tended machine might experience a hardware failure like once every four years, right, in like a well-cooled data center and all of that. And then what you're really taking on in exchange for way higher performance is like, you know, it will be down for a minute once every four years. And like that's not necessarily so bad. And you have a background rate of outages. Like I'm not going to sit here and say, oh, Superstar never has outages. this is going to be our primary cause of outages.

Like, no. We're humans. We write bugs the same as anyone. And like if you look at what causes system downtime, including for like big tech systems that don't have single points of failure, it turns out that it's almost always caused by software bugs and not by like the kinds of failures that people are designing against. And so anyway,

so here the point that you're pointing is a bit of regret is like the core metadata system, like was basically architected, with a kind of like higher, higher requirements, like made a bunch of tradeoffs in favor of very high availability that blocked you from having the kind of throughput you needed to really scale the system as far as we want. And throughput in this particular dimension. I think what we, the reason we made...

Query throughput is fine. Quarry throughput is fine. Total commit throughput across all of our tables is fine. It turns out the thing that we can't scale is just like the rate of commits to a single table. And I just don't think that we fully appreciated how much of a blocker that would be. when we were designing Superstore initially.

Got it. Okay, so let's talk about like the after Superstore world. So you no longer work on Superstore. What's your day job today? So today I work on Jane Street's compute cluster, which is called The Hive. So the hive is not like all of the computers at Jane Street, but it is sort of the place that you go to

if you have, you know, a computation that needs to be done by a lot of computers at once. So examples of this include, for instance, training neural networks, but also things like analyzing, you know, many days worth of trading to understand what the patterns might be that we see there. Or running large-scale simulations where you take the actual code of our trading systems

and run them in simulated mode, like over large swaths of historical data. Yes, exactly. Right, and this system is actually like, I have like a great fondness for it because it's like one of the first, the early prototype of this thing is one of the first systems I worked on.

The original version of the hive was like six Dell boxes piled up on a card table. And I think it was like, you know, set up what we used to call a bear wolf cluster back in the day. It was like very simple and very primitive. And like it's a very different system now. Like first of all then it was like a very kind of simple and naive kind of scheduling story of like you would run one thing at a time. It would get the whole hive. You know, you'd have like a master process which like go off and schedule jobs running on individual.

individual cores and like gather the results together and like write them down to a file. It was also just about CPUs. We didn't do any GPU stuff back then. It was like more than 20 years ago. Things are dramatically different now. Can maybe see a little bit more like what's like what is the kind of use cases that are like driving the work on the hive today? So there's a ton of different use cases on the hive. But if you look at where most of our growth is coming from,

most of the growth in demand for Hive compute is coming from training neural networks and processing the feature data that we're going to feed to these neural networks. Some of which ends up being on GPUs and some of which ends up being on CPUs. Yeah. So the training itself for neural networks happens almost exclusively on GPUs, but a lot of the data production and the data analysis that go before and after the training runs happen on CPUs. Right.

The hive has been, we have hundreds of thousands of CPU cores and over 10,000 GPUs running on the hive. And this is big enough that it's become essentially a giant DOS machine. Like, you can point this many things at almost anything and make it go down. So one of the big challenges has been just scaling the underlying infrastructure that powers the hive to make sure that it doesn't rely on anything that it's going to DOS. and that it can actually feed all of these computers efficiently. So what are some of the things that we've had to make sure that we can scale? So one of them is just storage. All of these computers have to have their input somewhere and then also have their output somewhere.

Sometimes this is Superstore. But in fact, some of the larger datasets on the hive are too big for us to want to store them in Superstore. we just store them directly on the storage appliance or on a large object storage system. And it's very easy to accidentally write code that makes these storage appliances follow. So we've been working very hard to make sure that hive jobs don't bring down a storage appliances. Is that a problem you solve by making the hive itself better or is it a problem you solve by isolating? the hive so that its queries don't reach out and smack some

unsuspecting system on the head and knock over. So we have to do both. We definitely isolate the hive in many ways. And we make it much harder for hive. We try and make it impossible for hive jobs to accidentally interact with trading systems, which would be a big no-no. But even the stuff that the hive is supposed

to talk to, we want to make sure that the hive talks to it gently enough that that thing doesn't get taken down. A great example, we're still using, for better or for worse, NFS as the protocol by which Hive machines access much of their data. And NFS has some bizarre locking semantics around certain interactions, especially when it comes to things like listing directories and creating new files in a directory. A very easy way to bring down our storage appliances on the Hive is to create thousands of files in the same directory from different computers all at once at the same time.

Oh, interesting. Just because there's the kind of consistency semantics on the directory itself. Yeah, pretty clearly, I think most people would realize that if you tried to write to the same file from 5000 machines at the same time, that that would go badly. But people don't realize that creating a file in a directory is effectively writing to a file. Right. It feels like it should be okay, right? It's just like, you know, some kind of eventually consistent, eventually all the files that you added show up there would be fine. But like, that's not how NFS works, right?

So obviously you shouldn't do that. We put some checks in place like disabled jobs that we think are doing that. But then a big part of this is just building the interfaces to the hive in such a way that people won't do that. Right. Rather than having people open files directly, we build frameworks that they can use that can manage their data in larger chunks. So one thing that's notable is that like your experience coming to work on the hive, is sort of the opposite of your experience of coming to work on Superstore.

You started working on Superstore when it didn't exist, right? And then you got to ask the nice question of like, what are the APIs I would like to have people to have in order to use the system? And how can I, like, pick just the right tradeoffs and all of that?

And, like, in the hive, you came up in a much worse position of, like, a system that, you know, the worst of all situations I initially wrote. And then, like, has been used by lots of people and evolved a lot over time and is, like, you know, the beating heart of a lot of our research work.

And you don't have the freedom just be like, oh, and here's what all the APIs are going to be, right? It's already being used actively and all the time. So like, what do you do about that? How do you think about improving the system and changing the kind of tradeoffs in a world where there's already a ton of uses before you get to make any of those decisions? It's really hard, I think, is the long and short of it.

One of the things that I really try and do is I try and understand the full stack of code. One of the things that I found was happening a lot when I started on the hive is that we were building features and then shipping the features and then nobody would use them. And so why was that? It's because the people who the features were designed for were not actually using our APIs directly. They were using other systems that used our APIs in turn.

So it's not enough to design a feature that solves a problem. You have to design a feature in a way that solves a problem in concert with the other layers of our research stack. Right. And there's like 20 years of people hacking away on these layers. So there's like a lot there. Exactly. Having that sort of full end-to-end understanding and like actually interacting with

researchers, in fact, the first thing that I did when I started on the hive team was I didn't work on the hive at all. I sat on a trading desk for a month using the hive with them. I worked with these traders to take some of their code and rewrite it in a way that was sort of more maintainable and better integrated with the rest of the rest of. of the code base, but that involved a lot of using the hive and changing the way that this code interacted with the hive, which really gave me an appreciation for at least one of the ways that actual training code interacted with the hive. And my goal is to use that kind of insights

to better align the way that we released our features with the ways that users were actually working. So now that you've had some experience going off and using it and experience seeing what's going on in the team and making changes to the system. What do you think of as the problems that there are to solve? Like what are the important things to be done to make the hive better fit for like both what we're doing today and like the insane amount of compute we are bringing up over the next several years. So bringing this all the way back to the beginning, one of the main challenges that the hive has is a mechanism design challenge. So one of the things we have to do is we have to schedule different jobs, right? We have to decide what computers are going to run

which people's code. And sometimes more people are demanding high resources than we have resources to give. So we have to decide how to prioritize them. And being a trading firm, we do this using money. People bid in like dollars per CPU hour or dollars per GPU hour for how much they want to run their code on the system and we like run a second price auction.

So this all sounds well and good. But in fact, the structure of the mechanism makes it very hard for people to say things that we want them to be able to tell us about how they want their code to run on the hive. Are you telling me you would like them to be able to express their true preferences about how valuable they think their job is? So the funny thing, again, is that it's not so much about getting the exact dollar value, right?

Which, like, they don't know down to the scent how valuable their thing is. It's about what is the structure of their preferences. And in particular, the thing that we haven't been doing a good job at modeling is urgency, right? Maybe this piece of work is worth $10,000 and this piece of work is worth $5,000. But the $5,000 work decays much faster. If I don't do it this hour, it's going to be worth $0. Whereas the $10,000 piece of work, you know, if I don't get until tomorrow, it'll still be worth $10,000.

That's a very valuable piece of information when you're trying to figure out what the computers should do right now. Yeah, this is like a key insight that was not at all obvious to me, but as I've kind of heard people talk about the kind of scheduling problems on the hive, just like makes a ton of sense, which is if you think about it in terms of a utility curve, the thing that matters is the kind of derivative, right? It's like if your utility curve is flat, like, I would like to get this done, but like it's worth the same if it happens today or tomorrow, then like, well, I'm not very urgent to get it scheduled right now.

I'm kind of happy to wait. And like, if there's free time on the hive, happy to schedule it then, but it can also wait and other things that are more urgent can go first. And so this like very simple thing of not like a single dollar value, but a curve. seems like a much better way of thinking about what's going on. Yeah. So assuming, again, that you give people good ways to specify this, and there's a real API design problem for, like,

how do you not just make this thing of running a program on the hive be like an entire economics quiz? Yeah, right. Having everyone, like, you know, have full freedom and writing out their utility curve for every job sounds also like a nightmare. Right. And plus you hope that they're, like, convex.

Sure. Right. It would also be weird. Yeah, it would be weird if they went up in time. Well, but sometimes they do, right? Like, think about a thing that runs every hour or every day, but that you want sort of a regular monitoring cadence. Okay. Right?

In that case, you really, your utility is how close you get to being exactly one day after the last thing and one day before the next thing. Right? You're probably fine with like a 20-hour gap and then a 28-hour gap, but like a four-hour gap and then a 44-hour gap sucks a lot. Oh, so that means there's like a peak around. where you would like to get the job to run. Not a convex utility curve.

But we in fact don't modeling that. Okay, good. Back to the... We're not modeling that. There's a paper from CMU that talks about it. So you want to model this thing over time. This, of course, creates an NP hard scheduling problem. I mean, it's already NP hard because we have these chunky jobs that like need a certain

number of GPUs in order to run, right? Right. So that's all gang schedule. Right. Gang scheduling. Now you want like a whole group. And this is because the GPU problems themselves. like a lot of the, in fact, all of the original stuff that we scheduled on the hive was this kind of like, you just have a bag of jobs, they're all independent, they all just want to run, you want to get the result, that's kind of that.

And this is before multi-corro candles. They all use exactly one core, you really had an apptime problem. And now, you know, we have especially GPU jobs where they're like, you know, locked into a tight barrier synchronization thing. And they are like, if you're missing any GPU, you can't do anything and you just kind of have to have this very tight scheduling. So even solving the scheduling problem at a given time T involves a lot of sort of of combinatorial work. As our GPU clusters get more advanced, we want to be topologically aware also, right? We want to make sure that people get scheduled on GPUs that have good connectivity to each other. And then, of course, as our GPS are down multiple data centers, we have to pick which data centers things run in, which often means copying their data from where it lives to where the free computers are. So that also gets very challenging. But then on top of that,

we want to add this time dimension, which just makes the problem much more complex. So we know that we're not going to be solving this problem in like an algorithmically perfect way. But we still think that the moves that we can do, like if you think about this as a local search problem, sort of the moves that we can do to go from the schedule to a better schedule are valuable enough that it's worth adding this extra dimension in the modeling. Right. And like the, what's the key like pathology in the original kind of just like we're going to have, everyone's going to bid for right now, whether they can run and you just have the bid against each other and the highest bid wins. Like what's the particular thing that goes wrong in that case? So the particular thing that goes wrong is people end up paying much too much money to get their large jobs run and then feeling sad that they didn't just wait until the night.

Like essentially people sort of schedule hive jobs for the future when they think the resources are going to be cheap because there's no good way of them modeling their urgency in the bid. Right. So the main issue is like the auction happens. when you propose the job. And what you really want to do is, like, give the system your job and be like, here are my preferences. Run it at some point when it makes sense, given what I've told you about my preferences. Exactly.

And that's the thing that's missing. Right. People end up sort of doing their own work in their head or like sometimes even like writing little Python notebooks modeling what they expect to be the future usage of the hive to figure out when to submit their jobs. And thus we have pushed a lot of complexity on the users. Exactly. Okay, so that's one thing that's wrong.

There's like some problems in the basic scheduling mechanism that kind of don't let us kind of exercise the kind of intelligence we want to in terms of like picking what jobs run away. What else is wrong with the system? Another class of problems is that we're not very good at starting at running people's work efficiently on like a worker by worker basis. Right. So I want to run this thing on the hype.

Maybe I want to run like a thousand different copies of the same process over a thousand different shards of my data. This means that the very first thing that the hive has to do is get my code onto a thousand different cores. Okay. There's a bunch of problems with this. One is, well,

maybe my code is quite large and just like literally physically copying the bytes is going to be an expensive thing to do. Another thing that's a problem with this is, well, right now all of these bytes are located in one particular place, and now I have a thousand different computers all trying to talk to the same one computer that happens to have these bites. And that's not going to necessarily go very well either. Yep. It turns out the way to solve this is more or less with BitTorrent.

The same thing that we use to download, like, copies of Redhead Linux back in the day. Yeah. It's built almost exactly for this problem of like, I want to have this, I want to have this data fan out from this one place to many different places. Right. The fundamental insight here is that think about the first person who successfully manages to get a chunk of your file. Well, now they can share that chunk with everyone. else while taking load off of the place that the file originally was. Right? So if you have your readers also be writers or your receivers also be transmitters of data, then you end up evening out the work distribution much more nicely across all of the machines

that are trying to do work at once. And in this particular case of like a hive worker that's trying to start running a binary that it doesn't have all of yet, like that's just wasted cores. You might as well use those cores transmitting chunks of the executable to other machines. It's funny, BitTorrent is also like an exercise in mechanism design, right? Like, and this is a problem that we don't have. The original BitTorrent design is designed for like mutually distrustful people to share data. And one of the things that it wanted to do was like incentivize people to be transmitters by giving them more download speed if they donated upload speed.

Yes. Which I guess is not actually a part of the thing that we have to worry about here. No, we like to think that all of our, in fact, not only do we like to think that all of our employees are good actors for stuff like this. But also, there's no particular benefit to you and not transmitting while you receive, given that you're just sort of wasting space out of it. So the incentives are, in fact, perfectly well aligned.

Yes. Cool. What about the APIs that end users get? Are there, you know, those grew very organically, like the initial APIs that we provided were like rock simple and not very scalable and not very expressive. And then people who built lots of stuff on the outside of those and kind of extending

all sorts of ways those original designs, but there wasn't ever like a, let's stop and think about like what the right API really should be. So like, what kind of problems does that create? So one of the primary APIs that people use to access the hive is essentially this computation graph API, where you build this sort of graph of tasks that have dependencies of data that are produced by other tasks and so forth and so on, right? And you end up developing this entire graph where you have your source data at one end of the graph and then like your computation result at the other end. And you tell the hive, hey, run this stuff and it will, like, do it in topological order and so forth. So this is nice that we have this. One question you might ask is, are there

optimizations you can do on this graph to make it faster to run? And well, hey, now you're back to writing a query planner. In fact, one very common case of this is where your task graph is actually using a data frame library like polars in each individual task. And then, in fact, you can look at you can analyze the computation that this that polars or a different library is doing and optimize not just how the computation graph is structured between the machines but what computation gets done on each existing machine versus when the computations get distributed. Right. And the polar story kind of brings us all the way back to SQL.

All the way back to SQL. A lot of the work that's done on the hive looks a lot like a big distributed query. Not all of it, God knows. Right? I'm not going to say that GPU training is something we could write SQL for. But a lot of the work that's done preparing datasets and analyzing datasets on the hive looks much more like an SQL query than most code in the world. Right.

And like if you go look at the difference between like Polars and Pandas, which are like pandas being like the standard dominant data frame library and Polars being this like, you know, growing upstart. like the big difference there is, again, the shape of the APIs, right? And Polars is like, rather Pandas is the eager mode thing. It's just like you tell it what you want it to do and it immediately does it. And Polars, you know, does, you know, what's called like lazy execution. But really the way to think about it is the operations don't really run the thing immediately.

What they do is they build up the query piece by piece. And then you get the whole query and you can see the whole thing and the system can optimize it. Exactly. And other distributed systems like Spark end up having the same insight. Like Spark also looks a lot like a distributed query engine. So really we want to build sort of a distributed query engine style interface onto the hive. And we've been taking a lot of steps in that direction. Are you seeing that like the API that we want is literally Polars?

Or do you think that something of that style is what you want? I want Pollars to be part of it. I don't think so Polars has a distributed product. I don't think we're literally going to have people like right. you know, a polar's lazy frame, and that is the way that you use this interface in the hive. But my guess is that pullers will be in there somewhere. But you'll also have other capabilities that map to some of the, again, the Jane Street specific or the finance specific things that we care about, where we sort of privilege things

like operations over many dates or operations over different symbols. And we can do a better job because we understand our domain of making sure that things are, for instance, balanced across shots. But like the kind of key enabler is you want the computation, to be run in a way that gives you a kind of graph of the computation that is like you need some declarative computation graph and the more that we know about what exactly you're trying to run over what exact data, the better we can do of scheduling this and making it efficient. Awesome. All right. So that is like the mission that you are currently on or like part of the mission

that you're doing. That's definitely part of the mission that I'm currently on. It's like how do we run these very large dags of computation efficiently? Right. And what's more information that we can get about these computations that can help us do so. Awesome. Okay, well, maybe that's a good place to stop. All right, thanks so much. Thank you, Ron.

This is a lot of fun. You'll find a complete transcript of the episode, along with show notes and links at signals and threads.com.