Microarch Club - 1000: Ben Titzer

Episode Date: May 22, 2024

Ben Titzer joins to talk about the history and future of WebAssembly, the design and implementation of V8's TurboFan optimizing compiler, and the Virgil programming language. We also discuss bringing high-level language features to constrained hardware, the V8 team's response to the Spectre and Meltdown side-channel attacks, and how to design high performance virtual machines.Ben's Site: https://s3d.cmu.edu/people/core-faculty/titzer-ben.htmlBen on LinkedIn: https://www.linkedin.com/in/ben-l-titzer-6b78584/Ben on Twitter: https://x.com/TitzerBLDetailed Show Notes: https://microarch.club/episodes/1000

Hey folks, Dan here. Today on the Microarch Club podcast, I am joined by Ben Titzer. Ben is the director of the WebAssembly Research Center at Carnegie Mellon and a co-founder of the WebAssembly Project. He previously worked at Sun Microsystems, IBM, and Google. Ben has been interested in computing from a young age, getting his start, as so many others have, by building programs for MS DOS. That interest evolved into academic research pursuits at Purdue University and UCLA, where Ben worked on Aurora, a simulation framework for AVR microcontrollers, and started the first iteration of the virtual programming language. We spend the early part of our conversation focusing on these projects and specifically

how the constraints of embedded devices impact language design. We also talk about Ben's deep involvement in research and development of Java virtual machines at Sun Microsystems and IBM. Ben eventually joined Google following Oracle's acquisition of Sun, and after initially working on large-scale internal monitoring tooling, joined the V8 team in Munich. We discuss Ben's work on leading the design of V8's TurboFan compiler, the successor to Crankshaft, before diving into the founding story of WebAssembly. Ben spent the latter years of his time at Google working on software mitigation

for microarchitecture side channel attacks, such as the Spectre family of vulnerabilities. We explored the complexity of modern processors and the performance impacts of guarding against hardware vulnerabilities in software. We wrap up with Ben's current work at Carnegie Mellon, both on the Wizard WebAssembly runtime and the third evolution of the virtual programming language, which Wizard is written in. Ben shares his vision for the future of computing and how WebAssembly can serve as the universal substrate. Ben differs from many past guests on the MicroArch Club because he has spent the majority of his career working on software rather than designing hardware. However, I originally reached out because I am fascinated by the design considerations of building virtual machines, which many programs are compiled to target in today's modern computing environment. Ben's deep expertise in this area brings a unique

perspective to the hardware-software interface and can inform how we build on either side of that boundary. I greatly enjoyed my discussion with Ben, both because of its technical depth and the stories behind his work on some of the most used software in the world today. With that, let's get into the conversation. All right. Hey, Ben. Welcome to the show.

Hi. Good to meet you. Absolutely. Thank you for taking the time to join us today. Happy to. Awesome. Well, I know we chatted a bit yesterday in our pre-show chat, and I've been following your work for some time, and you've worked on a number of really interesting things. So I think we're going to have a really full conversation today. One of the things that really drew me to your work and some of the research that you've

done, as well as some of the things you've done kind of in practice and industry, is that you've worked a lot at the software level, but in how it interfaces with hardware or how it emulates hardware. So I think you'll have a really unique perspective that's a little bit different than some of the guests that I've had on so far. Yeah, I'm pretty much, I like to go depth first. And as a software person that kind of started out and I want to know how everything works. So when I got into computing at probably age 12 or 13, it was in the MS-DOS days. And that was like around 93 or so. Okay. You know, I read the whole MS DOS manual and those days there was an actual book that came with software is about that thick.

And it told you everything about how the operating system commands worked and everything. Obviously that's been phased out long ago. The dead tree thing is, is dead now. Right. But yeah, I learned a lot just from reading the manual and trying things out.

Looking back though, I mean, so I had a computer that was my dad's computer and he used it for work and he had installed some software that was, that basically downloaded like stock quotes from a satellite feed. And he had some custom software that he was paying a lot for actually, that basically gave him a real time feed of actually the commodities market. And that PC had only two megs of memory. It had the one meg that it shipped with. And then the expander pack, which was four 256 kilobyte DIMMs or actually SIMs in those days.

So the thing was totally maxed out. It was a 386SX running MS-DOS 5. And I just, you know, on the weekends or in the evenings, I could tinker with it. I don't think he realized how dangerous it was for me doing that. I mean, there's no users or anything like that. I could delete anything. I could run anything. And DOS came with a few games. So I played those. And I think most people who use DOS learn basic and I did not. Oh, interesting. Weird. I learned how to write batch files and I had a, got a kick out of that. And I remember I watched that movie Jurassic Park and in Jurassic Park, there's a scene where Dennis Nidri like sets up some kind of password thing and I think they try to enter the password

and it says uh-uh you didn't say the magic word and I remember that was so cool I wanted to figure out how to do that well I never managed to make it do it like a gooey thing but i always was trying to make like password entry hacker things right and so i did a ton of tinkering on dos um there and at some point i think we were we were in like a computer store me and my dad and there was like a whole bookshelf full of books on programming and i just kind of picked one at random and I kind of begged my dad to buy this book for me and it turned out to be a book called teach yourself to program c in 21 days nice and there was like lots of other books that were all like who knows what it was but that turned out to be like a really good choice because I learned c which turned out to

you know obviously be such a pervasive language and be everywhere. But in those days, Pascal was still a thing. I don't know if there was a book on Python. It was kind of early for it to be a book on Python. But there was lots of books, and I just happened to pick that one. And yeah, I read that book. It probably took me longer than 21 days. And I didn't actually have a compiler because MSs-dos did not ship with the compiler so i saved up money and saved up money and bought turbo c++ for dos which was a borland product

and that had like a amazing for the time amazing technology for the time it had a an ide it had an interactive debugger like line by line you know i had like syntax highlighting it had a manual for everything so it's designed for dos right it had a manual for everything in the library so i could actually learn from it um and also had a it also came with the book too and that actually taught me c++ from that book it was like also like a full textbook where you know i read everything in there and learned how to use classes and all that. And, and those days I was, I was really interested in making, I don't know. I tried to make some encryption thing. I didn't really know what encryption was,

but I tried to make some encryption thing. And I made a, a gravity simulator because I saw that on like one of the school computers in the library, they had like this little demonstration. It was like Microsoft and Carta in those days, they had a demonstration of like the moon going around the earth. And I was like, I bet I could program that.

And I had a couple of math classes. I knew a little bit of trig and just kind of winged it and made a, made a 3d wireframe thing, and it was like you could rotate it around and zoom in and stuff. It was really cool. And I showed that to my math teacher, and he's like, wow, that's awesome. You should stay after school, and I'll show you my programs. And he showed me a program that kind of blew my mind.

It was animated, and it was a 3d uh simulation of like an atom orbital orbitals and so it kind of it did this thing where it tried to balance like like the the electrons push each other away push each other away and so it just kind of naturally settles in a state where it's like the lowest energy state. And so it was like bouncing around. I thought that was so cool. And yeah, it was so smooth too. So I just got, I got hooked on programming from that kind of stuff. Yeah. And then after that, I was like, how does all that work? Like how does DOS work underneath? And so I started reading about operating system kernels. Um, by the time I got on the internet, probably around 95 or 96, I could start downloading articles and, you know, like, you know, basic HTML stuff that

had documentation for how this, these kinds of things worked, right? You know, like instruction manuals are learning assembly. And it's just like, I had so much fun at that low level, that I kind of i didn't i mean i made a web page and stuff but i didn't really program like javascript like javascript was still not a big thing yet um and i just kind of i just kind of hanged out at the at the low level and i kind of always been down there assembly and machine code has always been like kind of part of my thinking about things right and was that still were you still using dos at that point or you moved on to something else

oh yeah so about 94 i bought the next book which was windows 95 programming so microsoft was way ahead of the game they released like programming books for windows 95 like a year before so that people could get ready. And I had, I didn't even have Windows, but I bought this book. And when 1996 or so rolled around, I bought my first PC for myself, which was, I think it was a Pentium MMX. Okay. And that had Windows 95 on it. And so I could program Windows 95. And by then, I had to buy a new compiler, Borland C++ for Windows. And I used that for some time. Like I did some, like I made some,

I think I tried to do the whole Dennis Nidri thing with the password thing again. Again. That one sticks around. It was a lot of fun. I made like a graphing calculator application, but I don't know. I somehow still got, I was still interested in the low level stuff. I wanted to make a programming language at that point, so I wasn't satisfied.

And I really didn't know what I was doing. Like the first forays into that, I made some, I made some kind of like interpreter for some byte code that I made up. And it was enough that I could write something like Fibonacci in this byte code. And I wrote the interpreter for it in x86 assembly. And I got really interested in like overclocking and like building your own PC and stuff. And I used to hang on,

hang out on forums under a pseudonym that I will not reveal ever. And I actually put my interpreter up as a benchmark because people were benchmarking CPUs. And that was like in the days where like clock speeds were still increasing and like doubling every year or year and a half. And so, you you know there's a new amd k6 coming out and all this right so i i put up uh some assembly program as a benchmark

and in those days people just downloaded assembly off the web and like yeah i'll run that on my computer like nobody ever thought it could be malicious right it's like the uh the you know curl to bash of of that day i guess yeah that's awesome still a little bit suspicious but people do it sure sure absolutely today absolutely and and so you're still in in high school at this time right and um was there any part of i know you mentioned like you know know, your math teacher doing some programming, that sort of thing. But I'm always curious about whether computer science or computing in any facet was part of folks education, especially, you know, before they get to college. Was that something that was common where you grew up or not so much? It wasn't common. They did have one CS course and it was my teacher, my teacher was Mr. Gunn and we, he taught

us Pascal. Okay. So in class we learned how to program in Pascal and I already knew how to program in C. And so I was thinking, Oh, I will make, um, a Pascal to C translator. I didn't actually ever follow through on that, but I, during class I was always dreaming like I will do this. And I think at some point i might have done gone as far as like making a c header that had a bunch of macros that would

expand so you could write pascal like code but it would expand to like curly braces and stuff but pascal has seemed kind of clunky like compared to c i i somehow i I don't know, I must be weird, but I like the curly braces. I think it's very pretty. Awesome. Awesome. So you've kind of, you know, gone pretty deep into programming down to the lower levels in high school and that sort of thing. When you're thinking about going to college and what you're going to study and that sort of thing, were you pretty set on studying computer science or was that something that you kind of decided once you got there? Yeah, I think by the time I was admitted, I mean, I only applied to Purdue. So I grew up in Indiana. So it was like all the kids that were interested in engineering would

go to Purdue. It's like either go to Purdue or IU or go to a local school. And so I applied to Purdue. I got accepted. My grades weren't super stellar. I mean, I was always good in math and things like that, but it was not like I was like a valedictorian or anything. So I got into Purdue. Computer science was like the obvious choice. And I managed to test out of like the first semester of programming. So I started in like the second programming course.

So I think the first programming course was like C++. So I skipped that one. I already knew how to program in C++. I didn't know how to program in Java. And so the first semester was Java programming. And I really liked that. I thought Java was really cool. And then I took operating systems, probably the next year. And in operating systems, by that time, I think, yeah, by the time I went to Purdue, I was already deep into writing a kernel. I mean, I wrote a very simple hobby kernel that you could just do like a terminal. It could do like virtual memory in like protected mode. Like there's many tutorials about like what's the special assembly sequence that you need to turn on protected mode and like where do you get

a boot loader and all that stuff um at some point i got more interested in languages than operating systems so i kind of left that by the wayside and i think i think sophomore year i took compilers and that was kind of like a watershed moment because i didn't have any formal training or even think to read up on how compilers should work. So Tony Hosking was my professor. He's still a personal friend. He was such an energizing character, too. He's like such an animated guy, an Australian guy.

He's got a pretty thick Australian accent. And I just love that course. And so I approached him maybe the semester after I had that course, I saw his name on some flyer for like undergraduate research. And I'm like, oh, Tony Hosking, that guy's cool. And so I talked to him and almost didn't get a chance to actually talk to him because I think we set up a time and then he wasn't there. And then I'd set up another time and he wasn't there.

And he was like kind of an absent-minded professor. Finally got him like the fourth time. And then he managed to get me connected to Jan Vitek, who was also at the, at Purdue at the time. And so I got involved in research. Jan was working on this VM called OVM, which is JVM. And so once I was into that research project, like I just started devouring research papers and, you know, hanging out with grad students and like became friends with everybody in the lab. And so by the time I was a senior, I was basically completely embedded in the language implementation and VM world. Right. And so you, you mentioned that you, there was a programming class with Java and then obviously doing this research was on a Java virtual machine.

Was there something about Java? Because throughout the rest of your career, as we'll get to eventually, right, it's been a very common thread in a lot of different things that you've done. And, you know, some of the things you do kind of tie back to it as well. What was it about Java that kind of like drew in your attention there? And has maybe kind of made it stick as well? I think with Java, it was like, you know, I programmed a lot of C and I crashed a lot of C. And I was kind of sick of that. And Java had just so much nicer ergonomics around like buggy programs. And that's something I internalized. So Java is a portable language.

I won't give you the sales pitch. I'm like, I, I'm not like a dyed in the wool Java person, but the great thing about it is that like the memory safety, right? Like you can't just go out of bounds and trash stuff. And then,

you know, you have to like spend forever figuring out what in the world went wrong at the implementation level. And so Java being a portable thing and getting into working on a jvm like the jvm has a very strict contract about what it implements like a jvm pretends to execute bytecode by bytecode the same way that a cpu pretends to execute an instruction by instruction but in reality it's like not doing that at all it's like dynamically compiling into machine code and like you know object representations are not at all visible at the source level.

So I guess I internalized the idea of like having the nice source level. But then the implementation level, which is not visible to you, is where all the smarts happen. happen and the the idea that the compiler in particular by this time like i was thinking wow jits are really cool like because you can use dynamic information to optimize the program which i'd never even thought about at the c level because it's like statically compile compiler's done its job so it's like kind of mind-blowing for two reasons the dynamic compilation aspect and also the safety aspect of like having an actual abstraction there that like works the same way everywhere and like i really like this contract where like the vm is going to do the same thing everywhere and and did you actually have experiences this is something you know i

hear a lot and i've experienced java to some degree as well in my career um it is the actual uh promise of it being portable across like, I know that you eventually kind of like got into exercising that a little bit more by working on more constrained devices. But initially, when you were getting into Java, was it mostly just like, you know, on a personal machine or on a server or something like that? Yeah, it was mostly all PC stuff. I think where Java broke down is the portability of the libraries and the platform stuff. Like the bytecode semantics themselves actually are very strict.

And so VMs really didn't diverge on what like your base level program did. It's all about the libraries and how threads work and like how GUI libraries and stuff like that work in concurrency. So I totally bought that story about it being portable. You know, later on, I worked in Sun Labs, which I guess we'll talk about in a little bit. And I did talk to James a bit, James Gosling. Java started for embedded systems. It was for cable set-top boxes, and it was all kind of interpreter-based. It's kind of weird that it went into the server space. That's clearly driven by the market that Sun was in and where money was.

Right. But yeah, it kind of made sense to have a low-level bytecode thing that you could run on these weird devices. That seems to be just recurs like every five to ten years is that there's another low-level thing. And usually the tool chains for those aren't very good. So you want to have like a, I mean, your own work here, right? You know exactly what I'm talking about. Right. Right. Absolutely. One of the guests I had on in the past was Robert Gardner, who worked at Sun and led the design of the Spark instruction set. And he he also kind of like recounted that story of the Java being for the the set top boxes, like you're saying, and then kind of getting co-opted.

I think, you know, perhaps as part of like Sun trying to figure out, you know, what was going to stick and what wasn't. But it is really interesting that, you know, it does at least, you know, for someone of, you know, my vintage, if you will, the it is very much like a server programming language, right? And that's kind of like the only domain that I've experienced it in, with the exception of like, you know, some weird microcontroller kind of stuff as well. And that might be kind of like a good jumping off point into after undergrad, you decided to go and pursue a master's and a PhD. And I believe you went to UCLA for that. And you got into working on Aurora. Is that how you pronounce it? Okay.

Yeah, it's kind of a mouthful. It's not really a nice word to say. I did. I read the background on the name, which I can appreciate. I think it's the Aurora Borealis and then the AVR microcontroller. So I appreciate the creativity there for sure. Yeah. Well, so when I named it obviously AVR like popped out, I can't remember. I was, must've been searching the internet for things. And I found this thing about this battleship, this Russian battleship. And just because of the different alphabets, like it's spelled with a V. And I was like, wait a second. I don't want to be, because it like played apparently it played some kind of pivotal role in like, you know, the Russian revolution to communism.

I'm like, I don't want any part of that. I don't want to take a political stance with a name. I'm like, this is just AVR aura. Right, right. So before we get into exactly how that system worked and what the purpose was, was your experience researching in undergrad, was it just kind of obvious to you that you would then pursue postgraduate studies? Or were you kind of undecided? And, you know, it just ended up going that route. It was definitely natural progression. I mean, being an undergrad, I basically started doing research at the end of my junior year, I think.

So like, basically, my entire senior year, I was in the lab, like doing undergraduate research. I had a desk, like they gave me an actual computer. So I was working around graduate students all the time. So I just kind of absorbed the culture. And, uh, I did an internship at sun like after my senior year. And so that was like just, uh, more of getting into the whole research swing of things. Gotcha. And so you get to UCLA, and do you jump right into this Aurora project, or was that kind of later down the line? It was almost immediately.

So one other thing I wanted to mention, like, man, Java was so cool in, like, 2001, 2002 for VM stuff. And, like I mentioned before, before like devouring all these vm papers like when i went to sun before ucla um because i did one year of grad school at purdue too um when i was at sun i was like at the sun labs and like james gosling had a office in the building used to see him every week on the weekend. And people were working on JVMs. They were working on the Hotspot VM. And that time, it was still closed source. But people were publishing papers.

IBM was publishing papers. It was so exciting to me to be around that. It's like, why would you step away from that? Right. So it was not only just a natural progression, but it was actually really exciting to kind of be to feel like I was part of this like movement towards Java being this cool thing. Absolutely. That makes that makes a lot of sense. And so when you get to UCLA, you jump into this project. You talk a little bit about what the purpose of the project was and also maybe, you know, for folks who aren't familiar, what the AVR architecture was slash is and maybe why that was the focus of this project.

Yeah, so when I went to UCLA, that was with Jens. So I switched advisor to Jens Palsberg. And he had decided basically because his fiance was in Los Angeles that he wanted to go. So he took me and two other students with him. And I actually was already working on Virgil and I wanted to go. So he took me and two other students with him. And I actually was already working on Virgil and I wanted to work on Virgil. But Jens is like, hey, we should do something with this sensor network thing because they have this big center here. And it was like headed by Deborah Estrin. And he was, you know, impressed by her work and wanted to work with her. And so there's

so much going on with sensor network stuff. AVR came into the picture because it was a microcontroller that was used in a very popular sensor platform. It's still apparently a popular sensor platform in academic research. And I just started hacking around. I'm like, there's not a good simulator. I don't really, I'm not really an embedded person. Like I don't really want to tinker with the devices that much even though it's neat so i wrote an emulator to start i wrote it in java and that was like probably the most that was definitely the most java code that i ever wrote like on my own it was like i did all the java things that you do all the interface crap and like all right right design pattern stuff i like went overboard i was using intellij iJ. I loved it. It was really, actually, it was wonderful. IntelliJ is such a cool tool and I kind of miss it. I used it like

maybe a couple of years back and like still kind of miss the refactoring capabilities. But anyway, it was all oriented towards doing sensor network stuff. AVR is an 8-bit microcontroller. You probably know about it, I'm sure, because you have embedded background. But just for the audience, yeah, AVR is actually pretty cool. And it's a low power thing. It doesn't have much memory. It's very simple. You have a couple memory map devices that you can poke the registers. And, you know, I spent a lot of time reading the manual. We wrote emulations of all the devices that come with the AVR. And the sensor networks had this wireless radio. And you could program the wireless radio by like poking these registers,

and then it would send out a packet. And the first paper that we had about that was kind of neat. I realized that it takes a certain amount of time to send a packet. And since they're doing this wireless sensor networks, they want to do simulation, a large-scale simulation. And the inside of the first paper was basically, well, you could run every node in its own thread. And since they have like this latency of communication, which is like, I don't know, 6,000 clock cycles or something like that,

they could run like out of sync by 6,000 clock cycles. So they could each be in their own thread. And 6,000 clock cycles, if you think about that, that's like maximum 6,000 instructions. But actually for an interpreter, that's like a lot. So there's actually quite a lot of parallel work that you can do and still be cycle accurate, like literally cycle accurate to when a packet is delivered by the radio.

Anyway, I don't want to go too deep into that because it's like an esoteric thing. But that was like one of the neat things that I liked is like I discovered a thing, I implemented the thing, I wrote a paper about the thing, I submitted the paper, they actually liked the paper, and I got to present the paper. So that like boost of, you know, like dopamine or whatever, like that rush was very addictive.

So I just, I kind of really liked that. What, what is the kind of like purpose of these sensor networks in the academic setting, um, that, that you were driving simulation for here? Hell if I know. You just thought they needed simulation, right? It didn't matter why. Yeah. Yeah. I mean, I think it was one of those things where it was a hot research topic and people were throwing out a lot of different ideas i remember the time it was like oh we're going to monitor like nature stuff which seemed like cool you know like wildfires and whatever bird populations but the issue is that the battery life isn't there like you can't really

just deploy them for a year like they will will last a couple of weeks or a couple of months on batteries. And that doesn't really work. You can't really go. Cause they were talking about putting them in remote areas and stuff. And so that kind of didn't really work out. I don't want to paint too broad of a brush. Cause I kind of moved out of the sensor network area, but it seemed to like to move towards more wired deployments in the end even though low

power is still important like yeah like the dream of having like these smart moats and like smart dust like somebody even literally had the term smart dust forget about e-waste can you imagine like just spreading microchips all over the internet. It's just in there. No, that's not a good idea. So yeah, like energy became a limitation. So I think that was the thing that – energy really drove everything in the design of sensor networks because one, sure, you could pick the lowest power microcontroller. But the radio was really the thing that kills you because you need energy to send radio waves. Like, it's just inescapable physics, right? And so people have developed lots of cool techniques about when to turn the radio off and, like, synchronizing them so they only turn on. So you have, like, duty cycles.

But I kind of, like, just was interested in the CPU stuff. Sure. Right. Right. Awesome. Yeah. And you mentioned, uh, before we jumped into Aurora there, Virgil in passing, that was kind of the, the kind of next thing that you moved on to, uh, after the Aurora project. Is that right? Yeah, actually. Um, I kind of did it a bit

concurrently and I put it a little bit on ice to work on Aurora, but I was thinking, like, I want to make my own language, and I had started off, I wrote a compiler in Java, and I put it on ice during the Aurora, maybe a year or two, and then I got back to it, and I was like, well, Virgil needs a reason to exist, so I compiled Virgil to C, and then compiled, excuse me, compiled the C to AVR. And then you could run, you know, Virgil on the microcontrollers.

So I mostly ran on the simulator, but, you know, students did actually put it on the microcontroller and make it work. So we didn't really do anything large scale with the language. Other than, you know, there's some really neat compiler techniques that I came up with that I wrote a couple of papers about. People can look up one of them in Upsala in 2006. That was really scratching my PL implementation itch and design itch. And that became a part of what I put in my dissertation. And I think that is the paper you're referencing. Is that the objects on the head of a pen or something like that? Yeah, I read that one. And it was super interesting,

because you kind of like systematically go through both the constraints, because it's in the context of the microcontrollers, and also, you know, how you can basically bring the desired features of Virgil into that constraint domain. What were some of the, um, the design decisions that were made in Virgil or at least in Virgil at that point, um, that made it suitable for that type of environment? One of the first things is that basically C compilers for that class of hardware, they really don't have a library that has dynamic memory allocation because I mean, like many AVR models have like four kilobytes of memory. Right. And so what are you going to do with that?

Like allocate like 12 arrays and then deallocate them? Right. So most people just statically allocate everything. They just declare it as top-level variables in C. And so that was like basically a hard design constraint. Like, okay, this language can't have dynamic memory allocation, which seems counterintuitive because virtual is kind of inspired a bit by Java. So it has like objects and it has like functions and stuff and arrays that are actually on a heap.

And it's a memory safe language too. So you think Java, you think like garbage collection, right? And so I had to square that. And the idea I came up with is basically you just, you pre-initialize everything when you compile the program. right? And so I had to square that. And the idea I came up with is basically you just, you pre-initialize everything when you compile the program. So that's why there's an interpreter for the language built into the compiler. It actually runs code. And I decided like, you know, I don't think C++ had constexpr at the time. Maybe it did, but I was certainly out of that community. But I was thinking like, why would you constrain the language?

Just let the any code run. So you could just, like, allocate whatever data structures you wanted. You can initialize them, hash maps or trees or arrays and stuff. You don't really need that complicated data structures for microcontroller programs. But then the idea is, like, well, then you just burn it into the, not the ROM, but like into the image, right? Right. And so one of the things in the paper was like, oh, well, once you have the whole heap and you know, you're not going to allocate any more than like, why don't you make the compiler just like optimize the crap out of it? Absolutely. So you're kind of getting some of,

you know, it's obviously not dynamically happening, which is one of the advantages here. Right. But you're kind of like bringing the some of the aspects of JIT that you were talking about earlier into this more compile time phase. How was that applicable in other like was that useful in other domains as well? Or was it kind of like microcontrollers is really where, you know is most applicable yeah so i did an internship at ibm in 2006 and we did this project called exovm and exovm was basically actually when i showed up for my internship so i interned with david bacon he's still a good friend of mine he He's such a, such a cool guy. Uh, they did a lot of GC research and, um, I don't think we even really had a plan for my internship. It was just like, come do some cool stuff. And it probably wasn't until about the middle of the summer when I, when we kind of like hit on something that was an idea for a project. And it was this thing called

the XOVM where basically, uh, what happens is you kind of do the same kind of like run some code and then snapshot it. And so you do that, but for a Java application and the VM itself together. What that paper was about was basically what if you wanted a VM which is effectively customized to an application and you only include the part of the vm that's necessary for the application so we were working on j9 which was closed source at the time and we're j9 is like a super weird configuration system uh but anyway so it had like an embedded version and so i was working on the embedded version. And it already had support for like snapshotting because they effectively had another in-memory representation for Java class files that was more efficient for embedded devices. And that was like an easy jumping off point for this project. And so what the XOVM thing did was basically reuse a lot of that to be able to snapshot the Java application, including its heap.

Like you could have an initialized heap and all that, if I remember correctly, and then just have like throw out the parts of the VM that were not necessary, like the exception handling mechanism or this part of that library or whatever. And so that, that idea of like having a snapshot of something and then optimizing the program against that snapshot, that's like a theme that reoccurred. So is that, it's kind of similar to something like profile guided optimization, but you have the, a VM that's in play here as well. Yeah. I would say profile guided optimization mostly is behavior, whereas this specialization includes data. Okay. So you're optimizing code and data together. So I'm not that familiar with LISP systems, but I know every time these topics come up, people bring up LISP systems that have done this in the past.

I think this is very common. Certainly Smalltalk had the concept of images. And I don't think they took specialization quite as far as some other systems. But this idea is extremely powerful. We keep rediscovering it. So if you're familiar with Graal Native Image, that is like a modern thing. That also uses this idea of pre-initialization and optimizing code and data together so i think this is actually a super powerful idea i think it's a very universal idea awesome yeah that makes a lot of sense uh you know you're bringing you mentioned objects and those sorts of things uh that you're kind of bringing into the domain of microcontrollers

that uh are not typically there i I guess, kind of, could you walk through a little bit about why maybe some of those, what some may call like higher level language concepts are not present, you know, when working with more constrained devices, and then how they have to be changed or adapted to be able to function in that environment? Because, you know, kind of when reading the Virgil paper, it kind of stuck out to me that it was like, you're walking this like thin line, right, of like trying to bring in the bits of Java or Java-like language into this world while also maintaining things like you need to be able to actually access hardware directly, right? So you can't have a runtime or something like that.

What are some of those things and how do you fit them into that environment? Yeah, you mentioned the big one, which is the runtime system. So working on a JVM, you realize that the JVM gives your program lots of services, which is good for programs that, you know, most people write. You want to have these services so you don't have to re-implement them yourself, like exception handling, garbage collection, like lots of libraries, right? All that takes space,

and space implies like a fixed cost and also like a per class or per function or per object cost. And in a micro-control environment, you basically cannot afford that. Like the base level metadata you would need for like a feature complete jvm is like far beyond a micro controller it's going to be like dozens or hundreds of kilobytes if not megabytes so for micro controller if you

got a couple kilobytes or even like 256 bytes of memory you just can't have that so what i was going for in the virtual design is like, make sure that the language doesn't imply a runtime system that has a big fixed cost. So like there's not like a lot of data structures laying around that you just need to run any code at all, like regardless of what the program is. And then also for the program,

so the part of the metadata that scales with the program or is proportional to the program, keep that as minimal as possible. So in Java, like, there's a lot of things you can do with a Java object. Like, you can lock an object. You can hash an object. Like, it has an identity and, like, lots of other things, too. And it's got, like, you know, lots of default methods. And all that implies metadata per object.

So Java object headers are probably two words on most vms and that's like a lot like if you imagine like you have a microcontroller and like you have an object that's got two integers in it and then it's also got two header words right right it's like half your memory is gone just for metadata right So I didn't want that. So I was explicitly trying to keep that down to a minimum. So about the most that I could afford was just to have a dynamic type for an object so that you could do like casts and virtual dispatch. So virtual object headers, at least for that,

are like minimal, as minimal as I could possibly make it. And there's no garbage collector because we talked about like it's initialized at compile time. And then like pretty much every other service where you go down the line, you look at it, like, what would you need to implement? And usually just don't have that. Right, right. And you mentioned kind of like, three key optimizations. I think, I think they were even mentioned in maybe the abstract for the paper. One that's kind of relevant to what you're talking about, and maybe we could step through each of them, is reference compression, where you're kind of like talking about the difference between, you know, a C pointer and, you know, a Java reference or something like that. And you're talking about the metadata encapsulated in it because you're, you know, when you bring type safety into a program, have to understand what you're what you're referencing right when you're pointing to something can you

talk about how virgil handle handled that and maybe how that kind of like struck the balance yeah um it might be going a little bit too deep on the the old virgil so to speak the micro controller virgil but that that work is basically like since since you have a fixed number of objects you know how many there are you can't allocate any more the idea is basically well instead of having a pointer to an object you just have an indirection and so you just have a table and so you can use a fewer number of bits to represent a reference because it's just an index into a table typical trick right it's like another layer of indirection. It's like the one weird trick that CS people use. So that means that like, you know, if you only have 15 objects and null to represent, you only need four bits. And so then you can bit pack a field.

And so wherever they go, you know, like it just takes less space. And that turned out to be a pretty big win. One of the things that made that even better is that Virgil doesn't have a class that's the superclass of all classes. So there's no object at the top. And that means that separate class hierarchies cannot possibly ever be aliases. You can't have a reference that points to this hierarchy or that hierarchy. It can only be one. And so you can compress the references separately just based on a type-based analysis as opposed to doing some important analysis. I mean, advanced pointer analysis

might be able to figure that out even with the presence of object. But I liked type-based analysis. It was easier to do it that way. And so it's like very easy to understand like, oh, this hierarchy is like only these four objects that ever exist. So that worked out pretty well. Another optimization was like some parts of objects are often immutable. So if you split the object in half,

and again, because you have an indirection, it's not a pointer to memory, it's a number, you can put the read only part in ROM, and then you put the read write part in RAM. And so that saves a bunch of memory too. Absolutely. Yeah, I was not familiar with the term ROMization, which I think is what that was referred to in the paper. But I like it. I think it's pretty descriptive of what you're doing there. The other optimization that you mentioned, and then maybe we can jump on to some of the things you worked on after grad school was a reachable members analysis. Can you talk a little bit about how that worked? Yeah, this is actually the most important optimization. So if you think about it,

dead code elimination and linker deduplication are really just scratching the surface of what a compiler can do for optimizing away dead code. What reachable members does is basically imagine that you don't have any code in your program at all. And what code do you need? Well, obviously you need the main function. So if you bring in the main function and then you analyze statically with main calls or what it uses, then you have to bring in those. So reachability, that's why it's in the name. It's like, obviously the things that main uses and then transitively what those things use. So reachable members analysis basically brings in both the code and the data, which is reachable from main. So if you have like a big

library that has a bunch of stuff in it, but you don't actually use that, only the stuff that your program actually does use that is determined by the compiler by starting from main. It's naturally and necessarily a whole program optimization. You need to have all the code in order to be able to analyze it. For microcontrollers, that's not a big deal because it may be only a few thousand files or lines or whatever. So you just have it all in memory and you just do this analysis. And it not only is for code, but also for data.

So imagine you have an object, but in the reachable code, you never actually refer to some field of this object. Well, then that's, it can't ever influence the program execution. So you can just remove it and obviously everything that it references. So I call it reachable members analysis. Other people might call it dead code elimination, but I like thinking of it in terms of reachability because the dead stuff it actually never sees. And so even in the cost of analysis, it's only the things that are reachable that the compiler spends any time or space on. Gotcha. Yeah, you could almost think of it, well, correct me if this is wrong, but the way you just explained it, it's almost like you're looking more at, instead of trying to look for what doesn't get touched, you're only looking at what does get touched, which could obviously be more efficient in some cases as well if there's a large discrepancy between those two things.

Yeah. So in the current version of Virgil, I took that idea even further. It does devirtualization based on reachable members analysis. It will do inferring constants for fields. So like imagine your program, it allocates objects, but there's one field is like a configuration and only ever points to objects of a certain type, or it only points to an object, a specific object, or maybe it's only 42. The compiler as part of that reachable members analysis will figure out that it's actually a constant, remove the whole thing and smash the constant in. So it will do specialization basically

to the data that is present from your initialization. And it turns out to like kick in so many things because so many optimizations follow from constant propagation. So like imagine like this, you know, you figure out this field is always null. And like, there's checks all over the place. Is this thing null? Well, you know, which way they go. And so like a bunch of code, like, folds away to right, right, absolutely.

So you mentioned, you know, the first version of Virgil and then subsequent versions. This is something that, you know, it's kind of been part of your journey through every stop. And I definitely want to look at some of those other stops. But can you talk a little bit about how Virgil has like grown and evolved and maybe even basically compiled for AVR, I called it Virgil 1. Virgil 2 was basically that, but like it had type parameters. And so that's basically Virgil 2. And then around 2009 or 2010, I basically threw everything out and started over. And I became Virgil 3. And it's evolved a lot since then. It kind of went on, I was kind of busy from 2010 until about 2018.

Right. But it's evolved a lot since then. I've added some features. And now it's kind of like much more a general purpose language. It's got like, you know, a compiler with multiple backends. You know, it's got like you know a compiler with multiple backends you know it's got a garbage collector it's got like support for jit compilation all kinds of things like that so it's definitely been the kind of thing that i've carried on the longest and invested the most over time like

integrated over time like lots of nights and weekends working on that and it definitely shapes like it's definitely the lens through which i see programming languages in compilers and maybe comparing and contrasting with some other languages um what are kind of some of the attributes of virgil 3 today that maybe make it unique amongst other languages? So for one, I know this is kind of passe these days, like kind of falling out of favor, but I still think that garbage collection works for systems languages. It's not a hill that I'm going to die on because I don't think it's the most important thing in systems. But what Virgil is unabashedly a systems programming language. It's not in competition with OCaml or Haskell or Scala or any of these languages that are for writing real applications. It's for writing systems.

And by that, I mean this is the kind of thing that you would write a virtual machine in. I'm doing that now because I can't stop myself from writing virtual machines like an addiction that I have. But I definitely envision it as like this is what you write the lowest level in. And so it needs to have features that allow you to do things that are icky and dangerous from the concept of like a pure programming languages person. Like if you're really deep into type safety and type theory, then you might be put off by the fact that you can use a pointer to the stack and like walk the stack that way, or that you can memory map something and you have a pointer to that, or that you can call the kernel. So that sets it apart. Just being a systems language means that you have to make some pragmatic decisions about what features go in.

It also means that it has to be pretty efficient and have a native implementation. So that was something I learned from both the OVM and Maxine projects is that, you know, we're building JVMs in Java. Java is not a systems language. And it was like you basically have to build lots of new things. You have to basically bootstrap the language in order to make a system with it. And yeah, virtual is the kind of thing like it should be designed for that. That makes sense. The what, what are some kind of like applications of Virgil?

Is it mostly been something that you've written programs in, or is that something that other folks have taken and kind of started to build systems with? I don't know if anyone has built a serious system with it, but people have done like some toy projects. People have done like, there's like a password combinator library out there. There's like socket libraries out there and a few things like that.

It's mostly been me building systems with it, you know, like the lone maniac in the cathedral, Oregon, right? Candlelight and the Cape. Uh, but so I built a wizard, which is a research engine that runs web assembly and is, uh, all written in a hundred percent virtual. There's no C code. Um, there's some assembly code. There's plenty of assembly code. That was another thing that I

had to make a compromise on. I realized at some point that I would never be able to teach Virgil to do everything that the CPU can do because it's kind of hard to design a language feature that I don't know. What about page tables? Are you going to put page tables in your language? Probably not. So it needed to eventually be able to deal with other code that was not Virgil at the machine level. So it was probably about three years ago when I started working on a fast interpreter for Wizard that I added a feature where you can basically just take machine code and be like, this is a function now. This is a Virgil function.

And it has the ABI of Virgil. And you kind of have to know the ABI and follow it. So it's absolutely unabashedly an unsafe operation. But you can make machine code and then get it into the system as a Virgil function. And you need to be able to do that because there are instructions that I'm never gonna convince my compiler to emit, like pop count or all the SIMD instructions that are necessary for implementing wizard.

So I realized that a systems language actually needs to be able to interface to other code at the same level of the machine. And so what better way to do that than just like be able to like new up some machine code and and kind of get it in there dynamically and that that uh yeah that means that you can build an interpreter you can build a JIT you can build all kinds of things eventually I mean it doesn't link against the other native code now but that's much easier.

Yeah, and it had the ability to call the kernel, so you can call the Linux kernel, any system call you want, anything you want to pass. And that was also a lesson for me too, because starting out trying to bootstrap the language, eventually the runtime system has to do something. It has to call the kernel to do io or to do memory mapping or memory protection or signals or something like that and originally i was like well okay i will just enumerate each of the kernel system calls that i need and then i'll add them

to the language one by one or like maybe some linux specific library and you realize that you need to go through a bootstrapping cycle where you add it to the library. If that's in the compiler, then you have to add it to the compiler. You have to bootstrap the compiler, make it a new stable compiler before you can use it in the runtime. And I was like,

this is not sustainable. How about we just give you the syscall instruction and you can just call the kernel with whatever you want. You can pass whatever you want. The language doesn't know what you're doing. It knows you're calling the kernel. language doesn't know what you're doing. It knows you're calling the kernel. It doesn't know what system calls mean. And then from there, you build everything up.

I think that was a high leverage pragmatic decision so that like now all the libraries and things that do like memory mapping of code and protection and stuff, it doesn't need any additional language support it's like the one big hammer is there it is a purpose-built hammer like the syscall thing is not for doing other things and so i've been kind of following that playbook of like figuring out what is the pragmatic big hammer thing that i know when i add it it has a purpose but it's not it's not like enumerative in the sense of like, I have to go through every single possible use case because it, you will make it, you'll either make a mistake and change your mind. And then like just the, the maintenance of going around that bootstrapping cycle. So there's things like that, that I've been adding recently to make things more safe,

but doing it in a way that's not, that doesn't like kind of, you know, follow that enumerative model. Makes sense. And, you know, being useful for these types of systems, is your goal kind of to further research using the language or is it, you know, eventually to have more like general adoption and, you know, get it used in more meaningful contexts? I do like the idea of more adoption, but I definitely don't want to be beholden to users that have demands.

That does complicate things, doesn't it? So please use it, but don't ask for anything. Use it, love it and say thank you. Yeah, well, I mean, that's obviously not a tenable position. I do have issues and respond to issues and things like that. But I'm definitely not looking for success. I'm not looking to compete with Rust. I'm not looking to compete with Zig.

I'm not looking to make people use it for their next production system. At the same time, I do enjoy having people use it and do cool things with it, contribute back. Most of the contributors have been students, but I've had some random people that have contributed things here and there. My goal is web assembly research. And so everything that I do at the language level has to be narrowly focused on making wizard better just because I don't have time to constantly cycle on language design issues. I could go on a long rant about how that's not really rewarded properly in the PL research community, but I won't. Fair enough, fair enough.

Well, okay, so you mentioned WebAssembly a few times now. We've kind of like gone up to present day with Virgil and talked about Wizard a little bit. Let's step back to, you know, you mentioned you're pretty busy from like 2010 to 2018. In 2010, after being at Sun for a few years, you joined Google. What was the decision to do that like, and what kind of drew you to Google at that point? I had a couple friends that worked at Google, in particular a climbing friend, Ken Russell, he's like the head of, he's the head of, what is it called?

Web GPU at Google now, but he was in charge of WebGL. So he's a graphics guy, but he had also worked at Sun. So he had moved to Google maybe a year before, maybe two years before. I left Sun because Sun got acquired by Oracle, and I didn't like that idea. I figured that might be the case given the timing of it. That's full disclosure. In retrospect, all the friends that I

did have in Sun Labs seem to have done well. They've gone on to move from Maxine VM to Growl VM, which is phenomenally successful and a really cool system. I have tons of respect for that system and I keep in touch with them but I moved to Google because it seemed like a fun opportunity my first couple years at Google

I worked on an internal system it's not super secret but it was not PL implementation related so I had a lot of fun learning on that system and in the team but it wasn't like my primary like career path and so I moved to Germany in 2013 to join v8 and to work on javascript because you know I have some compiler expertise initially it was like hey let's work on making Crankshaft, which was the optimizing compiler in V8, better. And at some point, so just a little bit of history about V8. So V8 was developed in Denmark by one of the VM gods, also a friend, Lars Bach. I have immense respect for Lars Bach and Kasper Lund and Slava and all the other team, Eric Corey, all the people that developed V8 initially, they transitioned to doing other things and moved the project to Munich.

And so my manager, Daniel Clifford, took over. I had a personal friend, Hannes Peier, who's now the tech lead of V8. Basically, he was an intern with us on Maxine. It was all personal connections. And I think that's a good thing in my career path. I don't know how to give advice to other people other than make friends with the right people. It certainly was not all me being awesome at my job. Maybe some of both. Yeah, a lot of luck. I have to admit, like there's a lot of luck

and I'm very grateful to the people that gave me opportunities. JavaScript V8, like it was an eye-opening experience. I mean, I worked on JVMs for quite some time and compared to Java, JavaScript is a phenomenally dynamic language. Right. Like, it does a crazy amount of things that would be just absolute sins in the JVM world.

Like, I don't know, like some dynamic scoping mechanisms and, like, eval and, like, all kinds of things like that. And prototypes is, like, so I kind of had my innocence wrecked by finding out how all the things work in JavaScript. And that was actually quite a good experience in realizing there's a lot of compromises in the sort of like, oh, let's make this a nice bytecode for a VM. Well, JavaScript just has a lot of challenges because of just some historical accidents and also just kind of the way people use it in practice. So the idea was we're going to make Crankshaft better, and we're going to make it generate better code. So I worked on that probably almost a whole year, like working in the team.

And at some point we decided, oh oh let's do a new compiler so every project requires justification and so i remember giving a presentation where i had to explain all the downsides of crankshaft and in retrospect i feel kind of bad about that presentation because i really had to dump on it and And those people actually worked really hard on it. And I still consider those people friends too. But I had to be like, you know, Crankshaft's got bugs. Crankshaft can't do this. Crankshaft can't do that.

And like, I could, I mostly believed it. Right. But it was enough to give like justification to work on a new optimizing compiler. And what were some of the things that you enumerated in terms of how crankshaft worked and and why that that wasn't um you know meeting the certain goals that you all had at that time it it was a lot of things that could probably have been fixed in the long run with enough um tinkering and engineering on it like for example it had a lot of bugs. Crankshaft shipped really quickly. Like they did an amazing job getting it to market fast, but it had a ton of bugs in it, in particular

around deoptimization, which is a hard thing to do. And like the team was getting pretty sick of fixing deoptimization bugs. It couldn't do anything that was more complicated to control flow than basically just folding branches and deleting dead code it was just something about the way the basic blocks were represented was like screwed up and it was like hard to fix anyway uh register allocator was kind of a nightmare although to be honest uh a variant of that same register allocator is still in production it's just an even more nightmarish version of it so it's just kind of kind of uh it's just kind of the way things are and the register register allocator in particular had this pedigree which might be an interesting side note like okay so uh so thomas vertinger uh the guy in charge of GraalVM, was our intern at Sun Labs in 2009. And he ported C1's register allocator to Maxine.

So he rewrote this thing from C++ into Java for a compiler in Maxine that was called C1X. Thomas was an intern on the V8 team working on Crankshaft. And he did exactly the same thing, porting the same register allocator to Crankshaft. I mean, he rewrote it from scratch. It's not like he copied the code or anything. But it's basically the same algorithm. It's a very small world because that register allocator is based on Christian Wimmer's master's thesis because he's the one that implemented it for C1, which is the client compiler in Hotspot. And so that register allocator was the one that was in Crankshaft and eventually got copied into TurboFan, which is the optimizing compiler that we built

next, and eventually got mutated through several rounds. And I say it was a nightmare. Like, I don't want to insult anyone. And Thomas and i had had beers and laughed about this many times but yeah it was like very hard to understand it it had lots of bugs and it was the kind of thing that like nobody wanted to touch it nobody wanted to tune it and if you got stuck with that duty people took pity on you uh so that was one of the things like when you're like, when you're trying to make a pitch to management, like if you have some part of the systems that's like that, it's easy to point to it and say, look, we need to rebuild this. So that was one. And yeah, like we want to do more optimizations. And I mentioned C1. And there's C2. So people still refer to them and see as C1 and there's C2. So people still refer to them as C1 and C2.

Those are two compilers inside of Hotspot, which is now OpenJDK. C1 is the client compiler. C2 is the server compiler, what they call the server compiler. And we took some inspiration from that dichotomy of those two compilers because C1 is a basic block SSA compiler with, you know, basic control flow graph type deal, straightforward optimizations. And C2 is like this weird C of nodes thing, which is this like relaxed program dependence graph that was, I wouldn't quite, I would say definitely, excuse me, brought to fruition by Cliff Click and the guys in the Hotspot team in the late 90s. And we're like, well, if we're going to build a new optimized compiler, we should do what they did, which is build a C of Nodes compiler.

And my manager, Daniel Clifford, we just call him Dano, he'd also worked on a C of Nodes compiler. And basically, we just started down that path. So TurboFan, we just called him Dano. He'd also worked on a C of nodes compiler. And basically we just started down that path. So TurboFan, we just started building a C of nodes IR from scratch. And that's how it kind of came to be about, came about. And how does the C of nodes, how does the C of nodes compiler work? It'd be difficult to explain it without having a diagram, but, um, imagine the difference between, uh, so I don't know how much of your viewership has compiler background, but like compilers before SSA and compilers after SSA,

they look quite different. SSA made a lot of things easier because it makes data flow explicit in the program. It gets rid of a lot of problems that happen because of like, oh, you're optimizing this expression. And then later something overwrites one of the variables involved in this expression. So SSA gets rid of that. It shows the data flow. The C of nodes kind of does the same thing, but for effects. So like it really shows you only dependencies between instructions. So it no longer has like orders of instructions in basic blocks. So everything has to be ordered by some kind of dependence edge. And so it inherits the property from SSA that there's data flow edges. So you will see like a little SSA graph inside of the C of nodes thing.

And so that kind of makes sense. Like if you have a plus, the two inputs to the plus are going to be edges. What the C of nodes does is that generalizes that to both effects. So like changes, updates to memory and also for control. So if you have a load in store in the program, they will go to memory, maybe to an object, maybe to an array or whatever. Those will be actual edges. So the load, the thing that represents a load is going to have an additional, not only what object or whatever field or whatever you're loading from, but also what kind of version of the world, which is your effect edge.

And usually loads also need to be guarded by some condition because there might be a safety condition. So the graph also has control edges. Control edges basically express the fact that some kind of code needs to check something in order to make it possible to execute something. So the C of nodes kind of generalizes SSA and then effectively renames memory and also kind of renames control. And what that means is that the graph or like the code is effectively way less constrained. So there are many new legal orderings for the code. It also, so the benefit of that is that you can reorganize the code. And as long as the dependencies are met, then it will compute the same thing. The version of the code that you generate will compute the same thing.

So it's definitely generalization. It's like more graph theory. And that also means that optimizations that sort of, they're called forward data flow optimizations where you know something about your inputs. Like imagine you have X plus zero. Since you know one input is zero, you can strength reduce that to just X.

Or if you are like comparing two things and you know that they're both greater than zero, then maybe you know the outcome of the comparison. So this is something you know about the inputs and therefore you can include something about the outputs. All the four data flow problems that you would ever have, you can do them all at once. There's no phase ordering problem for that. And that applies to branch folding. So if you have an if and the

condition is now known to be a constant, you can fold away the if and you can get rid of one of the branches. So this is the most elegant thing about the sea of nodes is that you can throw all the optimizations into one big pile. You crank that, you crank the handle once. Well, actually you crank it until a fixed point and then the graph is like optimized.

There are some caveats to that. Cause you can have an, you can have an optimization that does one thing and an optimization that does the opposite. And they both are forward data flow. And then like that kind of fight. So you have have to there's some tuning that you have to do like you have to be careful that it's kind of monotonic in some way but this is like this is often awesome so if you're a compiler person i think the end of the road in thinking is c of nodes so if you

haven't quite reached the enlightenment of c of nodes, you will. How common is it among compilers for various languages? It is actually extremely uncommon. Okay. That's what I was thinking. Which is unsurprising. Right. So LLVM is basically kind of like a C1 on steroids. It's a CFG-based SSA compiler.

It just has many many passes as many uh passes that are far beyond what like um you know crankshaft would do or c1 would do so lvm is actually a really awesome compiler it just doesn't go all the way to the c of nodes uh enlightenment in my opinion so we're getting into an opinion territory by the way so this is not scientific fact we'll make sure to put a disclaimer on the the show notes so there's also so there's like this peak of enlightenment too and i think uh i think we there's something beyond that which is all the downsides to the sea of nodes which we we hit every one of those branches on the way down so there are people on the team I know for sure to this day who think the sea of nodes is the worst idea ever. And they hate me. I mean, everybody's still friends. I mean, we have tons

of respect for each other. But I definitely see the downsides too. I would love to write about the upsides and downsides to the sea of nodes. But just for completeness sake, just not to leave people hanging, it really is not very good for speculative optimizations. And I think the Graal people would agree with that. I have talked to Thomas again over beers many times about what the drawbacks are. It requires a scheduling pass that is relatively expensive. So you have this very relaxed graph of what

the computation is. You actually need to turn that into code at some point. So you need to schedule it. And so you need to find an actual order for the code. And that order that comes out is actually a CFG. So you can kind of go from a CFG into the CF nodes and then optimize and then go out and then generate code. And so like the rest of the backend of a compiler looks pretty much the same once you've scheduled the code. But that scheduling is relatively expensive and that scheduling can totally screw up the efficiency. So if the scheduler is bad, it can be really bad.

So it can make your code worse. So if you start with a control flow graph that's actually pretty good and turn it in the sea of nodes and you do something and then you schedule it you can end up with code that's worse than what you started with and that can be bad so you need to have a really smart scheduler and so the scheduler becomes like the game where all the where all the like smarts go and that's, you're now in, like, register allocator nightmare land where if you don't understand what the scheduler does, you're going to hate life. So you might be, like, and this is actually a problem for team dynamics because, you know, somebody comes in, they want to do an optimization. They're not, like, fully up to speed with how the sea of nodes should work.

They do an optimization. Something goes wrong. Maybe an edge is missing, and then the scheduler takes it and it's just like, and now the code is maybe bad, maybe wrong because an edge is missing or something like that. So that's a real downside, just the sort of onboarding and also just like the amount of knowledge

about the system you need to have to understand what the output of this thing is going to be like. I mentioned the speculative optimization. So speculative optimization, what I refer to is basically you are harvesting some kind of dynamic information that you've gathered about the program execution, which is extremely common and absolutely required for JavaScript. So you might put a guard into the code, which is like, check this condition, like this variable should be this type or this object should have this shape and then the implication of that check is that the code below it everything that's dominated by that will now get simpler and the

issue is the sea of nodes like there's no such domination condition i mean there is but it's embedded in a graph that you need to compute and it with is expensive and so you can't just like willy-nilly start sticking in guards and Have it just do the thing magically so that that's a problem It's a lot bigger too It can be a lot bigger because you have a lot more edges and everything you do in compilation It's gonna be proportional to the size of the IR So more edges more nodes nodes, more time.

And so everything gets slower. So if your IR gets 2x bigger, it's almost certainly you're going to have 2x slower compilation time. So that's a problem. So usually C of nodes compilers are a lot slower than C of G compilers. So that can be an issue if you want to have like a kind of a mid-tier compiler. So I see that downside too. than CFG compilers. So that can be an issue if you want to have like a kind of a mid-tier compiler. So I see that downside too.

I don't know how all this plays out in the fullness of time. I started to think that maybe a compiler pipeline should be able to run without the C of nodes. That was something that we tried to do in V8. Many people wanted to get completely get rid of the C of nodes in TurboFan. I don't think that's totally possible because there are many optimizations that get so much better with that again turning the crank all at once um i don't know how this plays out in the full fullness of time i think the next optimized compiler i attempt will have the sea of nodes in some phase

but may not rely on it completely and i did definitely in the beginning in the first maybe one or two years of turbofan that was definitely in my head like this thing is going to be like sea of nodes everywhere it's going to be this like compiler utopia it didn't really work fair enough fair enough so we went down kind of the the technical path i think i i sufficiently derailed us on the the sea of nodes but from a kind of the technical path. I think I sufficiently derailed us on the sea of nodes. But from a kind of like progression perspective, you know, V8 is obviously at that point already, you know, quite popular. It's using Crankshaft. You'll want to introduce this new compiler in TurboFan.

What was the process of moving to TurboFan? And is TurboFan still the compiler that's used today? It sounds like it is. It is, yeah. The process was long and a hard one. So the issue was is that we couldn't just stop everything because we're in a competitive environment. You know, Firefox is, we were neck and neck.

Sometimes they were ahead and sometimes we were ahead. JavaScript core was neck and neck. So we basically, and then there was Edge. So Edge edge is now gone but they had their own javascript engine with chakra which was actually really good and so like in this competitive environment we can't just stop for six months or a year and then like just switch to this new compiler which like any compiler is going to be full of bugs and we have to stabilize and all this thing. So we developed like we had enough people on the team that we could keep Crankshaft going and have TurboFan. And about a year into TurboFan, was it about a year? Yeah, it was already about a year into TurboFan.

The VA team was notoriously insular, but we started having some contact with Firefox people in particular around Asm.js. So this is like a whole monkey wrench that got thrown at us, the Asm.js thing, because Firefox, the Mozilla people came up with this idea of like a subset of JavaScript where you could compile into it. And it was kind of like a low level machine.

And so they were compiling C++ into Asm.js and they had like a custom thing for Asm.js that made it go really fast. And so we were like terrified that this is going to take over the web. And so TurboFan was in some ways a response to that. Let's make, that was like the first thing we're thinking, okay, TurboFan should be able to do really well on asmjs code and so we like did that turned out to be easy and hard we did it the hard way um it was an easy task that we did the hard way in particular like we wanted to have all the

analysis that you need to make the asm.js optimizations be general enough it would kick in for regular javascript so we thought really hard about how to make the numerical analysis and the range analysis and the representation analysis just do asm.js as a special case. And I would say that worked about 85% well. It was 85% well enough that it was better than Crankshaft that we could turn it on for Asm.js and then we could get some skin in the game with TurboFan. That took probably about a year, maybe a year and a half to get TurboFan to that point.

But then after that, we started to talk into Mozilla and it turns out that they're not, they're not actually bad people. Actually, they're actually really cool people. And so we started working on WebAssembly because, uh, I remember I had this meeting with Luke and he's like, yeah, we're thinking about maybe, maybe we should have a bytecode. And I'm like, Hmm. And so that's how web assembly got started it was like at the end of 2014 luke and i were talking and you know luke and alone and dan had worked on asmjs

and then once i got in with like yes let's do a bytecode and my manager didn't kill me for working with the mozilla people uh then WebAssembly got going. And by about early 2015, or maybe, yeah, the spring of 2015, on the Google side, the native client, people who also had a relationship with Mozilla, they got interested. So JF Bastion and his team started to be involved in WebAssembly. And then it became, you know, Luke did some diplomacy with Microsoft, and I did some diplomacy with JavaScript core people. Also, again, personal connection, Philip Hislow was a student at Purdue, and I was his TA, and he was effectively the tech lead of JavaScript core at that point.

We basically bought him some beer and talked him into web assembly and that's how it got going in 2015 and probably by the end of that year i was so deep in that that my managers just asked me like why don't you just do that full time and then we'll work on javascript on the side you don't have to worry about that. And I'm like, that sounds all right. Yeah. And so when you look at WebAssembly, what is different? I mean, I know like fundamentally what's different about, but what is, what allows WebAssembly to have better performance or be more suited for the web than JavaScript? And then there's also, you know, this whole aspect of WebAssembly where, uh, the, the portability, you know, going back to, to your Java days, perhaps, uh, the portability is really important here, uh, both in terms of, you know, you can run

WebAssembly in lots of different places, but also you can at this point, at least compile almost anything to WebAssembly. Um, so. So what is kind of like the mix of trade-offs and how did that influence the design decisions for WebAssembly? Yeah, I think Asm.js came close. So I won't go into the details of it, but it came close, but there was things that you just couldn't do encoding things into JavaScript. And so it really needed to have a purpose-built solution. And so WebAssembly was basically like, okay, if we're going to design a bytecode from scratch,

let's do it and make sure we solve this use case. The use case being compile native code to it, like compile existing native code to it. And so we built a thing that had like one purpose, which is basically be a better native co-target than Asm.js and be portable, which native client and even portable native client weren't. And there was, I don't want to go into that long history, but the native client people are on board. It took some convincing, but then once everybody was like pulling the same direction, we had the same mission. We just started going down the list of things that we needed to like fix to make it better. And so, you know, it had to be fast to parse and to decode it

and to validate it, right? It had to be free of undefined behavior. We had some long discussions about that. It took a while to get to that point. And it had to obviously run fast. So we had to all know that we could get good code out of it in the end. And we all cheated. We all used our JavaScript backends in pretty much the same way to make WebAssembly fast. So WebAssembly is like unique in the sense that it shipped with the optimizing tier first. Like I even wrote that in one of my papers. All the tiers came completely backwards. We started with optimized code, and then we worked at baseline. But that was the value add that had to be there for it to even have a reason to exist. So we had to be close to native speed.

And so we all had our respective good compiler backends, and so we could just effectively just get the WebAssembly into the backend as fast as possible. Makes sense. And what about WebAssembly, like looking at it, from a more detailed perspective, makes it easy to parse and faster, close to native speed, like you were saying there? So for one, I think the file format is really easy to parse from a binary perspective. It's compact. There's a tradeoff in those two dimensions. You can't have both at the same time.

It could be slightly easier to decode, but not as fast. And it could be faster to decode, but harder. In terms of the runtime, I think what makes it easy to make it fast is that it's just a low-level model. It's just like you have this big memory. It's like a byte-oriented memory. You have loads of stores into it. The code itself. So I had worked on Java bytecode long enough and seen enough Java compilers to know that we should not have arbitrary jumps because it sucks to verify that. So that was one of the, the structure of control flow that WebAssembly has that was absolutely motivated towards making the verification better.

So like the consumer having a type system, which is really simple, made it easy. So you don't have to have like a, you don't, you don't need to have like a fancy type system to verify WebAssembly code. So that makes it like again easy to parse and validate and also easy to generate good code for so instead of having like some dynamic typing thing where you need to have feedback so javascript you basically need to have type feedback you need to have some dynamic information to make it fast web assembly you don't. You can just use static types. And so it could go straight into the backend with no feedback, no, you know, PGO or anything like that

and get good machine code out of it. So I think those are the key things. We had some tricks to make bounce checks for memory. Like we already knew how to do that um unnecessary so the the one weird trick that makes web assembly memory fast is that it's 32-bit inside of a 64-bit address space and so you just map out all the things that are inaccessible so you have like an eight gigabyte range and then you map out the things that are not accessible and then since you have a 32-bit index you can't go beyond the end of this 8-gigabyte thing. And so you don't have to do a balance check. The hardware is going to do that for you.

It's going to do it through the virtual memory system, through the MMU. So that was like one of the weird tricks that we knew that would make it faster. You can't really do that for Asm.js. I think Firefox was doing something, but because of JavaScript, it's kind of a pain. With WebAssembly, we're like, no, we can fix that. You're out of bounds. You trap, which is reasonable because your program probably has a bug. So we could pick the semantics that we wanted in order to make it efficient and also nice and portable.

So that's the advantage that we had. How about from the – this is less about performance, although I'm sure it does have, you know, somewhat of an impact. But one of the things that I think, in my own experience with WebAssembly, for example, that can be, you know, somewhat challenging is passing data in and out of a WebAssembly module, right? And there's, you know, been work, I think the component model and some of those sorts of things are working on making that perhaps a little bit easier, a little more structured and how that works. But like at the fundamental layer, right, you're only passing primitives back and forth. So you find yourself a lot of times, either, you know, doing something like writing into the WebAssembly module memory, and then passing, you know, a pointer to that memory, to a function in the WebAssembly module, or, you know, perhaps doing, you know, allocating something in the WebAssembly module, and then passing out, you know, to the runtime environment, hey, look here for this

result. What is kind of like, like, I think, from from a user perspective that can be somewhat cumbersome, but it also is a lot simpler, uh, to work with. Um, what was, I know, I know the, um, multiple return values was kind of like a big sticking point in various parts of the design for WebAssembly, I believe. How does all of that kind of like interplay and how did y'all end up at that model? I think it's all necessitated by the fact that's low level so you're in inescapably at a machine level kind of uh game right so you have bits and bytes and you have offsets into memory um the component model introduces higher level types

and those higher level types need to be lowered so it has an abi which defines how that lowering works um you know nine years ago when we were working on WebAssembly we had started to think about different ways of doing that. We had interface types proposed and the component model is essentially an evolution of what was interface types proposed maybe two or three years into WebAssembly. But we knew that we couldn't solve too many problems at once. So we couldn't really solve the high-level interfacing problem at the same time we're trying to get this binary thing to work, like this little machine code thing to work.

So it's partly an outgrowth of just solving the problem that we knew that we could solve and keeping the scope creep down. And that, I think it's kind of good in the sense that like it's it's like a it's like kind of like non-fiction right it's like bits and bytes that's what the machine does and we're not dressing it up to you for you so you have to kind of work that out for yourself and like the layers above should kind of deal with that so i'm big on like software architecture and layering like the layers above should figure out what data encodings you do and and since we just defined this like instruction set and like a low-level memory and while you can use it in javascript

and you can make your typed array aliases around memories uh then you can figure it out right like right absolutely do you feel like it's uh fairly i mean obviously you've worked on uh lots of different bytecodes lots of different vms do you feel like fundamentally like the the closer the instruction set or the virtual instruction set if you will of the the bytecoder or the virtual machine maps to you know if it maps like one-to-one to hardware instructions is that generally where you're you're gonna find the most performance yes and no i mean certainly in terms of operations like the the operations on data values like if you have plus minus divide and pop count and like a business a bunch of things you want to have a hardware instruction that does the same thing. The difference comes down to like the data flow. So like WebAssembly is a stack machine,

JVM is a stack machine, CLR is a register machine. Every CPU, almost every CPU in existence is a register machine. And so the encodings of the data flow is different. But compilers deal with that, right? Compilers effectively translate the data flow from one representation to another. So yes and no. I think one-to-one is good operational-wise, but in terms of data flow and control flow, you can translate. So what is it that makes Stack Machine so common for languages that have a virtual machine? Good question. I think it's an easier encoding for one so it's just like you know you don't have to figure out oh what if i run out

of registers so like what do i do do i have like 256 registers or 16 registers or whatever right to decode that crap and your interpreter it sucks i've written a lot of interpreters right it's like easier to do a stack machine for one and people have debated this question for many years um i don't think there's a definitive answer um i can it does matter we just settled on a stack machine partly for compactness i think that was the right decision i don't think it would have been, I think it would have actually been a lot messier to try to do a register machine. I think that WebAssembly also has kind of like the, well, it's not fully unique,

but the degree to which it's being like streamed around, right, the code is being sent over the network is different from a lot of other contexts in which you're designing a language, which I think has also had some interesting implications on the design, you know, why certain things were structured the way that they are and that sort of thing. So moving kind of towards some of your end of your time at Google, I know that when Spectre and Meltdown hit, which have been talked about multiple times on this podcast at this point, that I think the V8 team was kind of all hands

on deck, if you will, to kind of address some of those mitigations and you were involved with that. What was kind of the, first of all, I guess, like, what was the reaction when it was first announced? Or maybe you all had, you know, some information before it became more public, I imagine. But what was kind of like the reaction like, and then what were the subsequent steps in terms of starting to form a mitigation against it? Yeah. Oh, man. So we definitely knew beforehand. We were read into this. It was probably the most logistically complicated security issue, certainly, that I've ever been involved in, maybe ever, considering how many people were involved. V8 was read into it relatively late, but that's really not that important. the timeline is probably also not something i should talk about just because of indies and things like that sure but i know that when i found out i was like oh god yeah i think that was fairly universal reaction

uh yeah i remember thinking is because it's such a simple thing, right? Like, Oh, and this predicted branch can go wherever. And then, Oh, well you can smuggle things out through the cache. So once I realized the implication of that, uh, there was probably several weeks of, uh, not knowing, like basically freaking out on a team level but uh we did systematically investigate like what are the what are like sort of the parameters of this threat like okay how easy is it to create a gadget for example like once we got occupied in like the sort of you know understanding the the sort of like nuts and bolts of what we're dealing with. And so that kept us from like getting too frazzled.

And we started thinking about mitigations like a couple of weeks in. And we worked with like all the teams in Google. We worked with all the teams across browsers. So like we were having like weekly or biweekly calls amongst the browsers to talk about how we're going to deal with this. So it was definitely a coordinated thing. It wasn't just like us on our own. So we shouldn't take too much credit for it.

And of course, we worked with like teams inside of Google about it. So I don't know how public. Well, the record of our commits is public. So if you look at the commits to V8, you can see a few, which were mitigations that appeared before disclosure, which had stuff like, I don't know, check this thing, no reason given, you know, like very short commit messages. I think for WebAssembly in particular, which I feel grateful in retrospect for only being responsible for the WebAssembly part of it, the mitigations were pretty straightforward. Like we did a mask of all the memory accesses.

And that was like easy to see that that was going to be bulletproof for one particular kind of thing. But, you know, I was working with other people on the team that are working on JavaScript. And so we had to think about how our compiler mitigations are going to work and how universal they're going to be. So the people in the Spectre paper that we published a few years back, all those people worked extremely hard to figure all of that stuff out. And I think our strategy at the beginning was,

the compiler is going to be able to mitigate this we're going to be able to get back to the model that we had in our heads always which is that one javascript computation is separated from another javascript computation through software safety checks i think at the beginning everybody thought you know once we started to get into the swing of mitigating things in the compiler like we're like, yeah, this is going to work. Even though variant two, like the indirect branches, was like, this might be harder than we thought. But I think we were quite hopeful. And most of the JavaScript people were quite hopeful that we would get to compiler mitigation saving the day.

That pretty much all fell apart when we started looking at variant four and other variants coming down the pipe. And we're just like, okay, this is not going to work in software. And thankfully, Chrome was a bit ahead in terms of having a multiprocess architecture for various other reasons. And we saw the writing on the wall, and we're just like, the only way that we're going to be able to do this is to get JavaScript away from anything which is dangerous. Like it shouldn't have anything access in another JavaScript computation

that could be sensitive, and it shouldn't have any access to anything in the browser process, so there already is a separation between the browser process and the renderer process where JavaScript lives. So there's already some protection, but we knew that the only way this is going to work is to have process level isolation. So yeah, I think the whole thing, you know, that arc took more than six months or eight months for us to really realize that it was going to be inadequate. It's a very unsatisfying thing too, because throughout

this whole arc of my career, like software enforced boundaries and virtual machines like that's my whole bread and butter. Right. And it's like side channel vulnerability means it's all like not possible. Right. It's not like you can crash another computation, but you can see everything is doing and like again like the activities of like trying to figure out the um you know how long of a speculation window do you have how many uh attempts do you need to read a byte of memory through a side channel like how many cache line misses can you deal with or like how do you induce

cache line we basically came up with a number that oh you could read about four kilobytes a second like pretty fairly reliably i mean there's noise in the thing too and uh i just realized this like that's like the same as having a dial-up modem to whatever part of memory you want. Right. Full circle. My first internet connection was like a 336 modem. I'm like, well, you just have a modem connection to anywhere in your process memory. Right. So I assume this isolation that was necessary for full mitigation

had a performance impact as well. Oh, yeah. And people weren't happy about that. People are still complaining about that, the mitigations for kernels that you like, I don't think you should be able to shut off, but they want to shut off. This is another thing that I learned from Spectre is that, I mean, we've been really pursuing performance at all costs. It's the number one priority for too many of our systems. And security is like, you know, a distant, distant goal down there. And it turns out that in many situations, in an uncomfortably large number of situations, those are completely opposite of each other.

It just so happens in this case case that the performance you get through speculation and number of other optimizations that happen along with speculation give rise to exactly this security problem right and i realized that also in a broader sense in that like side channels appear anywhere you have shared resources shared resourcesared resources are all about efficiency. It's all about not necessarily performance, but like saving memory. And so we're constantly going to be in this tug of war between performance and security. And maybe Spectre and Meltdown scared people enough and made that trade-off stark enough that they can see it they will at least make more informed decisions about that but i'm not super optimistic because you can measure performance

and you can't measure security right so one of them gets optimized and the other one doesn't that's a good point so uh after after um you know spending some time working on these mitigations and that sort of thing, maybe to kind of like round out our conversation here, you've gone back towards academia. And I believe you're now the director of the Web Assembly Research Center at Carnegie Mellon. How did that come about? And what kind of drew you back to getting into more of an academic setting? For me, I decided that I wanted to look a little bit longer term in terms of research. I mean, I liked working on VA. I loved working with all the people on the team. And it's a high impact product,, important product, but it's also a little bit focused on the day to day and tactical stuff. Like I mentioned a little bit about the horse racing between different vendors. Like I didn't enjoy that.

I thought that was, you know, I understand the point of having a product that like leads the market and you need to have these performance goals, but I don't really want to spend my time on that. I want to think about the larger picture. I want to think about like what are some cool things that we could do with WebAssembly? How should we re-architect the software world for security reasons or portability reasons

or things like that? And I think that like being in that specific role and even being in a company, like they really don't have that long a term vision anymore. And I don't know, academia is like a, it's a different, it's a different environment. We can have a longer term view. We can do things that have, you know, they don't have to have an impact right away. We can muse about things. It's a different set of trade-offs, but I really enjoy the energy of academia and like, it was just like so many crazy ideas flying around. So many super bright people, particularly CMU is like just packed with so many people, just the density.

I like that. I like contributing to like the forefront of human knowledge. That's kind of the point of research. So I like doing that. And yeah, I mean, the next generation of students, they need to have instruction and, you know, be shown the ways and like, otherwise they'll end up working on lms or something and like they should work on vms like they should work on machine code and like optimizations and compilers and stuff and like yeah i want to get them excited about it right and you know your your role is kind of like specifically focused on web assembly i guess like what what is your vision for web assembly where do you see it going,

and do you see it growing into a more fundamental part of our computing stack? Definitely. I definitely see... So if I could say my vision, I think that WebAssembly should be the universal software substrate. It should be the thing that's at the bottom of software, in the sense that every language compiles to it. If you're able to run it, then you're able to run everything. So that means that it needs to have a few more features, but it also needs to have more ecosystem penetration and more tooling and stuff like that. I definitely see it as like people have used the term narrow waste. I won't use the term bottleneck because that's kind of a negative connotation. But like having a narrow

waste, where like, imagine like, imagine if all the code was WebAssembly, then a WebAssembly engine, like, we always want to get more programs to analyze, then you can analyze everything, right? There's this kind of of this security model built into it. You have a handle over what a program is doing. You have clear input and output through the import-export mechanism. I think it has so many different dimensions of potential that I think we must do this. And that kind of led me to one of the things that's mentioned on the uh i think it's on the virgil readme um kind of like circling back to this this uh language that's been a thread

throughout um your whole journey um is that i believe it has like a wasm and wasm kind of aspect to it right um because you know when you talk about like web assembly being the substrate for all of our computing, I naturally think, okay, well what's, you know, what's the, the, uh, host environment written in and that sort of thing. Um, but it sounds like, is that kind of the direction you see, um, the host environment going as well? Yeah. So once you have that narrow waste, you have like the clear separation to the above the line and the below the line. And it's not like, so I want to clarify that I don't think that machines, obviously CPUs are not going away. No ISA that we have now is going to go away, probably.

So there's still a role for them. And there will be even more cpus in the future so then below the line like what that host environment looks like obviously it'd be cool if everybody was writing that in virgil and and whatever like doing web assembly vm stuff like that and then above the line they can do whatever they like in whatever language that suits them um i don't know what the exact configuration of operating systems will look below the line, but certainly when you have that software substrate, it makes it very clear what the application wants. And everything below it, you can swap out. That's the whole point of virtualization, right? So you don't necessarily need to have all of the software components that

we have nowadays underneath them in exactly the same configuration because we can swap them out. So whether the kernel is a microkernel or a big monolithic kernel or whether it's running actual native Linux or it's got a Linux emulation on it or whatever like if the cpu has got cherry capabilities built into it and you're only running in a sandboxed environment and it's only giving you a little bit and then there's something running in parallel i think there's so many possible configurations once you have that like dividing line that yeah i mean it's like impossible to predict right right well it's definitely uh uh an exciting future and um you know it, you know, it's great that you now have like the space or the right place to drive us towards that. But thank you for being willing to join today. I think it's been a great conversation.

I'll certainly be keeping up with Virgil and Wizard. And I didn't get a chance before this show, but I need to write something in Virgil and report back on it. But thank you for joining. It's been a great conversation, and I'd love to have you back in the future. Yeah, thanks for inviting me. It was really fun. Absolutely. All right, have a good one.

Cool, you too. See ya.