Behind The Tech with Kevin Scott - Jaron Lanier: Father of Virtual Reality, Renaissance man

Episode Date: March 22, 2019

This scientist, musician and author is best known for his work in Virtual Reality, and his advocacy of humanism and sustainable economics in a digital context. His 1980’s startup, VPL Research, created the first commercial Virtual Reality products - introducing avatars, multi-person virtual world experiences and prototypes of major VR applications. Hear why he is worried the present Internet may be destroying societies, democracies and economies.

I am worried about where we are right now. I just feel like our present internet is destroying societies and democracies and economies. You know, I think it's bringing down civilization. It's bad. We really screwed this thing up. Hi, everyone. Welcome to Behind the Tech. I'm your host, Kevin Scott, Chief Technology Officer for Microsoft. In this podcast, we're going to get behind the tech. We'll talk with some of the people who've made our modern tech world possible and understand what motivated them to create what they did.

So join me to maybe learn a little bit about the history of computing and get a few behind-the-scenes insights into what's happening today. Stick around. Hello, and welcome to the show. I'm Christina Warren, Senior Cloud Advocate at Microsoft. And I'm Kevin Scott. And today we have an amazing guest. Indeed we do. Okay, but before we hear from him, I wanted to ask you a question first, Kevin.

All right, so you have a ton of interests. You're into cooking and photography, and there's a long list of stuff. Do you ever feel like there isn't enough time in the day to do all the stuff that you want to do, like, you know, to pursue your passions? Yeah, this is sort of the curse of my life. I wish there were more hours in the day. Same. Yeah, this is sort of the curse of my life. I wish there were more hours in the day. Same. But if you could stop and spend time during the day to do just the thing that you're most passionate about at this moment, what would that be?

It would definitely be making something. And, you know, with me, what it is that I would make changes over time. So at one point in time, it might be a dish to share with someone together at the dinner table. At another point in time, it might be a piece of furniture. Right now, interestingly enough, I'm sort of obsessed with machining. So I would go to my metal shop and like fabricate something mechanical. I love it. That sounds really cool. I think for me, I don't know, it'd be boring. I'd probably just write, continue, just focus on writing. I think that sounds great. Yeah.

That's what I should be doing, given that I'm writing a book right now. I should go into a nice, quiet place and try to pound out the last chapter of my book. Okay, but maybe do some machining, too. Maybe machine something to machine your own typewriter or something. I don't know. Yeah, maybe, you know, maybe do some machining, too. Maybe machine something to machine your own typewriter or something. Yeah, maybe, maybe. Well, today's guest, Jaron Lanier, is kind of beyond belief in regard to the things that he's accomplished. Like, how does he find the time, speaking of not having enough hours in the day? Yeah, Jaron is awe-inspiring in the breadth of his intellectual interest. When I first met him, I wasn't really

prepared for the full brunt of his intellect, really. Like, I knew just from things that I'd read from him and from admiring him, like, all the way back in the 90s when, you know, sort of there was a VR boom happening at the time. And, like, he was frequently on TV and, you know, sort of during the rest of the tech boom conveying the promise and possibility of virtual reality. But I think the first time I met him, like, I learned that he's got this, like, crazy musical instrument collection, and, like, we share this interest in classical piano, and he is, like, unlike me, like, I am a rank amateur, and, you know, so I spend most of my time listening and very little of my time playing. Like, Jaron's an accomplished performer. He's actually a composer. Like, he's done

performances with folks like Philip Glass. He's recorded records. He's been a session musician. And this is on top of him being one of the best computer scientists in the world, one of the best writers in the world, one of the best philosophical thinkers about our digital modernity. And, and, and, I'm not even covering everything. So, it's unbelievable. Like, I'm totally jealous. I am, too. I am, too. Okay. So, without further delay, let's meet our next guest. I'm delighted to introduce our guest, Jaron Lanier. Jaron is a scientist, musician, and author, best known for his work in virtual reality and his advocacy of humanism and sustainable economics in a digital context. His 1980s startup, VPL Research, created the first commercial VR products and introduced avatars, multi-person virtual world experiences, and prototypes of major VR applications such as surgical simulation. Some say his mind is as boundless as the internet. Welcome, Jaron.

Hey, I am so happy to be here. Thank you for having me here. Awesome. So let's talk a little bit about your history. You are, you know, I say this to many of my guests because I have interesting guests, but you have one of the most interesting careers and, like, set of life experiences of anyone I've ever met. So, like, let's start with you as a kid. Tell us a little bit about how you grew up.

Oh, my. Okay, well, my parents were both refugees from anti-Semitic violence. My mother was a Holocaust survivor. My dad's family was mostly wiped out in pogroms in Ukraine. And your mother, like, was in a concentration camp. Yeah. She was taken at 13. pogroms in Ukraine. And your mother, like, was in a concentration camp. Yeah. She was taken at 13.

And, well, we can talk about the stories, but they're truly just horrible beyond understanding. And my parents met in the bohemian New York of the 1950s. And I was born in 60 in Harlem Hospital, and they immediately fled. And I think the idea, although I never really got a clear answer from them, is that they wanted to be as far from civilization as possible, but not so far that they weren't next to a good university. So they ended up in southern New Mexico, and I would catch a bus across the border every day because there were better schools in Mexico. What was the strength of the schools back then? Because, like, obviously you've had this career in science and music and technology. Did that sort of get sparked there? Well, my mom came from an educated family in Vienna.

She was a prodigy pianist and that sort of thing. She had very high standards. And when I like just one continuous place. There wasn't like some big prison wall between the two. It just was a continuation. So she died in a car accident. I was about nine. And many years later, it turned out that there had been a mechanical flaw in that model of car that's likely to have been the cause. So, there's a layer of tragedy in that, and that she had deliberately gotten a Volkswagen out of a sense of wanting to find reconciliation with humanity.

This was, of course, devastating, but devastating on some levels that were unusual in that era because she was also the family breadwinner. My dad was always kind of the multi-career, slightly weird artist type. He had all these little careers. He designed windows for Macy's and was an architect for a while. And he was a science fact writer for Hugo Gernsback. Oh, interesting. Yeah, he would write the science fact columns in Fantastic and Amazing and Astounding in the 50s. So he knew all the Golden Age science fiction writers.

He was in their circle of friends. But none of that was particularly lucrative for him. And my mom was kind of a systems thinker and figured out how to play the stock market. And she would do it with phone calls from the desert in New Mexico, which nobody did at that time. These days, that's normal. In those days, it was highly innovative, and especially for a woman. And I suspect a lot of the people at trading desks in New York didn't know she was a woman. I think she figured out just ways to do it. At any rate, we suddenly ended up quite impoverished.

Meanwhile, I was hit by my mother's death very, very, very hard. I think she held me very close because of her background. But you went from there to like right around the time you were 12 years old or so, you were taking college courses. Well, so what happened afterwards, my dad was kind of backed into a corner. And what he did is he got certified to teach school so he could get a job because it was the only thing he could see that he could do out there. And he bought a piece of cheap desert land, and we moved onto the land in tents, and we gradually started building a house. He let me design it, and it was this crazy thing with geometry. And it lasted for like 30 years, then collapsed. So, don't let your 11-year-old,

I think by that time I was about 11, don't let your 11-year-old design your house. Although, that's- But it's somewhat remarkable that an 11-year-old could design a house that would stand for 30 years. Yeah. Well, I let my daughter at 11 design part of our new house, but I checked her work. Let's just say, like, I think it's great. Let your kids design the house. But in terms of actually living in the house, check their work. Anyway, it happened by luck that this place we were in in the desert was the perfect place for me. One of our near neighbors was Clyde Tombaugh, who discovered the planet Pluto and was the head of optics research at White Sands Missile Range.

And he started showing me how to make telescopes and lenses, which is the background that I used to be able to make virtuality headsets later on when I needed to be able to do that. So I learned optics as a kid directly from somebody who was as much a master as exists. And had you been a precocious kid before that? My mom had made a demand that I would be so. Like, this wasn't an option. I was informed, you know, you have to have your concert at Carnegie Hall, and you have to, I expect a Nobel Prize and not one of those sissy ones in economics or something, like a real one. And like, that was kind of the expectation. Like,

there wasn't really negotiating room on that. That was how I was raised. And I think a lot of Jewish kids of that generation were raised that way. It was how I was raised. And I think a lot of Jewish kids of that generation were raised that way. It was compensatory for all that had just happened. And there was just this expectation, this is what you're going to do. How different I'd be if my mom hadn't pushed me as much, I really just don't know. So you were 11 or 12 years old, like when you were sort of learning about optics and like getting into science, like this is in the early 70s, right? And so like the personal computer revolution is like a decade away at this point. Well, it's, yeah, it's, this is before you could make your own computer.

This is before the little Altair. So were you toying around with, like, computers at that point, like the big mainframes of the day? So what happened was in order to design this crazy house, which was a mixture of geodesic domes and these other geometric shapes, I had to learn trigonometry. And so I did. I just forced myself to learn it. And it was really tedious calculating all these angles and lengths for this thing, you know.

And I was really interested in computers. When I was 14, I went to a summer course in chemistry. They had like this chemistry summer camp at the university there at New Mexico State. And that was great. And I did all of the things that chemistry professors worry a 14-year-old would do in a well-equipped chemistry lab. And I was directed to do those things in the empty lot next door rather than in the building. And I used to visit it, and there were still a few pockmarks in one. But I think now it finally has a building on it.

But I, you know, I learned to make, you know, flavors and explosives. And I was just fascinated by the geometry of molecules, which was a little like the house I had worked on and all that. So the summer came to an end, and it just seemed somehow absurd to go to school. I wasn't – so by age, I would have been going to high school. So I just – I never went to high school. I just never – I just skipped it. And I sort of enrolled in – so the question is how did I get into the college?

And I'm not sure. I just sort of signed up for courses and either it's possible I might have falsified a document or two or they might have forgotten to check. I'm not sure. And they've had me back to the campus, and I think by now whatever happened would be forgiven. But at any rate, I just started attending. And which school was this? This is New Mexico State University. And how old were you?

Well, I was 14. I would have been 14, yeah. And it happened by wonderful coincidence that because New Mexico State was supporting the White Sands Missile Range, it had one of the earliest good computer science departments. Like actually way ahead of a lot of the fancy places. Way ahead of someplace like Caltech, as I learned later. So, there were actually good computers, good computer scientists in an excellent math department. So, I was like in heaven. I mean, I remember I would just haunt the basement of the math building programming at night.

And what was the first substantial program that you remember writing? A psychedelic graphics thing. And so the way you programmed in those days was on decks of cards. And so this is something that's hard to describe these days. What would happen is you'd have these stacks of cards, and you'd have to take them to this window in a place where a sullen graduate student would take them and then run your thing and then give you another stack of cards that came out. And one of the things about southern New Mexico is it can get quite windy. So, there was – as you approach this cinderblock building with your stack of precious— And order matters. Well, listen, there would be like these clouds of these punch cards just flying everywhere.

So there was a certain degree of cross-pollination between people's programs, I suppose. But it was actually—I mean, I remember seeing actual code tornadoes made of punch cards in those days. Wow, that's incredible. Yeah, so when we talk about cloud computing, like, we had fast cloud computing back then. Low latency. So as the 70s progressed, there started to be a few options for real-time computer graphics, and that really turned me on, and we started to be able to do some very simplistic, like, even, like, just a bitlet of just moving, like, a rectangle of stuff on a screen was still kind of challenging,

but we could start to do it. By the way, do you know how the bitlet was invented? I think I do, but why don't you tell the audience? Larry Tesler, one of the original Jocks Park people, invented the BitBlet, which is just moving a rectangle of pixels around on a screen. And he'd originally done it to control those little color cards that fans hold up at the Stanford Stadium. So it actually started as a stadium ritual before it was applied to it. That I actually did not know. That's fascinating. Sort of like, yeah, the antiquities department here.

So there was a good library at NMSU, which was, like, kind of became my comfort zone in a way. And you could just, like, get lost in these stacks of all different kinds of journals and crazy art books and everything. And there I discovered Ivan Sutherland's work. And Ivan Sutherland was the founder of computer graphics and the founder of graphical interaction on computers and user interface design with graphics and many other things. He's kind of the father of a lot of the experience of modernity. And he's still with us. He's teaching at a little school in Oregon these days. kind of the father of a lot of the experience of modernity.

And he's still with us. He's teaching at a little school in Oregon these days. And I keep up with him once in a while. But he had described computer graphics, and there was like this picture of a cube just painted by a computer. And when I was a kid, this was before we could get a computer graphics machine. That happened like a year or two after I arrived. I would like run down in people, just strangers in the streets.

Look at this. You can make images with computers. I was like so excited. I was like, because there was no internet, you couldn't like just reach. You had to just attack strangers on the street with weird journal articles. So, Ivan in something like 65 had proposed a head-tracked computer graphics headset, and he actually built one in 69. And that was – really, really turned me on because I – ever since my mom had died, I'd had this sort of feeling of incredible isolation from people. I was super socially awkward and a very weird kid. And I just always had this sensation that people were like the stars. Like, you can see the stars. You know there must be interesting things on the stars, but they're too far away to reach. And just people

felt that way to me. And I always imagined that if there was some kind of new medium, some way of sharing dreams, maybe it would be like a starship, you know, where you could reach those distant stars. And to me, when I read about Ivan's headset, I thought, okay, like if you could network these things, maybe you could have that shared dream thing. So that was how I got into this whole virtual reality, you know, whatever it is, this crazy adventure. Yeah. I mean, so, like, you are, I think, rightfully so credited with being the father of virtual reality. And I remember, like, the first time I was aware of you was in the 90s. Like, I saw you on,

like I forget what the name of this. There was a technology network that Leo Laporte was at. And I think Leo was interviewing you. And I was like, oh my God, like this is the most unusual and incredible thing I've ever seen. And it was you. And so like when did like this whole notion of virtual reality crystallize for you?

Like, when did you coin the name? Well, the name, okay, so the deal is Ivan Sutherland called the thing you saw through the headset the virtual world. And he got that from an art theorist named Suzanne Langer, who was writing about virtual worlds in the 40s and 50s. And I thought, well, if there was a networked or social version of it to share a virtual world, maybe that would be virtual reality. And so, I started like writing little things about it and zines and all kinds of stuff. And that probably started, like, in the late 70s or something.

But I didn't actually have any way to do it. And so what happened next – and by the way, in those days it was almost impossible to explain to somebody what this was. Yeah. Like, if you try to explain how quantum computing works today, you get a feeling for what it was like trying to explain virtual reality in the late 70s or the early 80s. It's just really hard. My sense was, even, like, the first time that I saw you being interviewed, that it was, like, in the 90s, like, after, like, you know, the internet had started to sort of take off

with these, you know, like the graphical web browser, that it was still difficult to explain virtual reality. Yeah. Like, it was, you know, like, it's not the easiest thing in the world even now, but, like, you know, there are far more, you know, examples and, you know, like there's a richer continuum, I think, of, like, different types of experiences that people are trying to build. Well, people can actually try it now. Right.

It's not rarefied, so. Like, you don't have to go into a room full of a million dollars worth of equipment. Right, exactly. Yep. It seems kind of almost mundane these days, which is probably good. It's probably a stage it has to go through. But it's sort of, you know, I always find things like these sort of disruptive innovations. Yeah.

So sometimes like when you look at startups, you know, you sort of see a company that's doing something and like they might be a few years early. And just being a few years early is enough to like kill them. Oh, yeah. And like you were 30 years early, maybe? And, like, you know, it's almost incomprehensible to me, like, how you maintain the intellectual stamina for, like, all this time. And, like, you're still pushing on it today. Like, how did, like, what made you sort of stick with it that long? It was so hard. Right. It was so hard. Right.

It was really hard. It was a very, very hard area to innovate in back then. I had good fortune in the early 80s. I made some money in video games. So I had one of the top 10 games in 83? 83? Yeah. And so I suddenly had some capital.

And so a bunch of buddies and I just invested this money from video games in the first VR company, which we incorporated in 84, although we'd been sort of doing garage experiments for some years before that. And that was called VPL Research. If it had more of a ring to it,

it'd probably be better known these days. But yeah, and VPL was quite an entity. It actually, even though it was always a small company, it had a kind of a feeling of being a larger, more influential company. Everybody knew about it, and it was like one of the things in the valley for a while. And yeah, of course, it was way too early, and I – the story of VPL is still emotional for me because I still feel some guilt about – I wonder in my head if I could have done things differently to keep it going. A lot of people thought it was going to become one of the big Silicon Valley companies, you know. And a lot of – there were just a lot of challenges and difficulties that mounted up simultaneously, and it just became too hard a thing to keep going. And – but there were just wonderful people, and we did a lot of stuff.

I mean, I'm still reading today about these new innovations that are actually things we'd already done back then. Give us an example. Well, I just got a pitch in the mail from somebody who has a new way of visualizing tumors in VR that's supposed to help radiologists. And they've just gotten all this money and all this research. And I read the papers and looked through it, and it's almost the same thing as a project that we had done in the 80s with visualization for radiologists. Of course, it's cheaper and higher resolution and more responsive and all that. Like, everything's better now than it could have been back then, vastly less expensive than it was back then. But it's sort of like a history that's forgotten.

But I think the right attitude to have about that is that the purpose of history is to enhance the present. So if people want to experience the myth that what they're doing is entirely new and if that helps them, then the better use of history is to be forgotten. If there's some lessons we learned or some inventions or something that can help them, then the better use of history is to be forgotten. If there's some lessons we learned or some inventions or something that can help them, then the better use of history is to be remembered. But I think obviously forgetting has to be part of life. We can't live in a way that sort of subdues the present moment for us.

It has to be fresh. It has to be its own thing. So I'm fine with the new waves of VR not knowing about the old waves of VR, you know, to a degree. I'd like it if they remember it once in a while, that you all did so much stuff in the 80s. And you had thought about, you'd had this clear still, even to this day, like, we just launched the second version of the HoloLens at the Mobile World Congress in Barcelona. And, you know, in a sense, like, that's just another waypoint along, you know, this sort of vision that you had decades ago. Yeah, I still have a triptych of drawings at home, meaning three drawings that are meant to be seen together, showing two people first in natural reality, then in mixed reality, then in virtual reality. But the term mixed reality is mine as well from back then. It

predates augmented reality by at least a decade. And you did have kind of a roadmap for these things and a vision of what they could mean to people. And, I mean, the first arrivers often can see more clearly because there's just less clutter and less crap, you know. So, I mean, I think Ted Nelson did the first design for digital network, and in many ways, I think it's more insightful and decent and reasonable than many of the ones that came later, and many other examples. I think the generation of Turing and Wiener and these first people had a sense of what computation was all about that, in a way, was kind of deeper than a lot of what computation was all about that in a way was kind of deeper than a lot of what came later. I like reading computer science conference proceedings from before I was born.

Like the ones in the 50s are fascinating because it's people coming upon this thing really afresh and having to think about it without preconceptions. And before there was all this money to be made and all. And it's amazing. I mean, it's actually – a lot of the old stuff reads as quite radical today. Well, and even the stuff that doesn't read as radical, it's, you about Dijkstra's algorithm on my doctoral qualifying exams. And I remember in that moment being irritated at why is this relatively simple thing enshrined as one of the most important accomplishments in computer science because it looks so simple. And then, you know, but you think about it and like not knowing that it existed, like it wasn't obvious at all.

And I think sometimes even like I think you're totally right, like in your point about, you know, like some of these older things reading is very radical today. But like I think even the simple things that look simple from our perspective today were quite radical, like, back then. Yeah, I think we're – I think computer science these days suffers from a couple of problems. One is just there's such a profit motive. There's just so much money to be made and people are so attracted to those, to the things that are probably going to make money. That's one issue. But then another issue is that there's just so many people in it and so many strong personalities and so much baggage on everything that it's kind of like trying to navigate in some giant city with no infrastructure or something like that. It's just at every turn you're just facing other people who are trying to go somewhere else. You don't have that. It felt more like this open wilderness. I mean, even like when I used to write video games or when we wrote our operating system for virtual reality and our applications, we did everything.

Right. We did everything. And we did it in a way that was completely unique to its time. We had a different architecture than anyone would use today in order to get the efficiency we needed. We had a high-level incremental compiler that's unlike anything else I'm aware of that was amazing. We'd actually see something that looked like code or graphical programming, but at the bottom it was instantly turning into opcodes that would never crash. And it was like this whole really interesting way of doing code that I've never seen anywhere else. And these days you can't do that. These days it's all about all the preexisting tools

and libraries that you learn. So you're kind of more in this downstream position. Yeah. So it's both better and worse in a way. Yeah. And it's sort of funny, like this has come up in a bunch of the conversations that I've had on this podcast about like one of the interesting things with contemporary computer science and software engineering is that the level of abstraction at which people are operating is so much higher now. Because like you aren't, you know, like writing the operating system kernel or the code generator for the compiler or whatnot. Like there's some people who do that, but, you know, you can do an awful lot by just sort of assembling like a bunch of the building blocks that are already there for you. And it's, you know, in a sense, almost miraculous. Like I even look over the course of my career, you know, just sort of thinking about, you know, the startup that I did in 2007. You know, I went from Google where, you know, we

had had to build all of this distributed computing infrastructure from scratch because it didn't exist. And like four years later, when I did this startup, like, there were all these open source things that were replicating some of those things. And so, like, I had a little bit of a start, but, like, I was still in 07. We were building our own data centers and, like, racking computers and whatnot. And if, like, you were doing that today,

you build it on a public cloud and, you know, not have to, you know, sort of, you know, spend a bunch of your startup's money, you know, not have to, you know, sort of, you know, spend a bunch of your startup's money, you know, sort of standing up a bunch of physical infrastructure. You know, so on the one hand, it's great. Like, you're just sort of putting this incredible amount of power into the hands of, like, individual creators. Like, fewer people can make bigger things inside of, you know, some sort of constraints. But, like, the downside is, like, you know, if you're not curious about what's on the other side of these abstraction boundaries,

like, you can limit yourself in, like, really interesting ways. And, like, that's sort of the thing. I mean, like, you weren't, when you were developing this stuff, you didn't feel constrained at all, right? You were just. Not at all. I remember feeling like even the built-in architecture in some chip in a CPU bothered me because it was enforcing structure on me. And, you know, like, I would hate that. I would resist it. I would say, these people at Intel, they're jerks.

Like, why would they make this? You know, well, it has to be something, you know, it has. Yeah, I think if you live inside an abstraction for long enough, it becomes like a religion for you. It becomes the framework within which you, your mind functions, right? There was a thing that's just been, there's a controversy in the last week in AI where somebody's saying that we must not understand. When I say somebody, there's a certain group of people who are arguing with another group of people. And the debate is, since there are mechanisms in our machine learning implementations like backpropagation that don't seem to have direct analogs in biological neurology, maybe we

don't understand biological neurology or something. And I'm like, what do you mean? This is just this code we made. What does it have to do with anything? It doesn't even work that well, if we're honest. Like, why is this even, how can this even be a controversy? But there's this thing where you start to believe in your abstractions so much that they loom larger than they should.

Yeah, and I, like, I could not more strongly agree with that point. And I think it's sort of especially interesting right now in all of these discussions about AI. You know, like these machines, these machine learning systems are far less mysterious than some people think they are. I mean, it's like all the way at the bottom, it's just a bunch of things doing linear algebra. And there's nothing wrong with that. It's nothing wrong with that. It's all great. But like it doesn't – and, you know, sort of arbitrary between, you know, biological intelligence and like a supervised machine learning algorithm.

I've always been bothered by not – I don't have any – I'm not opposed to the research program of the algorithms that are called AI. But I have a problem with the sort of concept and the culture of the term AI. And a little later, when I was a teenager, my main mentor early on was Marvin Minsky, who kind of invented a lot of the mythology of AI about this idea that we're building these little creatures on the computer. And I just find it, I'm actually thinking of writing a book that might be called something like AI is not a thing. Because what happens is we choose some bundle of algorithmic techniques and say, this is AI, but it's kind of arbitrary. Sometimes things go in, sometimes

they're taken out. And then we allow ourselves to believe in it like a monster or God or something. AI will take your job. AI will do this. AI has done that. Whereas, in fact, there's not any particular thing that's AI. It's just another example of people coming up with ways of using computation to do things. And this whole storytelling, I think, makes us a little nuts, you know. Yeah, you should definitely write that book. So, I've, like, I'm writing a book right now, and, like, I say something to that effect in one chapter of my book.

But, like, you are, like, the far more eloquent writer, and I think, like, an entire book on that topic would be amazing. Just totally amazing. Yeah, well, maybe it'll happen. Yeah, in your copious free time. Yeah, that's an issue. Yeah, it is an issue. So, like, after VPL, like, you went on to do a whole bunch of other things. And I don't want to like skim past

like anything super important, but like where I want to get to before we run out of time here is you have become like quite the, you know, sort of technical philosopher for modern digital life. You've written these amazing books. Like you are promoting a bunch of super interesting ideas.

Like you, like recently in some of the work that you're doing, doing at Microsoft, you've sort of coined the phrase data dignity, which is like a really interesting concept. So, like, I'd love for you to talk about, like, why it is that you took on this responsibility for, like, being the, you know, sort of the humane intellectual thinking about our digital world. I'm not sure if I coined data dignity, by the way. I think either Glenn Weil or maybe even Sachin Adela did. I'm not sure.

Digital dignity was a term I was going to be the title of Who Owns the Future, but the editor didn't like it, so it turned into Who Owns the Future. At any rate, so this is a whole long tale as well. In the 80s and 90s, there were a couple of really vociferous, intense movements within hacker culture, within technical culture, about what networking should be like whenever it really comes about. One of them was this idea that everything should be open and free. And that was started from a number of sources. One of them was a guy who was a friend of mine, Richard Stallman, back in Boston, and there were a few other source points for that as well. And then another was this kind of intense libertarian philosophy that government shouldn't be involved. We should leave everything to entrepreneurs. And in the late 80s and early 90s, I ended up spending time with

somebody named Al Gore, who's at that time a senator from Tennessee. He eventually became a vice president. And he was really interested in these things. And he came up with this idea of throwing some government money at people with nascent packet switch networks to bribe them to become interoperable. And that was the internet. So that was funded by the Gore bill. And so we used to debate, like, what this thing should be. And because of the extremely intense, those two dogmas, there was this feeling, well, it'll be minimalist. It won't have accounts, for instance.

It won't represent people. That'll be left to private industry. There won't be any persistent data on it. That'll be left to private industry. There won't be any persistent data on it. That'll be left to private industry. There won't be any transactions on it. That'll be left to private industry and on and on and on. There won't be any memory on it. There won't be any contextualization on it.

That'll be left to private industry. And I remember saying to him, you know, we're creating this gift of many hundreds of billions of dollars to persons unknown because there'll be natural network monopolies that form to fill these obviously needed functions. But whatever, that was, that was, there's just this feeling that that was the better way to do things. And since the experiment wasn't run the other way, we don't know. But then the other, everything should be free, I think set us down a terrible path because it feels all socialist at the first, at the, you know, it feels like this friendly socialist lefty thing. But since it's being mixed with this libertarian philosophy, you end up with only one possible business plan, which is advertising. So everything feels free, but actually the money is made by third parties who want to influence the users using user data.

And it ends up, it starts cute and ends up evolving into this sort of monstrous universal behavior modification scheme. Anyway, this is the stuff I talk about all the time where I think we've gone wrong and we've created a network that's more about deception than it is about reality. So what do you think we can do about that? Well, we're kind of in a pickle now, to use an expression from when I was a kid. It's a little, it's tricky. I mean, there are a lot of schools of thought about it. I think we can't combine socialism and libertarianism in the awkward way we did and expect to get

anything useful. And I think we should just choose one of them. And I personally think we're better off choosing markets. I'm worried about where we are right now. I just feel like our present internet is destroying societies and democracies and economies. You know, I think it's bringing down civilization. It's bad. We really screwed this thing up. So, you've been working on a bunch of concrete things to try to figure out, like, how to introduce these new incentive structures. Can you elaborate on that a little bit more? Yeah. Well, the problem is how to get from here to there. I kind of have in my head an image of what a society would be like with paid data. There's a few things to say about it. One is there are a lot of people out there who pretend to be making a living online but aren't because

they're fakers. It's all a big illusion. It's what we used to call a Horatio Alger illusion, where you create this illusion that there's this way of making a living when in fact there isn't. It's only for a token small number of people. However, there's another population of people out there who are making a living, but not within the rules dictated by a central hub, but as actors. Like for instance, there are tens of millions, maybe, well, we don't know the total number, but at least 50 million people in the world who are actually making a living delivering online video lessons and counseling and guidance. And, you know, this is anything from legal consulting to religious training to yoga teachers to musical instrument teachers. All those people have sort of cobbled together something that has to fight against the grain of everything because there's no—

There's no infrastructure to support them. There's no infrastructure to support them. There's no infrastructure. So each one of them has had to invent their own infrastructure by cobbling together little pieces from the different digital companies. And that population interests me. In a way, I see them as the future. Those are the people who don't have to worry

about their jobs being taken by robots unless, I mean, they could be. All we have to do is create some machine learning thing that steals all their data and makes a fake clarinet teacher without paying them for their data, just stealing their value. And that's what we've done in so many other areas. So the future I would see is to, first of all, try to support, to identify those groups and support them. And also identify those communities that are trying to create new structures to help people cooperate in decentralized ways.

And here, the blockchain community, not the Get Rich Quick blockchain, but the other blockchain, the blockchain of people who are interested in new ways of cooperation that can be mediated by networks, those people could be really important and helpful. I think we need to invent new structures. The reason that we treat data as being worthless, even though the companies that collect the data become the most valuable ones in the world, is that there's no collective bargaining for people

whose data is taken. So out in any other economic example, in order to have a productive economy, you have to have some – you have to invent some kind of structure so that people can cooperate and not have it not be this Hobbesian race to the bottom where each person is against each other person. And if you believe more in capital than labor, you call that a corporation or a legal partnership or something. So these people are incentivized to cooperate instead of try to kill each other. If you believe in labor over capital, you call it a union and you call it collective bargaining. or a legal partnership or something. So these people are incentivized to cooperate instead of try to kill each other. If you believe in labor over capital, you call it a union and you call it collective bargaining.

But on the internet, the difference is academic. And I was playing around with terms like UNORP and Corpian and they're terrible. So we just came up with a very, my research partner, Glenn Weil, and I came up with the term MID.

Actually, my wife came up with that. Mediator of Individual Data. So you'd have something that's a way for people to band into a group so as to not have the value of their data descend to zero through interpersonal competition, but instead have a degree of local cooperation. So we need to create those things. And MIDs can serve another function. Here, I'm talking fast because I know we need to create those things. And MIDS can serve another function here. I'm talking fast because I know we're almost out of time. But one of the things that's really terrible about what's happened in the world is we've been petitioning tech companies to become the arbiters of culture and politics.

But the thing is, do we really want tech companies to become the new de facto government? Is that what we want? I don't think so. So the mids could also become brands in themselves where people who bonded together to create a mid not only are collectively bargaining for the value of their data, but the mid itself has become a channel like, if you like, like a guild or a union or like a corporation or a brand that represents a certain thing. It might say, whatever data comes through here is scientifically rigorous and has been checked. Or whatever data comes through here is fashionista approved and is very beautiful. Or whatever data comes through here is guaranteed to be really amusing and suitable for your whole family or whatever. You know, like, what it creates is these in-between size structures that can take on this function of quality maintenance, you know, because you don't want a centralized source being the maintainer of quality.

That's a recipe for some kind of dysfunction, too much centralized power. So the mids both solve the economic problem and the quality problem, and we need to start creating them. So there are fledgling attempts to create them. Right now, they have no infrastructure tools to help them along. I'd like to change that. And that's just one little corner of the problem. I'm also just trying to – honestly, I'm just trying to get the tech companies to see the light. And here, you know, some of them are better than others. Honestly, I'm just trying to get the tech companies to see the light.

And here, you know, some of them are better than others. Yeah. So let's switch a little bit into, you know, like all of these other interests that you have. I think one of the fascinating things about you that folks underappreciate is that you are a composer and a musician, and you have one of the largest, maybe the largest collection of musical instruments in the world in your home. So your mother was a piano virtuoso, but how has this remained a thread in your life all these years? Yeah, after she died, I feel like music is my main connection to her, and I still play the piano,

but not so much straight classical playing anymore. I have my own style, and it's pretty unusual. I started just learning new instruments, and I have this voracious, perhaps not always healthy need to always be learning a new instrument. And so whether it's the largest instrument collection, I'm a little doubtful because there's some pretty big instrument museums. But in terms of playable collection, I'm pretty sure it is. And I don't know how many there are, but there are a lot of instruments. And I do run around, I can play them. And I have a...

And we're talking like hundreds, if not thousands. Certainly in the thousands, yeah. Yeah. Which is, you know, sort of a mind-boggling, interesting thought in and of itself that there are like a thousand, you know, thousands of distinct instruments that one could collect. Well, they're the best user interfaces that have ever been created. They're the ones that support peak human performance like no other invention ever. And they're profoundly beautiful.

And each one has a story. And each one is kind of a form of time travel because you learn to move and breathe like the people who originally played it, wherever it's from. So it's a kind of a cultural record that's unlike any other one. It's a haptic record,

if you like. Yeah. I mean, I've always been fascinated with piano. And I think the reason is it's always struck me that a piano is not too dissimilar from a computer. It's like this complicated machine that requires some non-trivial degree of mastery to get anything out of it. And, like, sometimes that struggle to achieve mastery is, like, this sort of long, isolating, you know, activity. You know, like, I've read biographies of famous pianists and, you know, like, some describe it as, like, you know, like I've read biographies of famous pianists and, you know, like some describe it as like, you know, you just sort of sit alone in this room and, you know, like struggle against this machine to like get it to bend to your will. And I'm like, holy, holy crap. Like that's sort of what, you know, you do sometimes as a programmer. Well, I've experienced feeling alone with a computer, but I've never experienced feeling alone with a piano.

Which is interesting. Even when you're practicing? Yeah, pianos are a bit mysterious because they're sort of the button box that transcends button box-ness. They have some kind of a life in them that they shouldn't have. I think that's one of the reasons that they're so provocative to computer scientists. Of course, the piano led to the computer pretty directly because around Mozart's time, somebody made a non-deterministic player piano, which is what inspired the Jacquard loom, which inspired the Babbitt generalized calculator, which inspired Turing, et cetera. So you can blame the piano for all this if you want.

That's awesome. So what has been your favorite performing experience over all of the years? Oh, performing? Yeah. I've had the good fortune to perform with a lot of interesting people. And I was living as a professional musician for a while in the 90s, but I've never been like a major one. I've never been a major star or anything. But as a sideman, I've had a special musician for a while in the 90s, but I've never been like a major one. I've never been a major star or anything. But as a sideman, I've had incredible performance experiences. I think my favorite one was when I toured with Yoko Ono and her plastic band. And Yoko and I would do these duets that were, a number of people said they were the strangest thing ever on stage. And I think we got there. I think we did it. John Perry Barlow called it the heavyweight championship of weirdness. And I think we got, yeah, that was good.

And I played with Arnett Coleman, this wonderful father of free jazz. And I've done a lot with Philip Glass, including just recently. In theory, we're doing a new record together. We did one in the 90s. I just did a show with Philip where I brought the pedal steel guitar into the minimalist music aesthetic for the first time, and it worked great. It was so fun.

And I just did a thing with T-Bone Burnett a while ago for Sara Bareilles' record that was really, really cool. And I've done, I've played with all kinds of people. I've done, I've played with George Clinton

and P-Funk. I've done more than you'd imagine, but always kind of in the background as a performer. I've done some solo stuff, too, but as a performer, you can only do so much. I've had this career in computer science and another one as a writer, and at a certain point, you can't really do everything always forever and have a family. So, unfortunately, I haven't been doing as much lately, but still a little bit now and then. Yeah. So, like, even though, like, sort of writer, philosopher, computer scientist, composer, performer, like, you still have other interests, which, you know, I guess your mother really did. She didn't remember. Well, she certainly convinced you that, like, you had to, you know, sort of exhibit excellence in a bunch of different domains.

But, like, maybe one of the most interesting things that you and I have chatted about is the Neural Information Processing Symposium is, like, the big deep learning conference for the field. And the best paper at this year's symposium was this thing called Neural Ordinary Differential Equations. Right. You know, the short idea is that neural networks are usually these sort of layers of connected artificial neurons. And, you know, you're trying to, you know, sort of figure out, like, the activation weights, you know, between the connections of all of these layers. And, like, which nonlinear functions you're using to, you know, sort of accumulate the weights into a network node and, you know, like whether or not you should use things like dropout to like impose some sort of memory loss as you're doing training and blah, blah, blah, right? Like there's back propagation and all this other stuff. And so like this paper is sort of interesting in that it sort of models a bunch of the interior state of these deep neural networks as a system of ordinary differential equations.

And it was sort of a sensational result because, you know, like it has some like really big performance implications for training deep neural networks and like the amount of computations required to train them is one of the sort of big things on people's minds. Sure. Yeah, so, like, that's great. But, like, when you saw it, the connection you made was to quantum field theory. Yeah, so here, I don't—it might be a bit premature to speak about this, but one of my—one of the dimensions in my life is I have a lot of physicist friends. I've done sort of weird projects with theoretical physicists from time to time.

And this all started because my first serious girlfriend was the daughter of the head of the physics department at Caltech. So when I was a teenager, I was sort of hanging out there, and I got an informal chance to learn directly from Feynman and Gilman and all these amazing people. Yeah, which is insane. Yeah. Ever since then, I've had this kind of – so one of my best friends is this guy named Lee Smolin who co-founded the loop quantum gravity approach to trying to find a unified theory, which is probably – since string theory kind of burned itself out, it might be one of the most prominent ones now as an alternative. Anyway, so there's this thing in physics where we've always had pretty – we want really simple equations to describe, let's say, fields. And then we want them to have emergent behavior that's complicated enough to be reality. But the thing is, we've kind of burned people. It's kind of like in a house where you put all your mess in one room to pretend the house is clean.

So, like, in string theory, they tried to simplify some stuff, and they ended up with this insane room full of mess. Really messy room of, like, you know, an unbounded number of possible, like this was a really messy room, the sort of range of possible string theories. And so when you try to, it's kind of like playing whack-a-mole, when you try to, you often end up creating this big mess in another corner when you're cleaning up one part of it. Yeah, I think it's fantastic how broadly your mind wanders. And with that, I think we should wrap up. Thank you so much, Jaren, for being on the podcast.

Delighted to be here. Thank you for having me. Awesome. Well, thanks for joining us for Behind the Tech. You just heard Microsoft CTO Kevin Scott speaking with Jaron Lanier. So what really struck me about that conversation was how far ahead some of the visions for virtual reality were, you know, decades ago and how similar those visions are to what we're actually seeing in the market now with both VR and with augmented reality. Yeah, it's this really fascinating thing with true visionaries like Jaron. He saw this thing like way, way, way, way before anyone else did. And it, but just the, his consistency and tenacity over time to

sort of stick with the vision. It's not like he's wavered, like he's been doing this for almost four decades now. And like, he's had this vision and he's kept pushing, you know, episodically for like this very, very long period of time. And like I find that almost as amazing as the vision itself, just the willingness to believe in something for that long and to just push against it as hard as you can. Yeah, it's so interesting to me. You know, when he was talking about how he's getting pitches for certain uses of VR and he's like, oh, yeah, yeah, I had a paper, you know, kind of predicting that 30 years ago. And he was right. And as you said, he continued to push and be committed to that, which is just kind of incredible.

Yeah. You know, and sometimes the frustrating thing with technology is timing matters way more than you would like to think. Like, unfortunately, like, vision and persistence aren't enough. Like, sometimes, like, the technology that you need and, like, the set of conditions in the ecosystem you need to exist in order for something to become broadly adopted by a whole bunch of people is just, just isn't there. And it's really interesting.

Like, it's sort of, you can almost see it right now that, like, you know, mixed reality, augmented reality, you know, like the whole grand, you know, the whole grand virtual reality vision, like, might actually be within reach now. But it's, like, taking all of that time. Yeah, no, it's so true. I mean, as you said, you know, Microsoft just showed off the new version of the HoloLens, and it feels so much closer,

and yet it's just still somewhat, you know, you can kind of see what Jaron's vision has been all this time, and we seem to just be within grasp. It's really exciting. The other thing that's sort of fascinating about his vision for VR is, on the one hand, I think it is like a very deeply technical thing. But I think, you know, as you heard in the conversation, the thing perhaps even more than the technology that motivated it is like this very, you know, humane desire that he had, like to connect with other people. Yes. And like that's something that, you know, you don't always get from folks who are trying to do something with like deeply, deeply, deeply technical technology.

No, you're exactly right. It kind of reminds me a little bit of, you know, Tim Berners-Lee and the World Wide Web, which was a similar thing. And that's, you know, celebrating an anniversary right now, too. And it's like, you're right, a lot of times it's rare to see these intersections between these highly technical things and these also highly social and personal and deeply connective. But sometimes, like, those are the things that, like, have the biggest impact on the world, is, like, you've got this desire to, you know, sort of facilitate more of our own humanity to, like, empower and ennoble, like, individuals and groups. And, like, those technologies can be

really, really profoundly transformative. I mean, I would actually argue, I think, that what you just described is kind of the basis for the most transformative technologies, whether we're talking about radio or television or transistors or anything else. Or the PC. PC, absolutely, is finding a way to facilitate humanity. Yeah, for sure. All right, so I think we're out of time for this episode, but we're going to meet another

icon on our next show. That's right. I'll sit down with Reid Hoffman, investor, author, and entrepreneur, someone I consider a true friend. Be sure to join us next time on Behind the Tech, and please help spread the word. Tell your friends, your colleagues, and all of the geeks and non-geeks you know. See you next time.