Microsoft Research Podcast - 046 - Designing the Future With the Help of the Past with Bill Buxton

Episode Date: October 17, 2018

  The ancient Chinese philosopher Confucius famously exhorted his pupils to study the past if they would divine the future. In 2018, we get the same advice from a decidedly more modern, but equally philosophical Bill Buxton, Principal Researcher in the HCI group at Microsoft Research. In addition to his pioneering work in computer science and design, Bill Buxton has spent the past several decades amassing a collection of more than a thousand artifacts that chronicle the history of human computer interaction for the very purpose of informing the future of human computer interaction. Today, in a wide-ranging interview, Bill Buxton explains why Marcel Proust and TS Eliot can be instructive for computer scientists, why the long nose of innovation is essential to success in technology design, why problem-setting is more important than problem-solving, and why we must remember, as we design our technologies, that every technological decision we make is an ethical decision as well.

If you're going to come and make an argument that something's going to have huge impact in the next five years, if you haven't got 15 years of history of that idea and can trace its evolution and history and so on, then you're probably wrong or you haven't done your homework or you might get your head cut off when you come to this presentation unprepared. Even if you're right and you don't have that 15 years, then that's gambling. That's not investment. That's not research. You're just lucky. Design is a repeatable profession. You're listening to the Microsoft Research Podcast, a show that brings you closer to

the cutting edge of technology research and the scientists behind it. I'm your host, Gretchen Huizenga. The ancient Chinese philosopher Confucius famously exhorted his pupils to study the past if they would divine the future. In 2018, we get the same advice from a decidedly more modern, but equally philosophical Bill Buxton, principal researcher in the HCI group at Microsoft Research. In addition to his pioneering work in computer science and design, Bill Buxton has spent the past several decades amassing a collection of more than a thousand artifacts that chronicle the history of human-computer interaction for the very purpose of informing the future of human-computer interaction. Today, in a wide-ranging interview, Bill Buxton explains why Marcel Proust and T.S. Eliot

can be instructive for computer scientists, why the long nose of innovation is essential to success in technology design, why problem-setting is more important than problem-solving, and why we must remember, as we design our technologies, that every technological decision we make is an ethical decision as well. That and much more on this episode of the Microsoft Research Podcast. Bill Buxton, welcome to the podcast.

Glad to be here. So I like to start by asking my guests what gets you up in the morning, but you've already answered that in print. And I quote, what gets me up in the morning is to realize what I dream about. So now you have to tell us what you dream about. It depends which morning it is, I think. I think there's an embarrassment of riches of things to want to do, and I think that that's one of the best things, because you're never at a loss to be motivated. But then the other problem is you have to make choices as to which one you pursue.

You can do anything and everything in your life, you just can't do them all at once. You always want to be falling in love with something that's just captured your imagination, but in so doing, you have to retire a previous passion, or at least move it to the background, because you can't go full throttle into more than one or two things. One description of what I do for a living is experience design. And I'm prone to say Jimi Hendrix had the greatest wisdom of this, and that's the most profound question, are you experienced? And if you don't have a breadth as well as depth

of experience to draw on, how can you be good at experience design? Because it's building up this repertoire and curating this repertoire of experiences in your life across the board that is the treasure trove that you can mine in whatever you're trying to do. Your bio says you're a relentless advocate for innovation, design, and the appropriate consideration of human values, capacity, and culture in the conception, implementation, and use of new products and technologies, which is a mouthful. But let's unpack that a little bit. I'm really intrigued by your statement of the appropriate consideration. Tell us what you mean by that in the context of designing new technologies and products. Well, one of my heroes is a historian of technology named Melvin Kranzberg, and he has some laws. But his first law is technology is not good, it's not bad, but nor is it neutral.

It will be some combination of the two. As soon as you say words like good and bad, that implies you have a moral compass. And the real question is that when you're making technological decisions and launching technologies into society, you are in fact making an ethical choice, whether you know it or not. And so maybe you'll do a better job of it and wait more heavily on the positive if you actually know what that moral compass is and that you are in fact making an ethical decision. I'm not trying to put too heavy a weight on this and that you're playing God, but you are in fact having an impact, but you're also human. So how can you just do the best? You will get some stuff wrong. So take the responsibility to clean up the mess without

throwing the baby out with the bathwater. And so it basically says that appropriateness is appropriate to the moral order of place or where it's going to be placed. That's the closest way I can put it. Let's talk about your job description at Microsoft Research. When you started at MSR, Rick Rash had hired you to, as you say, make design a core part of the Microsoft culture. So how did you go about doing what he said? That's about the vaguest job description I can think of, and yet it's perfect. Well, actually, what he really said was,

make a difference. And if you're not the best person to figure out how you should do that, you probably shouldn't have the job. Then my response was, okay, I'm going to try and help contribute to bringing a greater awareness of design to the company. And that meant actually not trying to design products, but trying to design the culture and change the attitudes and not elevate design to the point where everything's design led, but where it's an equal partner at the table. In the early days, when I would speak to different teams in the company in large or small groups, it would be kind of like, don't expect this to come from above or

from management or anything like that, because we are our own culture, we make it, and it's every individual. And if you can actually start to just feel empowered to, within your own, even if it's one other person, you can start to make adjustments along the way you want that can go viral because if we're shifting in a good direction, it will be noticed. And then people will say, well, what's the secret sauce that you're using? And nobody can own this. It can't be about any individual. It's got to be about empowering individuals to form groups and clusters that, because that's what culture is. It's a mutually agreed upon set of values. So today, like you, I'm going to use some literary quotes

to prompt our discussion on technology research. So let's start with Marcel Proust. He once said, the real voyage of discovery is not in seeking new landscapes, but in having new eyes. One of the major themes in research is looking for the next big discovery, right? How does having new eyes or different optics, as you've said it, inform the quest for innovation, or how should it inform the quest for innovation? So the net result is that, in some sense, I would describe my job description as being an optician and to find the right lenses. I'll give you an example. As you say, the industry

is heavily driven by people trying to find the next big thing, whether it's a new gadget or new application, killer app or a new service. And if you're just graduating from university or design school or whatever, that of course you want to become a millionaire by the time you're 24 or you're a failure. And so there's all these pressures. And so my automatic reaction, I just wrote a two-pager that said, the next big thing isn't a thing. And I said, it's actually a change in relationship amongst the things that are already there

and the things that are going to emerge. And when I say relationship amongst those things, it's about the social relationships, things like kinship, introduction, negotiation, approach, departure, all of these things, the moral order. These are all terms that we know about the society of people, but we aren't used to speaking about in terms of the society of technology.

What could you do that would have more impact than if things just worked, if things just worked together seamlessly? And if in working together, every new thing I added came at a great value in and of itself, but it also added value to everything else I already had and they to it. And furthermore, every new thing I added reduced the complexity, not only of that new thing, but reduced the complexity of everything else in the ecosystem and they it. We realized that hardly anything works well together, much less seamlessly. And what we've forgotten when we come back to the human side is that the better we get at making really desirable, beautiful, affordable, useful, valuable devices, the worse we're making things.

Right? The cumulative complexity of a bunch of desirable, simple, affordable, valuable things is way above the human's capacity to deal with. And that's why you must reduce complexity with everything you add. And that takes a very different approach because it forces you into thinking about an ecosystem. Albert Schum, who's a part of the Canadian mafia trying to change design at Microsoft here,

if he's a good friend and a fellow cyclist. And he has a nice way of saying it, that in the industry we spent a whole bunch of time learning how to design houses. The real challenge is building the communities and the city planning and the urban planning and the flow of things. And I think even the changes we've been making over the last year or two have been significant steps on this path. But the challenge in innovation is how do you go beyond that and say what are the right metrics for our aspirations and where we can be and how soon we should get there?

Because only when you find that can you set appropriate goals that most meet your objectives. You have become the collector and curator of more than a thousand computer hardware artifacts that chronicle the history of various aspects of human-computer interaction. So tell us about your collection or collections. How did you get started doing this? What kinds of things have you collected and how hard was it for you to get your hands on some of these things? I do for a living. I'm always looking for reference material, always scanning, collecting things around, surrounding yourself with them for inspiration, for ideas, and to trigger thoughts. And having them sitting there around you, and all of a sudden some new relationship pops out. When I'm at a loss for a solution to a problem, I go and surround myself with these objects. But over about 40, 45 years, I've never thrown any of them out. I've kept them all.

And so when anything came out, whether it was a brochure or an article in a magazine or something like that, I kept it and documented it for future reference, for teaching, for teaching myself. And to go back to say, hey, I think I've seen this before. And you could think of them all as prototypes and really expensive to make prototypes, which I could get for practically nothing sometimes like on eBay, where it's like a really expensive education or somebody else paid the tuition and they're sitting there. If you want to get the benefit of that education, you can. And therefore, when I do start to make something or when anybody in the company does, they can start at a much higher level because they've got these reference objects. Interesting. And so the base point of departure for any problem I'm looking at is somebody has already solved this problem and there's something out there that's already

done this. So I'm going prospecting before I go building. Tell us about the collection. What's in it? Well, the collection's sort of a cross-section of all of the input devices through which people have interacted with digital technology pretty much from the beginning. And so that would include mice and joysticks and trackballs and trackpads. It is PDAs. It's game controllers. It's foot pedals, head displays. It's smartwatches going back to 1978. It's the world's first smartphone. It's the history of portable music players. It's the history of AR and VR technologies going back to a reproduction I made of the very first stereo viewer from 1838. And it's also examples to use to serve as the basis for storytelling that illustrate some of the things that are really important about design. I don't think many people in VR know that it is due to virtual reality in an early form that led

to Yellowstone being made the first national park in the world, not just the United States. Or that the very first stereoscope from 1838 was already looking into a virtual space because photography wasn't invented until the following year. There were no photographs to make stereo images from. They had to be hand drawn. And so when you looked into Wheatstone's original reflective stereograph, you're looking into hand-drawn lines into a world that never existed. Wow. I think those things are really interesting because you start to see patterns if you go through it. But from those patterns, you say, okay, they probably haven't stopped. And so you can extrapolate.

So it's really hard to extrapolate from a point. If I have a line, it's much easier. And so I have this game I'll do with adults as well as children. I'll draw all these different lines and say, continue these lines. And then I'll put a point. And they have no idea what to do with the point, but all those other things, they can continue because they can see the

pattern as things were going. And it doesn't mean that the extrapolation is correct, but it gives you your initial bearing for your exploration. And usually because there's other things involved, there's probably a couple of lines that come and you'll start, maybe you'll see there's intersections from the extrapolations. And you have these ways to visualize. And this gives you a different way to think, accompanied by concrete examples that you can experience to get to the essence of the finest granularity. So you refer to something you call the long nose of innovation. I think researchers are familiar with the phrase, the long tail, but the long nose is an interesting one. And it's in context of new technologies and how long it

takes them to catch on. And you also had said at some point that our slow rate of progress is maybe not necessarily due to a lack of technology, but lack of imagination. How and why do we lack imagination and what can we do? What can researchers do about that? The long nose basically comes back and sort of saying, if we look historically at the evolution of technologies,

it takes at least 20 years from the first clear articulation of something to the point that it's mature, where let's measure maturity as it's a billion dollar industry. If you're going to come and make an argument that something's going to have huge impact in the next five years,

if you haven't got 15 years of history of that idea and can trace its evolution and history and so on, then you're probably wrong, or you haven't done your homework, or you might get your head cut off when you come to this presentation unprepared. Even if you're right and you don't have that 15 years, then that's gambling. That's not investment. That's not research. You're just lucky. Design is a repeatable profession. It's not, I get lucky once in a while. And so if you want to study design and innovation, is study the repeat offenders, the ones that can do it over and over. You don't have to wait for the muse to come and drive you. And that's what you learn.

And you can only do that if you have process. And the long nose is a key part of the process. Now, for those who doubt, the mouse, which everybody who saw one in 1968 knew is the right thing, but it wasn't until Windows 95 before everybody had a mouse on the desk. Now, why did it take so long? I first used a mouse in 1971.

Now, the thing is, you need a perfect wave of things. You had to perfect Windows icons. You had to train the developers how to write this type of graphic user interface. That was a whole new thing from DOS or Unix. And you needed the processors. You need graphics processors. You needed the displays to switch to bitmap displays rather than calligraphic displays, which dominated back in the time,

basically glorified oscilloscopes. Every technology goes the same route. And so Long Nose is basically this reminder of how long it takes. So it also says the following things, and reforces what I was saying about the combinations about innovation being aggregation of existing ideas, that everybody thinks that things are moving really quickly. And that is not true. We mistake a whole bunch of things moving really slowly with things moving quickly. It's the difference between amperage and voltage. Any single technology is evolving, statistically speaking, really slowly. But when you have a number of different things moving slowly,

slightly different paces, but simultaneously and at different stages on the nose, if you start to realize that's what's going on in the overall technological ecosystem, you can see those patterns and then project forward because you can extrapolate from history, and say, here's where you hit the inflection point, and that's when things are going to happen. Everything has a perfect storm, and there's methods by using this technique to actually predict when that perfect storm's going to happen. I'll give you a really quick example. I spent my

early career, after I switched being a musician musician to building digital music synthesizers for live performance. So I saw the evolution of how digital audio emerged. I went to Silicon Graphics and became chief scientist there doing animation systems. But the only act of genius I had, because I wasn't a computer graphics, I was literate, but I wasn't a specialist in computer graphics. But I knew that computer graphics was going to follow exactly the same pattern as computer music, but it was multiple orders of magnitude more complex.

So it was just shifted further along the timeline. And so all the planning over the eight and a half years I was there, we kept hitting that right. And the reason we could know exactly what to do and when was because I just was repeating what I'd already done in music. And so all I needed to do was to see that relationship. And I think overall, that type of pattern happens throughout, but you have to know those other areas where you go prospecting.

So the long nose, the notion of history, collecting, sampling, and not just going immediately to building. We spend far too much and go far too quickly into problem solving and don't spend enough time problem setting. And that's the ultimate skill. Can you define problem setting a little more clearly? Problem setting is basically, it's not enough to get the design right, you've got to design the right thing. And so if you just leap in and start design right. You've got to design the right thing. And so if you just leap in and start building something where you've got a solution, you have no idea if that's the best option.

There might have been a better way. And you didn't take time because you're already behind schedule. But here's the crazy thing. At the beginning of the product cycle, you have a small team just getting going. Your burn rate in terms of what's costing you per week in terms of the project and that is very, very low. So what you then should be doing is thoroughly exploring a range of different alternatives. Problem setting part of that process is this notion of you cannot give me one idea. You have to learn how to work quickly and give me multiples. That's a technique for this whole issue. How do you deal with the problem setting? And by exploring the space first, oh, that's the real problem. Put it this way.

You have a bunch of people that talk about user-centered design and they'll say, you know, go talk to your users and they'll tell you what to do. Okay. Would you go to a doctor where you walked in and the doctor said, okay, what's wrong with you? What operations do you need? And what drugs should I give you? And what dose? right? And that's how some people naively interpret user-centered design, is listen to users. And no, I'm going to ask you all kinds of questions, but I'm going to take all of those as part of the information that helps you make a diagnosis. And so where do we collect the symptoms to find out where the real problems are? You're telling me this,

I understand the situation. Now, I have to know enough about your industry to ask pertinent questions. And for me, that's what the problem setting is. The designer, your main equipment is to have that meta-knowledge. And that's where the diverse interests come in. So how do you get that knowledge? But if you don't even know that's the kind of knowledge you need to get, you're not even going to go looking for it. So you look at the product development cycles and even in research, what you're talking about is something that people would have to say, okay, we need to rethink how we work and what we make time for. So I'd throw the argument the other way. You can't afford not to do it.

So your cost per month on a project, if you put an extra month up front, it costs you almost nothing. And if it comes up with a much better solution that's a fraction of the price and can get it done more quickly and have a much better margin, first of all, you've made up for the lost time by having spent that up front. But let's pretend that it still takes the same amount of time. We never have time to do problem setting and so on sufficiently. We're getting better at it.

But we seem to be able to have time to be three months late where we are fully loaded with the highest burn rate possible, right? Yeah. I mean, if you're going to take an extra month, do you want to take it where it costs you the most or do you want to do it up front and you get a better product? The other part is it's not all in one. You don't make all your decisions up front and then go build.

The decisions that you make the earliest are the ones that are hardest to change later. So that's your basic architecture. In the software industry, we don't have architects. What we call an architect in architectural terms is actually a structural engineer, and we have no architect that has the design perspective at the very beginning.

But also, there's this notion that once you've got a representation, like a rendering of what the screens are or some of these other things, that that's the end of the design. There's only two places where there's room for creativity in design. So the first place for creativity is the heuristics process

whereby you enumerate the repertoire of things from which you're going to choose. And then the second is the heuristic you use to eliminate all but one. And it's that inhale, exhale. You start with nothing, you end with one. But you have to go with that whole thing. And you would love afterwards, you're gonna say,

I could have gotten it right from the beginning. And you could, but you never would have. And that's the biggest mistake. The fastest way to a mediocre product is to make a planet stick to it. Let's talk about AI for a minute. Because tech companies are putting a premium on AI talent. Is it important now? Apparently, people are using the terms gold rush, talent war, feeding frenzy.

And you've suggested that there's a risk that anyone who's not doing AI might be marginalized. So I have to preface by saying, I think what we can do today in AI is absolutely unbelievable. It's beyond my wildest expectations of what we'd be able to do at this point. It's unbelievably valuable, but it's nevertheless essential, but not sufficient. And as I said, you need a perfect storm of a whole bunch of things to get a sustainable system and ecosystem in place. And my fear is that if you focus too much on the AI component, that you distort the other requisite skill sets and disciplines that are needed to ensure that AI is successful. Every discipline represented in our company is essential to our success, but not sufficient.

And the trick is to find the balance. And one of the important elements here is to make a distinction between literacy and expertise. It is essential that everybody in the company has a level of literacy about AI. But it's equally important to have literacy about every one of those disciplines. And that means that AI should be working as hard to between literacy and expertise, that developers and designers are so focused on AI that they feel that if they're not somebody who's only got 20% of the AI competence, but they've got way better integration of the AI into their larger ecosystem. Because like any other technology, it's not good, it's not bad, but nor is it neutral. And there will be a positive and a negative consequence of that technological change. Another premise in AI research has its underpinnings in what we've referred to as the DIKW pyramid, where you start with data, which supposedly gets refined to information and then to knowledge and culminates in wisdom, which is the ability to make good decisions based on the data you have. And this, of course, has literary roots in T.S. Eliot's

The Rock. Where's the life we've lost in living? Where's the wisdom we've lost in knowledge? Where's the knowledge we've lost in information? Talk about this in the context of this idea that if we have enough data with machine learning, computational power and sophisticated algorithms will end up with wisdom. Well, first of all, Elliot left off two levels there. So where's the wisdom we've lost in knowledge? The knowledge we've lost in information, the information we've lost in data, and the data we've lost in noise. You have to remember noise cancellation. And people talked about a data

revolution. So I'm sorry. no, it's a data explosion. And in information technologies, no, it's not. It's only information if it can serve as the basis for informed decision-making. I think it is very, very healthy to have that hierarchy. I think it's extremely valuable to be able to fit things into moving up that food chain. But I think that the role that intelligence plays there and where intelligence lies is a sticky thing. And we have to base our expectations of the technology and therefore have our engineering guided by a sense of what's possible at any point in time along that path. Now, I know that we're talking in AI

about sensing an ecosystem environment and all this sort of stuff. Well, we have to be realistic about how much of that we can sense at what point in time and then understand what elements are being neglected and are not simply feasible at this point

to deal with, and therefore our notion of intelligence is limited. And how do we, at any point in time, make sure we're backfilling those gaps until it can be proven that we've got those other parts reliably taken care of? And again, by looking at the disciplines, doing the analysis, we can look at the timeline and take appropriate action for each thing to make sure that we've got the bases covered with the appropriate technologies for that moment in history and not make colossal mistakes and confuse the target with where we are right now. It comes right back to what I said earlier. It's not just being able to get the vision. It's how do I get there from here? What would you say to the people that are moving into this arena right now? What should they be thinking? What could their next steps be? In a way, my advice is less concrete in terms of learn this, learn that, in terms of some skill.

We've said already that the problems we face today require depth. You have to be really good at what you're doing if you want to really have influence. And for me, the only way you can get really, really, really good at something is if you're just so fashionably in love with it that it's not work. Now, people say, okay, you've got to find your passion. Well, the problem is, how do you do that? Get into the traffic. Because if it's not hitting you wherever you are, then move. But the other part is, by trusting my nose to stuff that caught my fancy

and chasing those things that made no sense, but in retrospect were the perfect career moves. Like why would anybody go to university and do computer music when nobody even knew what a computer was and spend four years doing that? But it was the most brilliant career decision that I never made. If it wasn't a career decision, I want to be a musician. But I would say, always be bad at something you love. And it doesn't matter if things make sense. That's the other part that's really critical. I purposely rejected any career path for which there was a brochure

in the guidance counselor's office in high school because it's already full. There's going to be already too many people doing that. And it's not that I'm not competitive. It's just that my main competitive advantage is I'm not trying to compete in the same race. And if you've got these interests and you become uniquely qualified, you can have the satisfaction you're the best in the world at what you do. You're just the only one that makes you also the worst. That keeps hubris from taking over.

But have the faith that at some point in your life, all that work will be recognized and somebody will need it. There's somebody in the world who needs it. And the question is, now I find it. For me, it took me until I was 40. But the time leading up to that was so full of rich experience that it never occurred to me that I wasn't making any money. I was the richest person in the world because I was doing what I love doing. Bill Buxton, thank you for joining us today. Thank you for having me.

To learn more about Bill Buxton and the latest innovations in human-computer interaction, visit microsoft.com slash research.