The Vergecast - How Google is trying to make ambient computing work

Episode Date: May 11, 2022

Google has big visions for the future of computing. It’s working on building what it calls an ambient computer: a virtual helper that can accomplish anything, anywhere, any way you want. But that vision won’t come easy. Ahead of Google’s I/O developer conference, The Verge sat down with some of Google’s most important executives to talk about the company’s vision, its new hardware and software, and how the company is changing to build the future it imagines. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Support for the show comes from Retool. Too many companies run critical operations on duct taped spreadsheets, Slack workflows, and whatever else they could cobble together. Not because they want to, but because building internal tools means weeks of waiting on someone else's backlog. That's where Retool comes in. Build custom internal tools just by describing what you need. Prompts something like,

Build Me a Revenue Dashboard on our Salesforce data. And Retool actually builds it on your company's data, in your cloud with enterprise security built in. Go to retool.com slash Verchcast. We all need to retool how we build software. What's up, y'all. I'm Skyler Diggins, seven-time WMBA All-Star, Olympic gold medalist, and mom. And I'm Cassidy Hubbard, host and reporter for nearly 20 years,

covering the biggest names and stories in sports and mom. And this is Am Mom, a community for athletes, game changers, and moms of all kinds. dropping May 14th. Tap in with us. Welcome to the Vergecast, the flagship podcast of tablet apps that are really just blown up smartphone apps. I'm David Pierce. So it's Google I.O. this week, which is Google's developer conference and also the company's biggest launch event of the year.

There's a ton of product news coming out of I.O. this week. And we're going to get into a lot more of that from the software to the hardware to the services to all the weird stuff Google is working on. Friday show. But today I want to look at the bigger picture. Google likes to call I.O. its State of the Union, which is actually a pretty good way of thinking about it. Google is this huge, sprawling, decentralized company where teams build lots of things. Often it seems like nobody seems to know what anybody else is working on.

And then once a year, the company tries to pull everything together and figure out what's Google, the whole thing, the entire company, all these hundreds of thousands of people actually up to. I spent a good chunk of the last week talking to Google executives about that exact question. And it's funny to do that, honestly, because you start to hear the same things over and over and over again. For starters, everybody really likes to talk about Google's mission, which is to organize the world's information and make it accessible and useful. I know this by heart now, because Google people won't stop talking about it. You know, our mission in Android is to bring computing to everyone in the world. I give Larry and Sergey a lot of credit for coming up with this

mission all those years ago. We've extended it recently to basically being about being helpful for users. But every year at Google I.O., there's the story Google is trying to tell about itself, and then there's the story it's really telling. And this year, the story it's trying to tell is about how powerful its computing tech is. Google talks a lot about the tensor chip in the pixel phones and how it lets Google do all kinds of powerful AI stuff. It also talks about large language models, which are AI systems that can develop a radically better understanding of and images and even videos, and how all of that helps Google organize information. Computing and AI over and over.

Computing an AI, computing an AI. And that's all true. But underneath all that, as I talked to folks at Google, I heard a different story. What's going on with Google right now is that Google has become this huge, sprawling, decentralized company, where teams build lots of things and nobody seems to know what anybody's working on. And as a result, Google feels kind of incoherent. For so long, you could think of Google as a search company. Google as a search company and mostly be right.

Now it's, I don't even know how to describe it. It does all the things. And honestly, even in search, it's confusing. There's Google Search, but there's also Google Lens, which is kind of like Google Search, but in a different way. And there's Google Assistant, which you can use for search, but is really a whole different thing. Google is still more or less on the organize the world's information and make it accessible

and useful grind, but as the company has gotten bigger, its pieces have stopped sort of making sense together. It makes phones and computers that can barely talk to each other. I mean, dear God, have you ever tried to just send a file from your Android phone to your Chromebook? The whole of Google sometimes feels like less than the sum of its parts. That is what Google is working on now. It's trying to take the many parts of Google, the countless features and products and initiatives and experiments and prototypes, and make them make sense together. I think Liz Reid, a VP of Engineering at Google who works on search, actually explain this better than anyone else I talk to.

A lot of these newer use cases, there's a style or a taste component. There's a way to process information. Some people really find it easy to process video and other people will find it distracting. On the other hand, some people's literacy is not as good. And so a long webpage that you have to read through not only takes time, but they're going to get lost. And a video that's spoken in their language is really intuitive. I think we need to assume that there is no best answer for many of these things. I mean, if there's a height of the Eiffel Tower, okay, there's an answer.

to that. But if it's like, I'm finding a wedding dress, there is no right answer. People have very different styles. They have different price capabilities as well. So how do we really let you find the dress you want versus your next door neighbor find? Multi-search, which Liz and her team have been working on, is one of the things Google is talking a lot about at I.O. And it's a pretty good microcosm of what it looks like for Google to start to put the pieces together. The idea behind multi-search is that you can't always express your entire question in one way, right? Google likes to give the example of a dress you like. You see the dress, you want that dress, but you want it in green. So far, there is literally no way for you to ask Google to find that for you. But with multi-search,

you can take a picture of the dress and then type in green, and Google understands and processes the search that way. One way Google is making all this stuff work is by making it conversational. I heard this also over and over from Google execs over the last week. They're absolutely convinced that rather than you just barking an order at your phone or speaker, that you should should have something that more closely approximates an actual back-and-forth conversation. And what that means is that in the process, Google can ask you questions. It can refine your search results. It can figure things out better than just doing it one exchange at a time. I am skeptical of this. First of all, have you tried having a conversation with a voice assistant?

It's either infuriating or it breaks, or actually, no, those are the only two options. But Google thinks it can work and feel natural and not be the worst. And the reason it does, is because of those large language models I was talking about. Rather than me try to explain, I'm going to let Zubin Gavramani, a distinguished researcher at Google, do it. This is kind of a long one, but it's worth it, I promise. So about 10 or 15 years ago, the field of language technologies really was revolutionized by deep learning. And of course, at Google, we've been using deep learning very extensively from at least 10 years ago in language and other domains.

And then in the last five years, so back in 2017, we actually develop a form of language model called the Transformer. And the Transformer is in some ways kind of like the grandmother of a lot of the modern language model. So the T in BERT, in T5, in OpenAIs, GPT3, in Meta's Opt, it all stands for Transformer. So this has been an incredible breakthrough that has enabled a lot of these other interesting language models to flourish over the last five years. Okay. let's skip ahead a little. Here's a little bit on how transformers and new language models have changed the way translation works, which is both a big deal and a pretty good explanation of what

it means for the whole system. You know, the traditional way you think about doing translation is you need a line text, like text in French and English, let's say, to be able to translate from French to English and vice versa, right? That's the way translation traditionally was thought to require. But actually, it turns out with some of these large models, you can just learn to translate in some of these languages using only monolingual text. So the model is exposed to many different languages, but it doesn't need that aligned translation. It just needs some monolingual text

without any translation pairs of sentences. And that's a kind of breakthrough that, again, is surprising to me as a researcher who's been in the field for 30 years. Of course, when you dig into it, it makes sense because there are patterns in the data that help it extract the common representation. between different languages that are needed to do that translation.

But it's still a breakthrough that really helps us, for example, deploy, translate across many more languages that are useful to lots of other people. Basically, Google is turning practically everything, from text to audio to video to images into language, and then using these language models to help understand it all. So you can change the way Google understands an image by adding a bit of text, or ask it a paragraph-long question and have it understand you. Here's Nino Taska, who's a director of product management on Google's speech team, who had a pretty good idea about where this all goes.

You can imagine like a 45-minute YouTube video of do-it-yourself steps one through nine. And a large language model can actually understand that video. And so now I can ask it to play and be like, oh, no, no, skip to step four, because I already know one through three. And the model understands that video. And if you go back, that's like how I would talk to you is how I want to talk to my assistant. So it comes back to the beginning of, okay, well, what's an actual conversation look like when it's this back and forth medium, not just play a video, and I'm going to sit and be silent while the video plays. I might play the video, and then I might want to interact and talk to my assistant. I might want to jump to a command.

I might want to pause, go to the next command. But having this full understanding of the video or, you know, Lambda can theoretically understand the whole internet, right? And so you can ask questions, cross-referencing documents that you just can today. And when you think about that kind of knowledge that can come into play, this need to have more of a conversational interface just becomes more apparent. Like Meno said, most of this isn't possible yet, but this is very much where Google is headed. Step one is making Google easier to talk with, which is starting to show up in little ways. The big Google Assistant news this week is that you can talk to the Google Nest Hub Max just by looking at it and then speaking to it. No hot word necessary because it's using on-device gaze recognition to notice you looking.

On the same device, you can also just say, set a timer for 10 minutes or turn off the kitchen lights, and Google understands the commands without needing an OK Google ahead of time. And in general, since so much of this stuff is happening on device, it's both more reliable and lower latency and generally just much more usable. So that's the big picture stuff. We're going to take a quick break and come back with more on the future of Google. Support for this show comes from Shopify. Every thriving, successful business has to start somewhere.

A good place to start is a relatively simple question. What if, given the right tools, I've really put my all into this. One tool that can help grow your sprouting business to new heights is Shopify. Millions of businesses around the world rely on Shopify for e-commerce. They offer a host of helpful tools you can take advantage of, from payment processing to analytics to website design. Their design studio includes hundreds of templates to help you create the exact website you've been envisioning for your business.

If you're wondering, what if I need help, then no worries because you're never left to fend for yourself. Shopify's award-winning customer support is available 24-7. It's time to turn those what-ifs into a thriving business with Shopify today. Sign up for your $1 per month trial today at Shopify.com slash vergecast. Go to Shopify.com slash vergecast. That's Shopify.com slash vergecast. Support for the show comes from Upwork. The days of doing it all, all by yourself, are over.

There's no romance in burning out while you're trying to scale. Instead, you can check out Upwork. Upwork helps grow your business by giving you fast access to specialized talent across more than 125 categories, so you can fill skill gaps, launch projects faster, and scale without committing to full-time headcount. And finding the right talent is easy. You can browse profiles, review past work, and get help scoping the role so you can get started quickly.

Seriously, you could connect with the right freelancer in just a few hours, especially when you sign up with Business Plus. Their AI powered shortlisting pairs you with the top 1% of talent in under six hours. No endless searcher required. You can visit upwork.com right now to post your job for free. That's Upwork.com to connect with top talent ready to help your business, That's UPWRK.com. Upwork.com. Welcome back.

So the reason I wanted to start by talking about search is that, well, Google's a search company. But one thing that really stuck out to me over the last week is that Google is also increasingly a hardware company. It built the tensor chip to be able to do more of its AI work on devices and has clearly begun to bet big on the pixel lineup in general. And for now, Google execs all over the company, acknowledged that practically everything Google wants to do has to start with your smartphone.

But when Google talks about the smartphone, it actually has a vision that's much bigger than just the smartphone. Here's Rick Osterlo, Google's SVP of devices and services. Our vision of where the future is is that it's totally different, that people will interact with devices in a way that's almost like it's one computer. You know, they interact seamlessly across different modalities, different form factors, different use case scenarios, context. They'll be able to interact with computers in various different ways, voice, tapping. And in the future, a lot of things will be done proactively for you. So you don't even really, it's like an agent is doing things for you. And we call this vision ambient computing.

We want to bring all of Google's latest innovations in hardware, software, and AI, put it together for a user to offer the maximum possible help we can be as helpful as we can for users. That vision is one that's very hard to build. It means you have to have expertise across a lot of different arenas, both in devices, but also in a number of different technologies in AI and software and now Silicon. Certainly for the foreseeable future, we feel like the most crucial part of that is the pocketable computer, the mobile phone, that continues to be the center of most people's computing focus. And certainly today, the most important element of a computing future.

Google in general is making this big push away from being a search box you type into and more of an always-on experience, a thing helping. you and working with you and sometimes literally talking with you, like the movie her, all day. And that means Google better make damn sure it's powering the phone in your pocket. The only way this works is if Google is everything and everywhere. Rick showed me the new Pixel 7 and Pixel 7 Pro, the new flagship phones the company is teasing this week but fully launching in the fall. And also the new Pixel 6A, the $449 phone launching now.

And Rick said that the key to the Pixel 6A is bringing all that AI, all of that Googling to as many people as possible without changing the experience. Well, I think this is really important is that it takes a lot of the capabilities we've offered for the first time with TensorFlow on Pixel 6 and 6 Pro and brings it to a different price point, brings it to a lower price point. 449, beautiful device. You know, it has Google TensorFlow. There's goods back on the rest of the device.

I think people are going to see it for great value for money. But the big punchline is that you get all of the AI power that comes with TensorFlow on a much more accessible price point. Google is also finally launching its own smartwatch. It's called the Pixel Watch, and it's a nice-looking round smartwatch running Google's Wear OS. It'll do normal smartwatch things, like tell the time and get notifications and work with Google Assistant, but it's primarily being sold as a fitness device with Fitbit built in.

You'll be able to sync with a Fitbit account. You'll be able to get all of Fitbit's latest algorithms for health and fitness, be able to track against your personal goals. Everything will, you know, sync back to the Fitbit. Fitbit service so you can see it on the web, on your app, and it's deeply integrated throughout, you know, the watch face and throughout the device. In addition to all that, Google's also making a pair of high-end earbuds called the PixelBuds Pro, which are basically, as far as I can tell, exactly the same as the AirPods Pro, but for Android devices. And Google is even going to make an Android

tablet after it spent so many years, basically not giving a crap about Android tablets. So what's going on here? Why is it? is Google suddenly going big and wide on all these devices? It had a brief run with smartwatches in the middle of last decade, before essentially appearing to give up on the idea. Now it's saying, no, just kidding, we're back, we love smartwatches, wear OS forever. And same with tablets. Every Android fan has fond memories of the Nexus 7 tablet, but that was nine years ago.

And Google not only stopped making tablets, but stopped seeming to care about Android for tablets at all. Now both the hardware teams and the Android teams told me they're all in on both counts. So what gives here? Samir Samat, a VP of product management on the Android team, gave me what I think is pretty much the correct explanation. We were early pioneers in watches, for example. We were actually very early in multi-sized form factors for tablets. There's been a number of different form factors that we've worked on. But I think consumer expectations have changed as well over time. So, you know, the phone is certainly super important. But it's also becoming very clear that there are other device form factors, which are complementary and also critical to, a consumer deciding which ecosystem to buy into and which ecosystem to live in.

That last thing he said, that thing about the ecosystem, that's the whole thing here. What Google wants is to not be a search engine, but to be a sort of AI companion that understands and adapts to you at all times. It's a cool vision, frankly, and it makes a lot of sense for Google to be following. It also requires Google to be essentially all things to all people at all times. It has to know everyone's taste, work on everyone's devices, speak everyone's language, understand everyone's workflows, and be perfectly personalized to every single person on planet Earth. That's a really high bar, even for Google. And if you're buying an iPhone because you love your Apple Watch, that makes life even harder for Google.

Apple has long understood that an ecosystem is powerful, that you can sell devices and services through your other devices and services. Google, at long last, is finally figuring that out. That idea has forced Google to change a lot about how it works, even at a company level. Here's just one good example. For years, Google treated its Android and hardware teams like two totally separate things, and actually used to talk about how Android treated Pixel just like any other Android manufacturer. Now, Rick told me, the two sides are getting much closer together.

In our very earliest days, it was like we're making platform software, and then we would make devices on that software. it's very different now and that we really sort of co-design where things are headed. And I think that's just sort of the nature of how computers have changed and computing models have changed. So like, you know, the better together initiative is like a key part of AME computing. The reason we're doing both of these things is that this is our vision for how the model of computing changes in the future and we're working on it together. We're making sure that our hardware complements the software that's being built and vice versa. That's one way this

push towards ambient computing is changing Google, but it's not the only one. Google knows it's set up in this crazy decentralized way, and it knows that to get to this ambient computing everything connected future, the company itself has to change to solve it. And if Google wants to do that, it's going to have to keep being focused on making all of its parts work together. Google has a long history of building something, getting sort of bored, and then building a new, similar thing that doesn't work with the old thing. That's how you, you get a thousand Google messaging apps over the years. That's why Google inexplicably operated two competing music services for a bunch of years. It's why it bought Nest but also built its own

operating system for the internet of things. It's why Google has worked on countless AR and VR projects but has never actually had one that took off. The history of Google's failed projects is like a crazy alternate universe of the biggest stories in tech over the last 20 years. And it's rarely because Google is dead wrong, though sometimes it is like when it tried to make buzz and wave and Google Plus happen. It's because the culture of Google is to try everything and kill pretty much all of it. Why? Because frankly, Google doesn't need any of those things to work. It can continue to fritter away all those good ideas as long as the search ad money keeps chugging along. And make no mistake, it is still chugging along. But you've got to think that for Google, faced with antitrust reform

and new privacy rules and even companies like Amazon starting to take a bite of the ad business, Google's feeling a little more urgency as it thinks about what comes next. It also has to continue to convince users, many of whom are increasingly wary of how much Google knows about them, that the trade-off for all that data is worth it. And if you really cast out in the future, there's one other piece of this too. If you think augmented reality is the future of computing, and a lot of people at Google do, this kind of ambient computer Google is building seems like exactly the kind of thing we're going to need. Nobody wants to walk down the street yelling, hey, Google, message Anna, I love you.

We're also not going to want to switch between eight devices just to get things done. We're going to need computers that are massively versatile and switch seamlessly from place to place and task to task and input to input and don't particularly care about their form factor. Google wouldn't tell me much about what it's working on here for what it's worth. This is all I actually got out of Rick Austerlo. It's definitely a place where we are focusing as well. And we don't see the market as here now, but we definitely see it as a key part of the future. For the record, ambient computing also isn't here now. But Google has built a lot of the pieces to make it work without thinking much about how they work together.

Now the job for Google is to take all those pieces and turn them into something magical, something more than the sum of its parts. That comes in really small ways, like Google making it easier to move files or cast video between all your devices. And it comes in really big ways, like a virtual assistant, you can talk to anywhere, about anything, any way you want. And that, at least it seems to me, is what Google is actually working on now, maybe for the first time. The ongoing question is, will this vision hold Google's interest long enough to get anywhere, or will something shinier come along and just distract the company again?

Who knows? Maybe next year's I-O will be all about the metaverse, and who cares about your life in the real world? But I would bet that if Google is ever going to find its next big thing after search ads, ambient and computing might be the best chance it has. That, or really awesome AR glasses. That's it for the show today, but we are definitely not done talking about Google I.O. We'll be back on Friday with all kinds of talk about all of the announcements from this week.

Plus, we're going to be spending some time with Sundar Pichai, the CEO of Google and Alphabet, getting into all of the stuff about what Google's up to. So stick around. This special episode of The Vergecast was produced by me, David Pierce, Liam, and Andrew Marino. The Vergecast is a production of the Vox Media Podcast Network. We'd love to hear from you about this show and everything. Please send us an email at Vergecast at theverge.com. Thanks for listening.

We'll see you on Friday. Rock and roll.