Benjamen Walker's Theory of Everything - The Clouds (part 1 of 3)

Episode Date: April 19, 2013

Twitter employee #7 tells us what happened when Justin Bieber joined twitter in 2009. An Amazon Data scientist, explains how the cloud is changing our relationship with technology, Obama���s CTO Harper Reed explains why the cloud is awesome + we tour Parse, a hot hot hot (BaaS). But can your host get inside the cloud? *********Click on the image for the whole story about this week’s installment**********

You are listening to Benjamin Walker's Theory of Everything. At Radiotopia, we now have a select group of amazing supporters that help us make all our shows possible. If you would like to have your company or product sponsor this podcast, then get in touch. Drop a line to sponsor at radiotopia.fm. Thanks. episode. Why is there something called influencer voice? What's the deal with the TikTok shop? What is posting disease and do you have it? Why can it be so scary and yet feel so great to block someone on social media? The Neverpost team wonders why the internet and the world because of the internet is the way it is. They talk to artists, lawyers, linguists, content creators, sociologists, historians, and more about our current tech and media moment. From PRX's Radiotopia, Never Post, a podcast for and about the Internet.

Episodes every other week at neverpo.st and wherever you find pods. You are listening to Benjamin Walker's Theory of Everything. The following installment is called The Clouds, Part 1. On May 11, 2009, Justin Bieber joined Twitter. Check out my single one time on my MySpace and spread the word for me. Thanks. O-M-B-E-B-E-R Follow me back. I love you. My boyfriend just joined Twitter.

Immediately, legions of the Biebs' followers began retweeting his message and pinging him with OMGs and follow me back. R-T-R-T less than sign three. Meanwhile, back at Twitter HQ, chaos was breaking out. You know, it's like people are running around with fire hoses and just like spraying them on servers. The whole company is like screaming, Bieber! Britt Savatell was Twitter employee number seven.

He says Justin Bieber's arrival was an all-hands-on-deck emergency. I feel like it's like the same thing as like if you're in the Navy, like you're in a submarine and the red lights start flashing. I mean, it's just insanity. It's completely insane. Now, let's fast forward two years and two months. July 11, 2011. This is the day Justin Bieber signs up for an account on Instagram and uploads a picture of some cars on the 101. LA traffic sucks.

Around this time, Brit Savatell ran into Mike Krieger, one of Instagram's co-founders, at a party. I said, oh, so Bieber joined your site this week. How are you feeling? And he effectively said, you know what, it was interesting because Amazon called us and said, hey, we noticed that Bieber joined your site and we've upped the number of instances that you've got

and it looks like the load is doing fine. Hearing that was so emotional on two levels. On number one, I was pissed because I wanted this poor story from him. And two, it's just amazing to compare and contrast these two experiences. Now, what makes these two Justin Bieber stories different is that the popular photo sharing application Instagram built its platform using Amazon Web Services, the cloud.

Amazon Web Services basically provides computers, servers, storage, and databases, which are available exactly like a utility. Matt Wood is Amazon's principal data scientist. The company built the cloud, he told me, to power its online e-commerce sites, Amazon.com and Amazon.co.uk. It was a massive global network of computers. But then, one day, Amazon realized it could make money renting out unused space in its cloud. The technological breakthrough is the availability of the resources to build out these services

and to be able to do that at scale. So it's allowing customers to focus back on their applications like Instagram and allows people like Justin Bieber or myself and you to use those services to share our thoughts and our images and our videos with our friends and more broadly. The real technological breakthrough, though, is actually a simple idea. Provide customers with computational power and charge them the way a utility would.

The advent of delivering computing as a utility really enables some pretty fantastic applications. We have customers running everything from cancer trials, looking for new mechanisms of identifying and curing cancer, all the way through to NASA running robots on the surface of Mars, all the way through to people sharing the interesting things that they find on the internet through sites like Pinterest, sharing where they are with their friends with sites like Foursquare, or sharing what they think about things through sites like Reddit. Amazon officially launched its web services in 2006, and at some point between Justin Bieber joining

Twitter and joining Instagram, the cloud came into its own. The exact date doesn't actually matter. What's important is that most of us, including technologists like Brit Savatel, didn't even notice. A lot of people don't realize that with something like AWS, you can effectively press a button and have everything you need to go live in minutes. That's so amazing. I kind of missed this revolution because I was so heads down at Twitter the whole time. I missed this revolution too. But unlike Brit Savatelle, I don't have a good excuse. In 2009, I began producing and hosting a radio program called Too Much Information,

a radio show about life in the digital age. By the time Justin Bieber had joined Instagram, I was regularly making radio programs about new technologies, social media, crowdfunding, WikiLeaks, online privacy. And even though all of these things connect to the cloud, the cloud itself was abstract, immaterial. It never even registered on my horizon. Of course, today in 2013, the cloud is impossible to miss. It's everywhere, at once, and on all the time. Authoritarian governments may still have the power to turn off the internet, but they can't keep their citizens from using the cloud because borders and nations are irrelevant to the cloud. But as the cloud grows, its abstract immateriality is growing too. And I find this extremely

disconcerting. It's like something important is taking place right in front of me, and yet I can't see it. So I decided to ask one of the smartest technologists I know for help. You know, if you're not in the cloud and you're doing technology, then you probably have too much of a neck beard and you should probably shave. I'm not sure what exactly Harper Reed has against neck beards, as he sports some pretty wild facial hair himself. But I am sure that Harper Reid knows a lot about the cloud. He was the 2012 Obama re-election campaign's chief technology officer,

and the campaign used the cloud for everything. It supported 100% of our apps, from our mobile app to call tool to dashboard to some stuff on the www.barackobama. It powered our identity service, which allowed you to log in to all these great things. We would not have been able to accomplish what we accomplished at Obama for America without cloud infrastructure. Full stop. It's just not possible. Amazon's cloud infrastructure enabled Harper Reed and his team to prepare for the worst imaginable scenarios, like a complete East Coast election night blackout. We had Hurricane Sandy, a whole other type of cloud system, was coming to attack us. And so we were able to basically light up a copy of our infrastructure on the other coast, on the West Coast. And with, I mean, a lot of work, of course.

But this allowed us to, with just a click of a button or a short script, migrate everything, all at once, to the West Coast. As amazing as that hurricane-proof, coast-to-coast backup hard drive sounds, what really made the difference, Harper Reed told me, was the flexibility of the cloud. Amazon's utility model allowed the campaign to build scalable applications like their call tool. This, in a sense, was a cloud application. There was no software you installed. You didn't have to have an account if you didn't want one. And it was directly linked to cloud servers. So if we had, let's say, 10 callers, that meant one server. And I'm making numbers up because I don't have specifics.

But it was a very simple formula. So we could look at it and we could say, okay, for every caller, we need this much resources. So our capacity was directly linked to the number of volunteers we had at a given time. And it was a very fun kind of thing to watch. You'd watch the number of callers go up, and you'd watch the number of servers go up. And the callers went down at night, and the number of servers went down at night. There were times when we wouldn't even notice it had worked. That's how well it worked.

And so you'd just be like, I wonder how many servers we got over here. And you'd look and you'd just be like, holy moly, it's a lot of servers. Amazon provides the resources so that an application can handle Justin Bieber or election night levels of activity. But it doesn't charge when these extra resources aren't being used. And Amazon's prices for cloud services are low, like lower than the cheapest books on Amazon.com low. So it's a pricing system that makes scale available to anyone. What the cloud products allowed us to do, which was awesome,

is allowed us to build this without determining scale. And I keep going back to this, because this is really what the cloud did for us. It said to us, hey guys, why don't you just use a little of me when you need to use a little of me, when you need to use a lot, just use a lot. The Obama campaign, along with Instagram and the other pioneering companies who've used Amazon's web services, have in a sense vetted and tested out the cloud for the rest of us. And the results, Harper-Reed says, are undeniable. I think at this point, the cloud is here to stay, and it's just how things are.

I can't think of technology without it. Like, it is the only way for us to do things from here on out. There is no other path. As much as Harper helped me understand how the flexible infrastructure of the cloud works, there are a number of things that still trouble me. A couple months ago on the subway, I saw this ad for the cloud. It wasn't an ad for Amazon, but one of their competitors. It was a picture of a man climbing up a mountain, the face of a mountain, with his bare hands. He didn't have a shirt on, and you could tell he worked out in an expensive gym, perhaps with a personal trainer. And there was an iPad or a tablet tucked into the back of his

shorts. Apparently, he was so good at climbing, he didn't even have to worry about it falling out. In the sky, written in big white fluffy letters was a sentence that read, experience the mobility and freedom of working in the cloud. Now, I know what freedom and mobility are, but for the life of me, I just could not imagine what this guy's job was. And I rode the subway around for hours just staring at this ad, trying to envision what it was this guy could possibly do. Did he have co-workers? Insurance? I mean, am I the only one who finds this suspicious?

This guy is asking us to follow him up the side of the mountain, but what if we all end up like lemmings, going down, head first, over the other side? This guy's job seemed just as immaterial and abstract as the cloud itself. Perhaps it would make more sense if I could just talk to someone who actually does work in the cloud itself. Perhaps it would make more sense if I could just talk to someone who actually does work in the cloud. So I flew to San Francisco, home of Twitter, Instagram, and a million other cloud companies. In San Francisco, there are ads for cloud businesses and cloud conferences everywhere, and people walking around in t-shirts with cloud logos. I even met a guy who made a song about the cloud. We're gonna turn it up.

We're gonna rock your clouds. This is the open future. That's how we're talking about. So baby, listen close. We're gonna rock your clouds. Because you know you gotta get this. It was not difficult to find someone to talk to. I work on the back end.

I administer databases and queuing systems. Basically all of the bones of the platform. Charity Majors is a systems engineer at Parse. They may not have an auto-tuned song about the cloud on their website, but they do provide cloud infrastructure for mobile applications. Amazon is the cloud for servers, and we are the cloud for mobile apps. I mean, if you think of a technology stack, we're just, we're the same concept, but a little bit higher up the stack. Companies use Parse to launch apps that can scale big over a short duration of time.

Apps for intensive marketing campaigns, mobile games tied to college football seasons. The industry speak name for this is back end as a service, computing as a utility. And it's really taking off, especially with startups. Small startups have really taken to back-end as a service, just like a fish to water. I mean, they're amazing because they free people up to do what they do best. The scarcest resource in any startup is almost never money. It's the time and attention of the people in the startup,

especially developers. One afternoon, around the time Facebook acquired Instagram, some of Charity Major's developer colleagues decided to see how long it would take them to develop something with similar functionality. We were able to create a clone of Instagram in 30 minutes of development. It's called AnyPick, and it's start to finish. The functionality is identical to the original Instagram app,

but because we didn't have to build any of the backend, it was already built on Parse, all that our developers had to do was just make the app. But the cloud is not just for startups. According to Matt Wood at Amazon, giant enterprise companies are also discovering how quickly they can set up shop in the cloud. So News International actually recently purchased the rights to host near real-time feeds of Premiership football. And they were told that it was going to take several months for their IT department to go out and build the infrastructure required to host and deliver the Premiership football matches to their customers.

But because they run on Amazon Web Services, they were able to put in the infrastructure that they needed in minutes and hours rather than having to wait several months. But what happens after a company moves to the cloud? Shell Oil, Matt Wood told me, recently instituted a cloud-first policy stipulating all new projects must be built using cloud resources. This policy completely changed Shell's IT culture. The IT guys have gone from being the guys in the meeting that have to say no to people, that the ideas that they're putting forward are too expensive and are going to take too long to implement, to being the heroes inside the organization. These are now the guys that can get things done. They're saying yes to all of these projects that are coming out to them,

and they can deliver those projects in a much, much shorter timeframe. So they can have an idea over coffee, spin up the resources that they need, try out their idea, and if it works, fantastic. If not, they can close that down at the end of the day, and they stop paying for it as soon as they do that. So in terms of Shell, their IT culture is now much more innovative. I actually didn't meet anyone in Silicon Valley who was unhappy about working in the cloud. In fact, when I asked Charity Majors if there were any drawbacks to her job, she laughed at me. I've been working as a systems engineer for, you know, over 10 years. And I remember being on call when that meant you literally got a phone call at three o'clock in the morning and you have to call a taxi or ride

your bicycle to the data center just to push a button and reboot a machine. And in the meantime, your entire service is down. These days, I don't have to go to data centers. I haven't been to one in years. We run on Amazon, so Amazon manages the data centers. This is another thing that concerns me about the cloud. It used to be that all of the smart people I knew claimed it would be disastrous if the internet ended up being defined by one way of doing things or by a single company. But today, everyone seems to be rushing to put as much of the internet as possible onto Amazon's cloud. Why isn't all this consolidation and concentration

setting off alarm bells? According to charity majors, it's because it's worth it. Obviously, it creates some big single points of failure. I mean, you see that whenever Amazon goes down and suddenly Netflix doesn't work, Instagram doesn't work. It feels like half the Internet goes down when Amazon goes down. You know, and Paris has over 60,000 apps. And if we go down, you know, all of those apps go down.

You know, and a lot of people have just said, well, you know, it's a deal that I'm willing to make because I go down less frequently. I don't have to be the one who is getting called in the middle of the night to deal with it. And everyone else is kind of in the same boat anyway. People still cling to the illusion of control. And they still think that they should do all this work themselves. They feel like if they own the hardware, they have control. And if they built the stack themselves, they have control. You know, if they can't see it and feel it and fix it themselves,

it's not as real to them. I don't think this is a very good reason to not use the cloud. I mean, Amazon may go down a couple times a year, but I guarantee they have better hardware engineers and data center engineers than almost anyone else does. Trading in the illusion of control for freedom and mobility. That does sound like a pretty good trade. But after talking with charity majors and Harper Reed, I realized that before I can embrace the immateriality of the cloud, I need to see it first, with my own eyes.

What I needed was a tour. So I set out to find someone who could take me into the cloud itself. And I found him. The guy, one of Amazon's top engineers. He lives on a yacht with his wife and his cat, traveling the world, building and maintaining Amazon's cloud. So what it looks like is it's really, we operate these services inside big buildings,

which we call data centers. Yeah, that's Matt Wood again. Actually, he's the only person Amazon's corporate communications would let me speak with. The guy on the yacht? No way, they said. In fact, they told me the entire back end of the cloud was off-limits. I spent weeks begging and pleading the people at Amazon, but eventually they stopped taking my calls.

It turns out that the immateriality of the cloud is like a trade secret. Amazon, HP, and Google, they're all racing each other to build the fastest and biggest clouds possible. And they're building these from scratch. They don't want any prying eyes. There's a co-location center in Silicon Valley called Equinox that houses some of Google's drives. But when Google arrived for the install, they insisted that the lights go out

so that no one could see the custom hardware as it was being carried in. Some data centers, I've been told, keep the lights off all the time. They really don't want prying eyes. Perhaps this is the way it has to be. Perhaps in order to enter the kingdom of the cloud, we must accept the immateriality and the abstraction as articles of faith. But just as I was set to give up on my quest, I learned about rare earth. Over the years, technology has become more and more invisible, but all of that invisible technology is really based upon increasing use of more and more exotic materials like rare earths. And it's the rare earths that make it all work.

That's Alex King, the director of the Ames Research Lab for Critical Materials. And in part two of this story, he'll reveal that at the heart of the immaterial cloud, in fact, powering it, exist very real materials. Rare Earths. This episode was produced by myself, Benjamin Walker. Bill Bowen did the sound design. And I got editorial assistance from Karen Furlman. Special thanks to Amy Mayers and Amber Cortez for help with the recordings,

and to John Barth and the PRX Global Story Project, who made it possible for me to travel to China, where 95% of the world's supply of rare earth comes from. So I'm definitely a novelty here. I'm standing out. I wouldn't even say that you're a novelty at this point. Here you might be a little bit of a liability. You can hear all about what happens to me on that trip,

and then some, in The Clouds Part 2. You've been listening to Benjamin Walker's Theory of Everything. This installment is called The Clouds, Part 1. Radiotopia from PRX