The Indicator from Planet Money - The dawn of search engines

Episode Date: May 26, 2025

Today on the show, we bring you a special episode from the Understood feed at CBC podcasts. It's an excerpt from a series called Who Broke the Internet hosted by Cory Doctorow. The four part series details his criticisms on the state of the modern internet and what we can do about it. From his conversations with Eric Corly the publisher of 2600, an iconic hacker magazine, best known under his hacker name Emmanuel Goldstein, to Clive Thompson a tech and culture writer to Steven Levy the author of "In the Plex: How Google Thinks, Works, and Shapes our Lives" this excerpt digs into how search engines started.You can listen to more of the podcast here.Related episodes:The hack that almost broke the internet (Apple / Spotify)For sponsor-free episodes of The Indicator from Planet Money, subscribe to Planet Money+ via Apple Podcasts or at plus.npr.org.Music by Drop Electric. Find us: TikTok, Instagram, Facebook, Newsletter. See pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences.NPR Privacy Policy

NPR. This is The Indicator from Planet Money. I'm Adrian Ma. Today, we've got a special guest episode for you. It's an excerpt from a series called Who Broke the Internet from our colleagues up north at the Canadian Broadcasting Corporation. It's from their understood feed at CBC Podcasts, and the host is Corey Doctoro.

You might have heard of him? He's a journalist, blogger, science fiction writer, and internet commentator. And his series digs into his criticisms on the state of the modern internet, and what we can do about it. So in this excerpt you're going to hear after the break, Corey's going to talk about how search engines got started. Eventually, the internet got too big for people to know where everything was, so waves of clever people created search engines.

Early search engines just looked for pages containing the words you typed, giving priority to pages that contained more of those words. This worked okay, but when it failed, boy, did it ever fail badly. If you wanted your page to rate high on the search results for a query like Mexican food, you could keyword stuff it by adding the words Mexican food a thousand times in tiny white-on-white type to the bottom of the page. The primitive search engines would count these all up and conclude that your page was the most important Mexican food resource in the world, which is useless because most hungry people aren't looking for a site that just has the words Mexican food, a thousand times.

To weed these bad hits out of your search results, you'd have to master all kinds of arcane search engine syntax so you could exclude words and phrases, putting minus signs in front of the kind of junk that was typical of spam sites. It took forever, but those search engines were all we had, so we kept using them. We were using really awful search engines

that we thought were pretty good. And you saw Google, and all of a sudden the film cleared from our office, We thought, wow, you could actually find what you were looking for on the internet right away. This is Stephen Levy. He wrote the canonical history of Google, in the Plex, how Google thinks, works, and shapes our lives. Back in the early 90s, Stephen was Newsweek's tech reporter, and he knew right away that Google was a story. When I first saw the Google search engine, my mind was blown.

So he tracked down their PR person. I called her up. I said, I got to meet these guys. These guys being Google's founders, two Stanford grad students named Larry Page and Sergei Bryn. So Stephen booked a ticket from New York to San Francisco and flew out to meet them. It was near Halloween. Everyone was dressed in a Halloween costume. So Larry was dressed like a Viking. And Sergey was dressed like a cow. He had these big plastic udders coming out from his chest. So the Viking and the cow took me into a little room and explained how page rank works.

PageRank is the algorithm behind the magic, a new way that Google had developed to deliver its search results. It was totally audacious. And I was so audacious because the way they were able to locate the best result for your query was by basically, not basically, literally downloading the whole web, all of it, which some people thought was ridiculous, but Larry Page understood this could be done. All Larry and Sergei needed to pull it off was big, powerful servers, which they could get. Because they had access to servers at Stanford, and they would literally hijack them as they came in the loading dock meant for other departments.

With all this yoint power, they set up their first server, building the case for it out of Lego. Yeah, Lego, because it was cheap and cool and colorful, like the eventual Google logo. And then they would do math, which was like Sergei's specialty, to analyze all the links that happened between websites and figuring out what important websites linked to other sites. So that way you could filter out which were the perfect answers for the queries you put in. into the search field and that turned to be just a quantum leap better than the previous things. The early internet wasn't perfect. Google came into existence because the search landscape sucked. But the point is, Google could come into existence. There was oxygen and sunshine that reached the forest floor so new things could grow. If you could code, if you had some money, a computer or two,

a couple of bins of Lego. You could. You were allowed to. to build something new that made the web better. Clive Thompson again. They start off by having a really good search engine, the best search engine, and the reason why it's so good is that they are in ferocious competition with like 12 different search engines,

and they have to be better than everyone else. They have to fight to the top, and they do that. Just be the best. Make the best product, and people will switch. And we did. We all switched to Google because it worked. and that mattered to Google in their early days. And there's, you know, the very famous motto that Google had with don't be evil, you know,

which was coined by an engineer when, you know, someone from HR had a meeting. And, you know, so what is our motto? What is our, what is our value? And, you know, one guy said, what are you talking about? It's just like, don't be evil, you know? I mean, that's it. So back in 1999, Stephen wrote that first story for Newsweek, and he kept writing about Google. Google's world was our world.

You know, things didn't exist, but it wasn't on the web. It's like when Google did well, we all did well, because we could find the stuff we were looking for. Right, and that came into play even more when they came up with their first big successful business product. That was ads. Yeah, yeah. When Google met Advert. advertising, everything changed. The introduction of ads to Google marked a major shift.

Emmanuel Goldstein again. Google in their early days, you didn't have ads on the front page. It was relatively fast. And they just seemed like cool people, inventing cool things. Larry and Sergey actually hated advertising. And the paper they wrote, talking about page rank, they had a little thing in the end saying, boy, we hate advertising. It's terrible.

Stephen's talking about an article that Google founders published in a journal in 1998, and in it they explicitly wrote that, quote, advertising income often provides an incentive to provide poor quality search results. But they had to make money. So then they figured, well, maybe there's a way to do ads that wouldn't be terrible. So in 2000, the founders assigned an engine. engineer to figure this out, one of Google's first aid employees. And he came up with something.

It's called AdWords. So what it would do was when you put a query into the search engine, it would start an auction instantly. And your search term would be auctioned off to advertisers who wanted to sell a product that was related. If you search for Mexican food online, on Google back in the early aughts, the results would have been the most linked to websites for Mexican restaurants in your area and reviews from local blogs and newspapers.

And now with AdWords, ads would appear in a colored bar, atop the normal page-ranked results. You know, you get the same results, but you get a new extra little, you know, a set of possible results that actually might even help you if you were looking to buy something related to the search term. And the advertisers would get an audience which, you know, was primed to buy something. And it worked for Google. It worked for advertisers. And most of all, it worked for users.

That is until it didn't. Thanks to our colleagues over at the Canadian Broadcasting Corporation. If you want to hear more of Understood Who Broke the Internet, you can find a link in our show notes or find it wherever you listen to podcasts. The original version of this episode was written and produced by Matt. at Muse, A.C. Rowe and Corey Doctoro. Roche Neyer is the coordinating producer. Julian Uzielli did the mixing and sound design.

Story editing by Veronica Simmons. The executive producer is Nick McCabe Locos. The indicator version of this episode was produced by Cooper Katz McKim and engineered by Cine LaFredo. Kate Cannon is our editor and the indicators of production of NPR.