The Good Tech Companies - Surviving the Google SERP Data Crisis

Starting point is 00:00:00 This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Surviving the Google SERP data crisis, by Bright Data. Greater than revolving light breaking news. Google now requires JavaScript to perform searches, yes, you read that right, your trusty old automated SERP bot relying on HTTP clients and HTML parsers? Stop sign completely busted, this shakeup has wreaked Havocon countless CO tools, causing data delays, outages, and a buffet of service meltdowns. But why did this happen? What could be Google's reason behind the change, and how can you deal with it? Were all tools affected? Most importantly, what's the solution? Thinking FaceTime to find out.

Starting point is 00:00:42 What's the deal with Google requiring JavaScript to perform searches? Here's what you need to know. On the night of January 15, Google pulled the trigger on a major update to how it handles and tolerates automated scripts. Robot detective the culprit, JavaScript. JavaScript execution is now mandatory to access any Google search page. Without it, you're met with what some users have dubbed the scriptal. A block page that laughs in the face of old-school bots. Cold sweat smile the result? Full-scale confusion, rank trackers, SERP data tools, and SEO services everywhere either stopped working entirely or began experiencing outages and data lags. Collision as Google shared in an

Starting point is 00:01:23 email to TechCrunch greater than enabling JavaScript allows us to better protect our services and users from greater than bots and evolving forms of abuse and spam. The reason behind this move? According to the same spokesperson, on average, fewer than 1% of searches on Google are done by users who disable JavaScript. Sure, that makes sense and 0.1% seems like a tiny number, until you remember it is Google. Astonished face we're talking about millions of searches. And guess what? A huge chunk of that's liver likely comes from CO tools, web scraping scripts, and data aggregation services. So, is this a direct swipe at CO tools? Why now, and what's the real story? Let's dive in and find out.

Starting point is 00:02:07 Face with monocle hold up. Are CO tools the villains here? TLDR. Nah, not really. Google probably did this to protect against LLMs, not SEO tools. As Patrick Hathaway, co-founder and CEO of Sightbulb, pointed out on LinkedIn, this isn't likely to be an attack on SEO tools. These products have been around since the early days of search engines and don't really harm Google's business. But large language models, LLMs, might. It's no surprise that Chad GPT and similar services are emerging as rivals to Google, changing the way we search for information. Patrick's point makes sense, although it's still unclear exactly why Google made these changes, as the company hasn't released an official statement. Person shrugging the scriptal move isn't about blocking web scraping, it's about

Starting point is 00:02:55 protecting Google's ranking system from new competitors, hello, AI companies. Google is making it harder for these competitors to cite pages and use SERP data, forcing them to build their own internal PageRank systems instead of comparing their results to Google's. Paper SEO Data Outage – The Fallout of Google's Latest Scraping Crackdown The fallout from Google's new policies is straightforward. Many SEO tools are struggling, going offline, or facing major outages and downtimes. Downward trend users are reporting serious data lags in tools like SEMrush, SimilarWeb, RankRanger, SE Ranking, ZipTie. Dev, also asked, and likely others caught in the chaos. It's safe to say most players in the CO game felt the hit. Bullseye If you check X, you'll find plenty of comments from frustrated users alike and updates from industry insiders https colon slash slash x com tomec rudsky status one quintillion 879 quadrillion

Starting point is 00:03:54 894 trillion 440 billion 736 million 895 473 mx equals 2 and embeddable equals truehttps colon slash slash x com glenn gabe status 1 quintillion 879 quadrillion 921 trillion 149 billion 179 million 539 708 embeddable equals true a side effect of google's SEO changes? The struggle to scrape accurate SERP data might be messing with how SEO tools track rankings, leading to potentially unreliable results. Bar graph don't believe it? Just look at the SEMrush volatility index after January 15th. That sudden spike is hard to ignore. Fearful face was it because of SEO tracking issues or some other changes in Google's algorithms? Tough call, headless browsers as the answer to Google's new,

Starting point is 00:04:51 scriptal. If you've checked out our advanced web scraping guide, you probably already know what's the fix here. The answer? Just switch to automated browser tools that can execute JavaScript, tools that let you control a browser directly. After all, requiring JavaScript on webpages isn't exactly a real blocker, unless Google pairs that with some serious anti-scraping measures shield. Well, if only it was that easy, switching from an HTTP client plus HTML parser setup to headless browsers like Playwright or Selenium is easy. The real headache? Browsers are resource-hungry monsters, and browser automation libraries just aren't as scalable as lightweight scripts parsing static HTML. Balance scale the consequences?

Starting point is 00:05:32 Higher costs flying money and tougher infrastructure management for anyone scraping CO data or tracking SERPs. Trophy the real winners? Oz, GCP, Azure, and every data center powering these heavyweight scraping setups. Grimace the losers, the end-users, if you don't choose the right CO tool, prepare for price hikes, more frequent data lags, and, yep, those dreaded outages. How Bright Data's SERP API dodged major outages. While many CO tools were thrown off by Google's changes, bright data stayed ahead of the curve. Biceps How, our advanced unlocking technology and rock-solid architecture were designed to handle complex challenges like this. Google isn't the first to require JavaScript rendering for data

Starting point is 00:06:16 extraction. While other CO tools, focused solely on Google, scrambled to build JS rendering from scratch, we simply adapted our SERP scraping solution to leverage the robust unlocking capabilities we already had in place for hundreds of domains. Glowing star thanks to a top-tier, dedicated engineering team specializing in web unlocking, we quickly addressed this fallback. Sure, the update threw the industry for a loop and caused some outages, but BrightData's response was lightning fast. High voltage as you can see, the outages were brief, lasting only a few minutes. In under an hour, our team of scraping professionals restored full functionality to BrightData's SERP API. BrightData's web unlocking team kicked into high gear, stabilizing operations at lightning

Starting point is 00:07:02 speed while keeping performance rock solid without inflicting additional costs on users. A critical factor as many of our existing users started shifting 2 to 5x more traffic our way to meet their demands. Briefcase how did we pull it off? 3 o'clock with our advanced alert system, high request scalability, and a dedicated R&D team working around the clock, we had it fixed before any other CO platform could react, and well before customers even noticed. Exploding head this is the power of working with a company that goes beyond basic SERP scraping. With world-class scraping tools, professionals, and infrastructure, Bright Data ensures the availability and reliability of its products. Fire no surprise here, Bright Data's SERP API ranked number one

Starting point is 00:07:46 in the list of the best SERP API services. Trophy want to know more? Watch the video below. https://www.youtube.com. Watch? v equals ma5 underscore hi2ac and embeddable equals true summary. Google has just rolled out some major changes that have shaken up the way bots scrape and track SERP data. JavaScript execution is now required, and this has led to outages, data lags, and other issues across most SERP tools. Warning in all this chaos, Bright Data cracked the problem in under an hour stopwatch, ensuring minimal disruption and continuing to deliver top quality SERP data. If you're dealing with challenges in your CO tools or want to protect

Starting point is 00:08:29 your operations from future disruptions, don't hesitate to reach out. We'd be happy to help. Hand-waving thank you for listening to this Hackernoon story, read by Artificial Intelligence. Visit hackernoon.com to read, write, learn and publish.

The Good Tech Companies - Surviving the Google SERP Data Crisis

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.