The Good Tech Companies - Elevate Your Scraping Project With Puppeteer Extra

Starting point is 00:00:00 This audio is presented by Hacker Noon, where anyone can learn anything about any technology. Elevate your scraping project with Puppeteer Extra, by Bright Data. Ash highlighted in our guide to web scraping with Puppeteer. This browser automation library is a fantastic ally for extracting data from dynamic content sites. Still, like any other tool, it has its shortcomings. That's where Puppeteer Extra steps in. In this guide, we'll introduce you to a library that wraps to extend it with plugin support. Get ready to take your Puppeteer scraping project to the next level.

Starting point is 00:00:37 Rocket What's Puppeteer Extra? Puppeteer Extra is a lightweight wrapper around that enables plugin integration through a clean interface. Although it's not developed by the team behind Puppeteer, this community-driven project has hundreds of thousands of weekly downloads and over 6k stars on GitHub Upward Trend. Check out the GitHub stars chart below, it's clear that the repo has been on a steady rise in popularity over the years. The plugins officially supported by Puppeteer Extra are Puppeteer Extra Plugin Stealth, to make it harder to detect Puppeteer as a bot. Puppeteer Extra Plugin Recaptcha to solve recaptchas in HCAPT CHAs automatically. Puppeteer Extra Plugin Ad Blocker to reduce bandwidth and load times by applying a fast

Starting point is 00:01:17 and efficient blocker for ads and trackers. Puppeteer Extra Plugin DevTools to make debugging the Puppeteer browser possible from anywhere. Puppeteer Extra Plugin DevTools To make debugging the Puppeteer browser possible from anywhere. Puppeteer Extra Plugin Reeple To make Puppeteer debugging and exploration easier and more enjoyable with an interactive REPL. Puppeteer Extra Plugin Block Resources To programmatically block resources like images, media files, CSS stylesheets, and more while loading pages. Puppeteer Extra Plugin Flash, to allow

Starting point is 00:01:46 Adobe Flash content to run on all sites without user interaction. Puppeteer Extra Plugin Anonymize UWA, to anonymize the header on all pages, with support for dynamic replacing. Puppeteer Extra Plugin User Preferences, to set custom Chrome, Chromium user preferences. On top of those, it integrates with the following community plugins. Puppeteer Extra Plugin Minmax to minimize and maximize the Puppeteer browser window in real time. Puppeteer Extra Plugin Portal to remotely view and interact with Puppeteer sessions via the Chromium Screencast API. Why do we even need an extra version of Puppeteer? No doubt, Puppeteer is one of the

Starting point is 00:02:26 top headless browser libraries for scraping and testing. But let's be honest, it has its limits, especially when fassing gante bot tech like browser fingerprinting and captchas. Read our guide to learn how to deal with reCAPTCHA automation. Websites armed with anti-bot defenses can easily detect and block Puppeteer scripts. If only there was a way to extend and block Puppeteer scripts. If only there was a way to extend and customize Puppeteer's default behavior. Well, that's exactly what Puppeteer Extra is all about. Puppeteer Extra is like a power-up for Puppeteer, adding plugin support to tackle those major drawbacks. Instead of overriding or extending everything for you, it wraps Puppeteer and lets you register only the plugins you need. Superhero. Set up in plugins for web scraping. You can add Puppeteer extra to your

Starting point is 00:03:10 project's NPM dependencies with. Warning note. Requires to work, so make sure both packages are installed in your project. Then, you have to import the object from instead of the library. Everything in the Puppeteer API stays the same, but you get a little extra magic sparkles. The object now exposes a method to plug in Puppeteer Extra Plugins. Time to dive into what these plugins can do, and see how they'll level up your web scraping game. Puppeteer Extra Plugin Stealth Puppeteer Extra Plugin Stealth, also known simply as Puppeteer Stealth, include us a set of configurations designed to reduce bot detection. It overrides Puppeteer's detectable properties and settings that might

Starting point is 00:03:50 expose it as a bot. For more details, check out our guide on how to avoid getting blocked with Puppeteer Stealth. Gear installation, light bulb usage, Puppeteer Extra Plugin Block Resources a plugin to prevent the Puppeteer browser from loading specific resources. These supported resource types include, resource blocking can be configured both globally and locally, gear installation, light bulb usage. You can then configure the resources to block globally on all pages. Similarly, you can locally select the resources to be blocked. Puppeteer Extra Plugin anonymize UAAin to anonymize the set by the browser controlled by Puppeteer. Face Mask It gives you the ability to strip the string from the Chrome user agent in headless mode and supports dynamic replacement of the user agent through a custom function.

Starting point is 00:04:38 See it in action in our Puppeteer User Agent Guide. Discover what's the best user agent for web scraping, gear installation, light bulb usage. Next, you can configure the anonymous user agent. Also, you can set a dynamic user agent via a custom function. Puppeteer Extra is not a panacea solution. Just like with Playwright, no matter how slick and customized your puppeteer script is, advanced anti-bot systems can still sniff you out and shut you down. But how is that even possible? Thinking face the documentation breaks it down for you. Greater than please note. I consider this a friendly competition in a rather interesting

Starting point is 00:05:14 greater than cat and mouse game. If the other team, hand-waving, wants to detect headless chromium greater than there are still ways to do that. At least I noticed a few, which I'll tackle greater than in future updates. Greater than greater than greater than it's probably impossible to prevent all ways to detect headless chromium, but greater than it should be possible to make it so difficult that it becomes cost prohibitive greater than or triggers too many false positives to be feasible. So, while Puppeteer Extra can dodge most basic bot detection like Neo and Matrix, it can't surely bypass Cloudflare. Sure, you could integrate a proxy into Puppeteer, but even that might not be enough.

Starting point is 00:06:03 https colon slash slash www.youtube.com.watch?v equals 7p n c h j p l g t w and embeddable equals true the problem isn't Puppeteer itself, because let's be real, Puppeteer rocks. Sign of the horns, but the browser it's controlling. The real solution? A powerful browser that operates in headed mode like a regular browser to reduce bot detection. Scales in the cloud for you, saving you time and costs in infrastructure management. Offers rotating IPs powered by one of the largest and most reliable proxy networks on the market. Automatically handles captcha solving, browser fingerprinting, cookie and header customization, and retries for optimal efficiency. Seamlessly integrates with leading browser automation libraries like Playwright,

Starting point is 00:06:42 Selenium, and Puppeteer. Believe it or not, this isn't some distant dream. It's real, and it's exactly what Bright Data's scraping browser has to offer. Final thoughts. Puppeteer is one of the most widely used browser automation tools in the tech world, but even superheroes have their limits. The community stepped in with a package that gives Puppeteer some seriously cool new abilities through custom plugins. But here's the thing. While these plugins can make your scraping operation way stronger, they won't magically turn you into a ghost ghost. Sites with advanced bot detection might still be able to block you, bypass all anti-bots with Bright Data's scraping browser. A non-detectable cloud browser that integrates seamlessly with Puppeteer.

Starting point is 00:07:28 Join our mission to make the web a public space for everyone, everywhere, even through automated scripts. Until next time, keep exploring the internet with freedom. Globe thank you for listening to this Hackernoon story, read by Artificial Intelligence. Visit hackernoon.com to read, write, learn and publish.

The Good Tech Companies - Elevate Your Scraping Project With Puppeteer Extra

There aren't comments yet for this episode. Click on any sentence in the transcript to leave a comment.