Why Your Web Scraper Gets Blocked (And How to Fix It)

[00:00] Why does your web scraper keep getting [00:01] blocked? In order to understand that, [00:03] let's look at two scrapers. The one on [00:05] the left is getting blocked after about [00:07] 10 requests. The one on the right does [00:09] not get blocked. Why? Well, the [00:11] difference is just three lines of [00:13] config. Let's get into it. In this [00:15] video, I'm going to build a small [00:16] Node.js project that collects these same [00:18] products' prices across three regional [00:21] storefronts: UK, Germany, and US. We are [00:24] extracting the price in this script. So, [00:26] then if I run it, the results look like [00:27] this. So, we are getting the prices [00:30] getting the prices, then uh-oh, we are [00:32] blocked. We have hit the rate limit. [00:33] Now, let's add this config. We're going [00:35] to use a tool called a residential [00:37] proxy. Sites such as Data Impulse offer [00:40] this service. We're going to grab our [00:42] Data Impulse credentials of the login, [00:44] password, host, and port to fill this [00:47] out. And then we need to add the country [00:50] suffixes. This will help us build a [00:52] proxy agent that exits from a specific [00:54] country. It works by reading the country [00:57] suffix attached to the login to pick the [00:59] exit region, and then rotates the [01:01] gateway to hand out a fresh IP per [01:03] request. So, in essence, it's the same [01:05] workflow as before, just routed through [01:07] a Data Impulse residential proxy. And [01:10] let's run it. And success, we get all [01:12] the prices, no rate limits hit this [01:14] time. And that's it. Starting at $1 per [01:18] gig, the next time your web scraper gets [01:20] blocked, try using a residential proxy. [01:23] That might be the answer.