Proxies for scraping

• Protocols: IPv4 + SOCKS5, plus HTTP/HTTPS. Pick what your software needs.
• Pool size: 12,000 active IPs per day, list refreshed in real time.
• Traffic: unlimited on every plan. You pay for term and threads.
• Threads: from 1000 on base plans up to 3000 on Enterprise.
• Output formats: IP:PORT or IP:PORT:LOGIN:PASS.
• Speed: low latency, high uptime.

A captcha does not appear because the site recognised a robot, it appears because a counter tripped. Platforms track how often requests come from an address, how even the intervals are, the order of transitions and the consistency of headers. Once a figure crosses a threshold, the check switches on.

Which means the captcha follows from rhythm. The same scraper on one proxy address will hit a check after a hundred requests, while the same volume spread across a pool passes without a single one.

So when tuning collection, the count of proxy addresses is not the first question, the rate per address is. You find the safe rhythm for one IP first, then multiply by the number of proxy addresses to reach the total speed.

The calculation rests on three figures: how many pages you need, in what time, and how long one request takes including processing. Divide volume by time to get the required speed in pages per second, then multiply by request duration to get the number of concurrent threads.

A worked example. A hundred thousand pages in a day is a little over one page per second. If a request with parsing takes two seconds, three threads cover it. If the page is heavy and takes ten seconds, you need twelve.

Then you check the second boundary: whether the target sustains that rate given rotation. If the safe rhythm per address is lower than the calculation demands, you close the gap with more addresses.

Beyond the address, a platform sees the set of service fields in a request. The user-agent string reports the browser and system, separate fields describe accepted languages, encodings and content types. The order of those fields differs between programs and is a signal in itself.

Hence a typical collection mistake: the addresses rotate while the headers stay identical across every thread. To the platform that reads as many different networks all producing the same client, which stands out more than ordinary traffic.

The second layer is the parameters of the secure connection. A script on a popular library assembles them differently from a real browser, and that difference alone marks a request as automated before any behaviour is analysed.

The practical approach is simple: variety of addresses is worth backing with variety of clients. Different user-agent strings, different header order, and for heavy targets running through a real browser engine.

The robots.txt file is an agreement. It sits at the site root and lists which sections the owner asks crawlers to leave alone. Technically it forbids nothing: a request to a section closed there goes through like any other.

Technical blocking works differently. That is the server acting: a refusal code, a demand for verification, a rate limit or a cut-off based on client signals. It fires regardless of what robots.txt says.

Confusing the two costs time for a practical reason. Steering only by robots.txt, you easily run into limits on sections it leaves open. And conversely, a section closed in the file may carry no technical protection at all.

For planning collection it is more useful to watch how the platform actually behaves: at what rate checks appear, which sections switch on restrictions, how the server answers as threads grow. Those boundaries define the working mode.

1. Pick a term: 24 hours, a week or a month, and the number of threads.
2. Order the plan (IPv4 + SOCKS5 included) and pay for access.
3. Get access in your dashboard and test scraping on your own task.

Which protocol depends on your software: SOCKS5 proxies for antidetect, bots and scrapers, IPv4 proxies for maximum site compatibility.