How to Integrate Proxies Into Scrapy Spiders (2026)

Scrapy is one of the most efficient Python frameworks for large-scale web scraping, but sending all your requests from a single IP will get you blocked fast. Adding proxies to your Scrapy spiders routes each request through a different IP, keeping your scraper running without interruptions. Whether you are scraping e-commerce sites, search engines, or any target with rate limiting, proxy integration is the step that makes it production-ready.

In this article, we'll explore how to set up and configure proxies inside your Scrapy spiders the right way.

How to Add a Proxy to a Scrapy Spider

The simplest way to add a proxy is through the HTTPPROXY_ENABLED setting in settings.py:

1HTTPPROXY_ENABLED = True
2
3DOWNLOADER_MIDDLEWARES = {
4    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
5}
6
7HTTP_PROXY = 'http://username:password@proxy.proxyon.io:port'

This routes all requests through your proxy. For per-request control, pass the proxy through the request meta inside your spider:

1def start_requests(self):
2    yield scrapy.Request(
3        url='https://example.com',
4        meta={'proxy': 'http://username:password@proxy.proxyon.io:port'}
5    )

Use this when you only need the proxy on specific requests rather than the entire spider.

Also Read: Datacenter Proxies for Web Scraping

Using Rotating Proxies with Scrapy Middleware

Rotating proxies in Scrapy are best handled through a custom downloader middleware, which assigns a different IP to each request without touching your spider logic:

1import random
2
3class RotatingProxyMiddleware:
4    def __init__(self):
5        self.proxies = [
6            'http://username:password@proxy.proxyon.io:port1',
7            'http://username:password@proxy.proxyon.io:port2',
8            'http://username:password@proxy.proxyon.io:port3',
9        ]
10
11    def process_request(self, request, spider):
12        request.meta['proxy'] = random.choice(self.proxies)

Register it in settings.py:

1DOWNLOADER_MIDDLEWARES = {
2    'myproject.middlewares.RotatingProxyMiddleware': 100,
3}

With residential proxies, you skip the proxy list entirely. Rotation is handled on the backend through a single endpoint, so your middleware stays clean, and every request goes out through a different IP automatically.

Choosing the Right Proxy Type for Scrapy

The proxy type you use depends entirely on what you are scraping. Residential proxies are the safest choice for heavily protected targets like e-commerce sites and search engines. They use IPs assigned to real households, making them nearly impossible to detect as automated traffic. Residential proxies start at $1.75/GB with no subscription required.

Datacenter proxies are faster and cheaper, and they work fine for targets without aggressive bot detection.

Mobile proxies are the hardest to block but also the most expensive. They are rarely necessary unless your target is specifically filtering out both residential and datacenter IPs.

A good rule of thumb: start with datacenter proxies and switch to residential if you are hitting blocks. Mobile proxies should be your last resort.

Also Read: How to Scrape a JavaScript Website With Python

Final Thought

Integrating proxies into Scrapy comes down to three things: the right setup, the right rotation method, and the right proxy type for your target. Proxyon's residential proxies start at $1.75/GB with no subscription. Get your endpoint at Proxyon and start scraping.