Settings¶
This Scrapy downloader middleware adds some settings to configure how to work with Zyte Smart Proxy Manager.
ZYTE_SMARTPROXY_APIKEY¶
Default: None
Unique Zyte Smart Proxy Manager API key provided for authentication.
ZYTE_SMARTPROXY_URL¶
Default: 'http://proxy.zyte.com:8011'
Zyte Smart Proxy Manager instance URL, it varies depending on adquiring a private or dedicated instance. If Zyte Smart Proxy Manager didn’t provide you with a private instance URL, you don’t need to specify it.
ZYTE_SMARTPROXY_MAXBANS¶
Default: 400
Number of consecutive bans from Zyte Smart Proxy Manager necessary to stop the spider.
ZYTE_SMARTPROXY_DOWNLOAD_TIMEOUT¶
Default: 190
Timeout for processing Zyte Smart Proxy Manager requests. It overrides Scrapy’s DOWNLOAD_TIMEOUT
.
ZYTE_SMARTPROXY_PRESERVE_DELAY¶
Default: False
If False
Sets Scrapy’s DOWNLOAD_DELAY
to 0
, making the spider to crawl faster. If set to True
, it will
respect the provided DOWNLOAD_DELAY
from Scrapy.
ZYTE_SMARTPROXY_DEFAULT_HEADERS¶
Default: {}
Default headers added only to Zyte Smart Proxy Manager requests. Headers defined on DEFAULT_REQUEST_HEADERS
will take precedence as long as the ZyteSmartProxyMiddleware
is placed after the DefaultHeadersMiddleware
. Headers set on the requests have precedence over the two settings.
This is the default behavior,
DefaultHeadersMiddleware
default priority is400
and we recommendZyteSmartProxyMiddleware
priority to be610
ZYTE_SMARTPROXY_BACKOFF_STEP¶
Default: 15
Step size used for calculating exponential backoff according to the formula: random.uniform(0, min(max, step * 2 ** attempt))
.
ZYTE_SMARTPROXY_BACKOFF_MAX¶
Default: 180
Max value for exponential backoff as showed in the formula above.
ZYTE_SMARTPROXY_FORCE_ENABLE_ON_HTTP_CODES¶
Default: []
List of HTTP response status codes that warrant enabling Zyte Smart Proxy Manager for the corresponding domain.
When a response with one of these HTTP status codes is received after a request that did not go through Zyte Smart Proxy Manager, the request is retried with Zyte Smart Proxy Manager, and any new request to the same domain is also sent through Zyte Smart Proxy Manager.