Settings¶
This Scrapy downloader middleware adds some settings to configure how to work with your Zyte proxy service.
ZYTE_SMARTPROXY_APIKEY¶
Default: None
Default API key for your Zyte proxy service.
Note that Zyte API and Zyte Smart Proxy Manager have different API keys.
ZYTE_SMARTPROXY_URL¶
Default: 'http://proxy.zyte.com:8011'
Default endpoint for your Zyte proxy service.
For guidelines on setting a value, see the initial configuration instructions.
ZYTE_SMARTPROXY_MAXBANS¶
Default: 400
Number of consecutive bans necessary to stop the spider.
ZYTE_SMARTPROXY_DOWNLOAD_TIMEOUT¶
Default: 190
Timeout for processing proxied requests. It overrides Scrapy’s DOWNLOAD_TIMEOUT
.
ZYTE_SMARTPROXY_PRESERVE_DELAY¶
Default: False
If False
sets Scrapy’s DOWNLOAD_DELAY
to 0
, making the spider to crawl faster. If set to True
, it will
respect the provided DOWNLOAD_DELAY
from Scrapy.
ZYTE_SMARTPROXY_DEFAULT_HEADERS¶
Default: {}
Default headers added only to proxied requests. Headers defined on DEFAULT_REQUEST_HEADERS
will take precedence as long as the ZyteSmartProxyMiddleware
is placed after the DefaultHeadersMiddleware
. Headers set on the requests have precedence over the two settings.
This is the default behavior,
DefaultHeadersMiddleware
default priority is400
and we recommendZyteSmartProxyMiddleware
priority to be610
.
ZYTE_SMARTPROXY_BACKOFF_STEP¶
Default: 15
Step size used for calculating exponential backoff according to the formula: random.uniform(0, min(max, step * 2 ** attempt))
.
ZYTE_SMARTPROXY_BACKOFF_MAX¶
Default: 180
Max value for exponential backoff as showed in the formula above.
ZYTE_SMARTPROXY_FORCE_ENABLE_ON_HTTP_CODES¶
Default: []
List of HTTP response status codes that warrant enabling your Zyte proxy service for the corresponding domain.
When a response with one of these HTTP status codes is received after an unproxied request, the request is retried with your Zyte proxy service, and any new request to the same domain is also proxied.