scrapy-proxy-pool: Handle Binary Responses in Ban Detection

Restrict a scrapy-proxy-pool text-pattern check to TextResponse while retaining status and empty-body decisions.

A PDF or image can arrive through a proxy successfully and then fail inside ban detection. Scrapy’s base Response exposes body bytes, but its text property raises “Response content isn’t text.”

If the installed scrapy-proxy-pool policy searches response.text unconditionally, replace that policy with a type-aware version:

from scrapy.http import TextResponse
from scrapy_proxy_pool.policy import BanDetectionPolicy

class BinaryAwarePolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        if isinstance(response, TextResponse):
            if self.BANNED_PATTERN.search(response.text):
                return True
        if response.status not in self.NOT_BAN_STATUSES:
            return True
        return response.status == 200 and not response.body

Select it through the package’s extension setting:

PROXY_POOL_BAN_POLICY = "myproject.policy.BinaryAwarePolicy"

This retains the policy’s status and empty-body rules while restricting the text search to responses supporting it. The inherited exception policy remains available.

Review those remaining rules for the application too: an intentionally empty 200 response or an ordinary destination error is not universally evidence of a bad proxy. Keep the middleware active and change the classification that caused the failure, rather than disabling all ban detection for binary downloads.

Artiglyn mynjee