scrapy-proxy-pool: ukuphatha izimpendulo ze-binary ekutholeni ukuvinjelwa

I-PDF noma isithombe singafika ngempumelelo nge-proxy bese sihluleka ekutholeni ukuvinjelwa. I-Response eyisisekelo ye-Scrapy iveza ama-body bytes, kodwa impahla yayo ye-text iletha iphutha elithi “Response content isn't text.”

Uma inqubomgomo ye-scrapy-proxy-pool efakiwe ifuna i-response.text ngaphandle kwemibandela, faka enye indawo yayo enolwazi lohlobo:

from scrapy.http import TextResponse
from scrapy_proxy_pool.policy import BanDetectionPolicy

class BinaryAwarePolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        if isinstance(response, TextResponse):
            if self.BANNED_PATTERN.search(response.text):
                return True
        if response.status not in self.NOT_BAN_STATUSES:
            return True
        return response.status == 200 and not response.body

Yikhethe ngesilungiselelo sesandiso sephakethe:

PROXY_POOL_BAN_POLICY = "myproject.policy.BinaryAwarePolicy"

Lokhu kugcina imithetho yezimo nomzimba ongenalutho yenqubomgomo futhi kunciphisa ukusesha kombhalo kuphela ezimpendulweni ezisekelayo. Inqubomgomo yokushiywa ezelwe nayo iyatholakala.

Buyekeza futhi leyo mitherho esele yohlelo lokusebenza: impendulo engu-200 engenalutho ngamabomu noma iphutha elijwayelekile lendawo akusona ubufakazi bobuxhwala be-proxy njalo. Shiya i-middleware isebenza futhi ushintshe isigaba esibangele ukwehluleka, esikhundleni sokuvala konke ukutholwa kokuvalwa kokulandwa kwe-binary.