scrapy-proxy-pool: sarrafa martanin binary a cikin gano haramci

PDF ko hoto na iya shigowa ta hanyar wakili cikin nasara sannan ya kasa a cikin gano haramci. Response na asali na Scrapy yana nuna baytocin jiki, amma nannaukar text nasa tana haifar da kuskure “Response content isn't text.”

Idan manufar scrapy-proxy-pool da aka shigar tana bincika response.text ba tare da sharadi ba, maye gurbin wannan manufar da sigar da ta san nau'in:

from scrapy.http import TextResponse
from scrapy_proxy_pool.policy import BanDetectionPolicy

class BinaryAwarePolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        if isinstance(response, TextResponse):
            if self.BANNED_PATTERN.search(response.text):
                return True
        if response.status not in self.NOT_BAN_STATUSES:
            return True
        return response.status == 200 and not response.body

Zabi shi ta hanyar saitin kari na fakitin:

PROXY_POOL_BAN_POLICY = "myproject.policy.BinaryAwarePolicy"

Wannan yana rike da ka'idojin matsayi da jiki mara komai na manufar yayin da yake takaita binciken rubutu ga martanin da ke goyon bayansa. Manufar keɓewa da aka gado tana nan a samu.

Kuma duba waɗannan sauran ka'idoji don aikace-aikacen: martanin 200 da aka yi da gangan babu komai ko kuskuren manufa na yau da kullum ba koyaushe shaida ce ta mummunan wakili ba. Bar middleware yana aiki kuma canza rarrabuwar da ta haifar da gazawar, maimakon kashe duk gano haramci don saukar da binary.