scrapy-proxy-pool: maaraynta jawaabaha binary ee ogaanshaha xannibaadda

Ku xaddid hubinta qaabka qoraalka ee scrapy-proxy-pool TextResponse, adigoo ilaalinaya go'aannada xaaladda iyo jirka madhan.

PDF ama sawir ayaa si guul leh u soo mara proxy kadibna ku guuldareysta ogaanshaha xannibaadda. Response-ka aasaasiga ah ee Scrapy wuxuu muujiyaa body bytes, laakiin text-kiisu wuxuu keenayaa qalad “Response content isn't text.”

Haddii siyaasadda scrapy-proxy-pool ee rakiban ay si shuruud la'aan ah u raadiso response.text, ku beddel siyaasaddaas nooc-ogaal ah:

from scrapy.http import TextResponse
from scrapy_proxy_pool.policy import BanDetectionPolicy

class BinaryAwarePolicy(BanDetectionPolicy):
    def response_is_ban(self, request, response):
        if isinstance(response, TextResponse):
            if self.BANNED_PATTERN.search(response.text):
                return True
        if response.status not in self.NOT_BAN_STATUSES:
            return True
        return response.status == 200 and not response.body

Ku dooro iyada oo loo marayo dejinta fidinta xirmada:

PROXY_POOL_BAN_POLICY = "myproject.policy.BinaryAwarePolicy"

Tani waxay ilaalinaysaa sharciyada xaaladda iyo jirka madhan ee siyaasadda waxayna ku xaddidaa baaritaanka qoraalka jawaabaha taageera. Siyaasadda ka-reebista la dhaxlay ayaa sidoo kale diyaar ah.

Sidoo kale dib u eeg sharciyada hadhay ee abka: jawaab 200 oo si ula kac ah madhan ama qalad goob caadi ah ma aha had iyo jeer caddayn proxy xun. Ka tag middleware-ka mid firfircoon oo beddel kala-soocidda sababtay guuldarada, halkii aad ka damin lahayd dhammaan ogaanshaha xannibaadda ee soo dejinta binary.

Maqaallo la xiriira