Dib u dayo GET-ga Scrapy iyada oo loo marayo proxy kale

Isticmaal caawiyaha dib u dayashada Scrapy si aad u qorsheysato codsi cusub oo proxy cusub leh, adigoo ixtiraamaya miisaaniyadda dib u dayashada.

Beddelka response.request.meta wuxuu beddelaa shay Python. Si aad u samaysato soo dejinta kale, waa inaad Scrapy siiso Request kale oo la qorsheeyo.

GET-ga uu callback-kaagu si cad u go'aansado inuu dib u dayo, caawiyaha dib u dayashada ee dadweynaha ee Scrapy wuxuu hayn karaa xisaabta dib u dayashada codsiga cusub:

from scrapy.downloadermiddlewares.retry import get_retry_request

def retry_through(self, response, proxy_url):
    retry = get_retry_request(
        response.request,
        spider=self,
        reason='choose_another_proxy',
    )
    if retry is None:
        return None
    retry.meta['proxy'] = proxy_url
    retry.headers.pop('Proxy-Authorization', None)
    return retry

Kani waa caawiye spider. Wacaha waa inuu soo celiyaa ama yield sameeyaa Request-ka soo baxa marka mid la heli karo. None waxay ka dhigan tahay in caawiyuhu diiday dib u dayasho kale oo ku jirta xadka lagu dabaqayo.

Caawiyaha dib u dayashada wuxuu koobiyaa codsiga, cusboonaysiiyaa tirada dib u dayashada wuxuuna bixiyaa dhaqanka jadwalka ku habboon dib u dayasho. In kaliya la yield sameeyo isla URL oo ah codsi cusub caadi ah ayaa laga yaabaa inuu la kulmo shaandhaynta nuqulka.

Isticmaal URL proxy ay taageerto HttpProxyMiddleware firfircoon. Dib u dayashada codsiyada GET ee dadweynaha ee la oggolaaday, musharrax HTTP oo aan la xaqiijin laga heli karo liiska proxy-ga HTTP ayaa bixin kara proxy_url. Nadiifinta goob proxy-authorization oo hore loogu lifaaqay waxay ka hortagtaa in madaxa dhexdhexaadiye hore u galo xulashadan; u ogolow middleware-ka inuu ka soo saaro akoonka cusub URL proxy-ga cusub marka ay khusayso.

Kala saar xulashadan heerka callback-ka iyo maaraynta otomaatiga ah ee RetryMiddleware ee guul-darrooyinkiisa la habeeyay. Go'aanso lakabka leh dib u dayashada si hawlgalku uusan u isticmaalin laba miisaaniyadood oo madax banaan. Tusaaluhu wuxuu ku saabsan yahay shaqo GET oo dib loo ciyaari karo, ma aha ballanqaad in dib u noqoshada macaamil si aan kala sooc lahayn loo diray ay ammaan tahay.

Maqaallo la xiriira