Sake gwada GET na Scrapy ta wani proxy

Yi amfani da mataimakin sake gwadawa na Scrapy don tsara sabon buƙatu da sabon proxy yayin mutunta kasafin sake gwadawa.

Canza response.request.meta yana canza wani abu na Python. Don yin wani saukewa, dole ne a ba Scrapy wata Request don tsarawa.

Ga GET da callback ɗinka ya yanke shawara a fili ya sake gwadawa, mataimakin sake gwadawa na jama'a na Scrapy zai iya riƙe lissafin sake gwadawa tare da sabon buƙatar:

from scrapy.downloadermiddlewares.retry import get_retry_request

def retry_through(self, response, proxy_url):
    retry = get_retry_request(
        response.request,
        spider=self,
        reason='choose_another_proxy',
    )
    if retry is None:
        return None
    retry.meta['proxy'] = proxy_url
    retry.headers.pop('Proxy-Authorization', None)
    return retry

Wannan mataimaki ne na gizo-gizo. Mai kira dole ne ya mayar ko ya yield da Request da aka samu idan akwai. None yana nufin mataimakin ya ƙi wata sake gwadawa a ƙarƙashin iyakar da ta dace.

Mataimakin sake gwadawa yana kwafin buƙatar, yana sabunta lissafin sake gwadawa kuma yana ba da halin mai tsarawa da ya dace da sake gwadawa. Yin yield da URL ɗaya a matsayin sabuwar buƙata ta yau da kullum zai iya fuskantar tacewar kwafi a maimako.

Yi amfani da URL ɗin proxy da HttpProxyMiddleware da ke kunne ke goyan bayansa. Don sake gwada buƙatun GET na jama'a da aka yarda, ɗan takarar HTTP mara tabbaci daga jerin proxy na HTTP zai iya samar da proxy_url. Share filin proxy-authorization da aka haɗa a baya yana hana ɗaukar kanun matsakaicin tsoho cikin wannan zaɓi; bar middleware ya samo sabon asusu daga sabon URL ɗin proxy idan ya dace.

Ka ware wannan zaɓi na matakin callback daga yadda RetryMiddleware ke sarrafa gazawarsa da aka saita kai tsaye. Ka yanke shawara wace ƙasa ce ta mallaki sake gwadawa domin aikin kada ya cinye kasafi biyu masu zaman kansu. Misalin ya shafi aikin GET da za a iya maimaita, ba alkawarin cewa maimaita duk wata ciniki da aka gabatar yana da aminci ba.

Labarai masu alaƙa