Phinda uzame i-GET ye-Scrapy ngokusebenzisa enye i-proxy

Sebenzisa umsizi wokuphinda uzame we-Scrapy ukuhlela isicelo esisha nge-proxy entsha ngenkathi uhlonipha ibhajethi yokuphinda uzame.

Ukushintsha i-response.request.meta kushintsha into ye-Python. Ukuze ulande okunye, kufanele unikeze i-Scrapy enye i-Request ezohlelwa.

Ku-GET i-callback yakho enquma ngokusobala ukuyiphinda, umsizi wokuphinda uzame womphakathi we-Scrapy angagcina isibalo sokuphinda uzame nesicelo esisha:

from scrapy.downloadermiddlewares.retry import get_retry_request

def retry_through(self, response, proxy_url):
    retry = get_retry_request(
        response.request,
        spider=self,
        reason='choose_another_proxy',
    )
    if retry is None:
        return None
    retry.meta['proxy'] = proxy_url
    retry.headers.pop('Proxy-Authorization', None)
    return retry

Lona umsizi we-spider. Obizayo kufanele abuyisele noma enze i-yield ye-Request ephumayo uma ikhona. I-None isho ukuthi umsizi wenqabile okunye ukuphinda uzame ngaphansi komkhawulo osebenzayo.

Umsizi wokuphinda uzame ukhophisha isicelo, ubuyekeza inani lokuphinda uzame futhi unikeza ukuziphatha kwesihleli okufanele ukuphinda uzame. Ukwenza i-yield nje kwe-URL efanayo njengesicelo esisha esivamile kungase kuhlangane nokuhlunga okuphindwe kabili esikhundleni salokho.

Sebenzisa i-URL ye-proxy esekelwa i-HttpProxyMiddleware enikisiwe. Ukuphinda uzame izicelo ze-GET zomphakathi ezivunyelwe, ikhandidethi ye-HTTP engaqinisekisiwe kusukela kuhlu lwe-proxy ye-HTTP inganikeza i-proxy_url. Ukusula inkambu ye-proxy-authorization enamathiselwe ngaphambili kugwema ukuthwala inhloko yesixazululi esidala kulokhu kukhetha; vumela i-middleware ukuthi ithole i-akhawunti entsha kusukela ku-URL entsha ye-proxy uma kufanele.

Gcina lokhu kukhetha kwezinga le-callback kuhlukile ekuphatheni okuzenzakalelayo kwe-RetryMiddleware kokwehluleka kwayo okulungisiwe. Nquma ukuthi yiluphi ungqimba oluphethe ukuphinda uzame ukuze umsebenzi ungasebenzisi izabelomali ezimbili ezizimele. Isibonelo sikhuluma ngomsebenzi we-GET ongadlalwa futhi, hhayi isithembiso sokuthi ukuphinda umsebenzi wokuhweba othunyelwe ngokungafanele kuphephile.

Izihloko ezihlobene