Ama-proxy esicelo ngasinye nge-HTTP proxy middleware ye-Scrapy

Sebenzisa imininingwane yesicelo ye-Scrapy kanye ne-proxy middleware yayo ngaphambi kokukopisha i-authentication middleware endala ku-crawler.

I-HTTP proxy middleware ye-Scrapy ifunda i-proxy kusuka kumininingwane yesicelo:

yield scrapy.Request(
    'https://example.com/',
    callback=self.parse,
    meta={'proxy': 'http://203.0.113.10:8080'},
)

Shintsha iphoyinti lokugcina kudokhumenti futhi ushiye i-middleware ivuliwe. Uhlu lwama-proxy e-HTTP lunikeza abazongenela umncintiswano abasemphakathini kule URL yesicelo. Lokhu kukhetha umzila wesicelo; iphuli lama-proxy idinga umqondo wokukhetha ohlukile.

Ukuqinisekisa kulandela iphoyinti lokugcina elikhethiwe

I-middleware eyakhelwe ngaphakathi iphatha izinkomba ezinikezwa nge-URL ye-proxy esekelwayo kanye nekhanda lokugunyaza elihambisanayo. Escape izinhlamvu ze-URL ezigciniwe futhi ugweme ukuloga leyo URL.

I-middleware ehlukile akufanele igcine ikhanda le-Basic le-proxy yangaphambilini ngemva kokushintsha kwephoyinti lokugcina. Vumela isendlalelo esisodwa silawule umzila kanye nezikhonzi zaso ndawonye.

Ukulandelana kokukhetha kanye nokumisiwe

Uma i-middleware yokulanda yesiko ikhetha i-proxy, yisebenzise ngaphambi kokuba i-proxy middleware eyakhelwe ngaphakathi icubungule isicelo. Hlola ukulandelana kanye ne-API yenguqulo efakiwe; okungenisiwe scrapy.contrib okuphelelwe yisikhathi kungokwenguqulo ezindala.

Indawo yokusebenza yohlelo ingase futhi inikeze okumisiwe okuqondene nesikimu. Esicelweni esiqondile ngamabomu, sebenzisa indlela ekhethekile ebhaliwe yemininingwane ye-proxy: ukungabi khona kokhiye akusho ngaso sonke isikhathi ukufinyelela okuqondile.

Hlola umphumela wangempela. I-407 iphathelene nokuqinisekisa i-proxy, kuyilapho umkhawulo wesivinini noma ukwenqatshwa kokufinyelela kwewebhusayithi kungafika ngemva kokuba umzila usebenze kahle.

Izihloko ezihlobene