Scrapy AutoThrottle nge-proxy: izikhala nokubambezeleka

Ngokuzenzakalelayo, i-Scrapy yabela izikhala zokulanda ngokusekelwe esizindeni se-URL yesicelo. Ukunikeza isicelo i-HTTP proxy akushintshi leyo URL ibe igama lomsingathi we-proxy. Ngakho-ke izicelo eziya kumasayithi ahlukene zingagcina izikhala ezihlukene ngenkathi zabelana umlamuli oyedwa.

I-AutoThrottle ilungisa ukubambezeleka kwesikhala ngasinye ngokusebenzisa isikhathi sokulanda esibonakele. Umkhawulo ojwayelekile wokwenza kanyekanye ngesizinda ngasinye kanye nokubambezeleka okuncane kokulanda kusasebenza.

Hlola ukuhlukaniswa ngamaqembu ngaphambi kokushintsha i-algorithm

I-middleware yesiko ingabela request.meta["download_slot"], yeqe ukukhethwa kwesikhala okuvamile. Ezinye izandiso ze-proxy zisebenzisa ukuhlukaniswa kwazo ngamaqembu. Hlola lezo zilungiselelo uma amasayithi angahlobene ebonakala abelana isilawuli esisodwa.

Ngokuphambene, ukwabela isikhala esihlukile ku-proxy ngayinye kungavumela ama-proxy amaningi ukuthi axhumane nethagethi efanayo ngesikhathi esisodwa. Lokho kungase kubhidlize umkhawulo wesayithi lonke obekumele ugcinwe isiseshi. Khetha ukhiye wesikhala ngokuvumelana nomsiza ovikelwayo.

Ukubambezeleka kufaka umkhondo oyisebenzisile

I-Scrapy ikala ukubambezeleka kusukela ekuthumeleni isicelo kuze kufike izihloko ze-HTTP. Isicelo esihanjiswa nge-proxy singafaka kulokho kubonwa ukubambezeleka komlamuli emgqeni nasekudluliseleni. I-AutoThrottle ayitholi isikhathi sokucubungula esihlukile somthombo kuphela.

Ukukhulisa ukwenza kanyekanye akukwenzi lokho kukala kuhlukanise isayithi. Gcina inhlonipho ethagethini kanye nomthamo we-proxy njengemikhawulo ehlukene lapho kokubili kubalulekile. I-proxy eyabiwe ingadinga isabelomali esengeziwe sokwenza kanyekanye kuzo zonke izikhala zethagethi, kuyilapho ithagethi ngayinye igcina inqubomgomo yayo yejubane.