Ukuhlanganyela kwesikhathi kwe-proxy ye-Scrapy kumadomain amaningi
Sebenzisa umkhawulo womhlaba wonke wokuhlanganyela kwesikhathi komlayishi uma amadomain amaningi ahlosiwe esebenzisa isabelo esifanayo se-proxy, bese ubala namanye ama-crawler ahlanganyela leyo akhawunti.
Uma indawo ngayinye isebenzisa i-akhawunti eyodwa ye-proxy, isabelo sezicelo zesikhathi esisodwa sale akhawunti singasetshenziswa kuzo zonke lezo zindawo. Umkhawulo wedomain eyodwa kuphela awubonisi leso sabiwomali esabiwe.
Nge-akhawunti ecatshangelwayo evumela okungenani izicelo ezingama-20 zesikhathi esisodwa, shiya isikhala esincane bese ubeka umkhawulo womlayishi womhlaba wonke we-crawler:
CONCURRENT_REQUESTS = 16Lokhu kuwukwabiwa okuyisibonelo, hhayi inani elinconywayo kuwo wonke umhlinzeki. I-Scrapy ichaza i-CONCURRENT_REQUESTS njengenani eliphezulu lezicelo zesikhathi esisodwa zomlayishi. Izilawuli zedomain ngayinye zinganciphisa ukuhlanganyela kwesikhathi ngaphezulu; azithathi indawo yophahla lomhlaba wonke.
Faka namanye ama-crawler ezibalweni. Ama-crawler amabili azimele, ngalinye livunyelwe izicelo ezingama-16, angadlula ndawonye isabelo esabiwe esingama-20. Yaba isabiwo ngamabomu noma usebenzise umkhawulo ohlanganyelwe lapho lawo maklayenti ehlangana khona.
Nquma nokuthi umhlinzeki ubala ini. Izicelo ze-HTTP, izixhumanisi ze-TCP ezivuliwe, amathumbu asebenzayo kanye nezicelo ngomzuzwana yizilinganiso ezihlukene. Izixhumanisi eziqhubekayo kanye nokuphindaphinda kungase kwenza izibalo zazo zihluke. Ngakho-ke isethingi yokuhlanganyela kwesikhathi sezicelo ayisona isilawuli esiphelele sezixhumanisi noma sejubane.
Gcina ijubane lesayithi elihloswe njengomkhawulo ohlukile. Ukuhlanganyela kwesikhathi okuphansi noma ukubambezeleka kokulayisha kungase kusafanele indawo ethile ngisho noma i-akhawunti ye-proxy inomthamo ongasetshenziswanga.