Layisha izilungiselelo ze-proxy ze-Scrapy ku-CrawlerProcess script
Dlulisela get_project_settings() ku-CrawlerProcess lapho isiqalisi esizimele kufanele sisebenzise i-downloader middleware efanayo ne-scrapy crawl.
Isicabucabu singasebenzisa i-proxy elindelekile ngaphansi kwe-scrapy crawl futhi sihambe ngqo lapho siqalwa yi-Python script. Hlola ukuthi ley script yakha kanjani izilungiselelo zayo ngaphambi kokushintsha i-proxy server.
Okwephrojekthi ye-Scrapy ekhona kanye nesicabucabu sayo esibhalisiwe:
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
settings = get_project_settings()
process = CrawlerProcess(settings=settings)
process.crawl('catalog')
process.start()Faka esikhundleni sika-catalog igama lesicabucabu somsebenzi. Umsebenzi kufanele utholakale nge-scrapy.cfg yawo, noma nge-SCRAPY_SETTINGS_MODULE esethwe kahle ekhomba imojuli yezilungiselelo engalethwa.
Ukwakha i-CrawlerProcess() kuphela akulethi ifayela lezilungiselelo lomsebenzi ngenxa nje yokuthi i-script iletha nekilasi lesicabucabu. Ngakho-ke i-middleware yangokwezifiso enikwe amandla yi-DOWNLOADER_MIDDLEWARES ingase ingabikho kulowo msebenzi.
Ukulayisha izilungiselelo kunikeza izingxenye ezilungiselelwe; akusakhi ikheli le-proxy. I-middleware ye-HTTP proxy ye-Scrapy isadinga i-proxy yayo evamile yemvelo noma inani elicacile le-request.meta['proxy']. Hlola ukuthi i-HTTPPROXY_ENABLED ayikuvimbanga. Ekuholeni kwamakhasi omphakathi okuvunyelwe, uhlu lwe-HTTP proxy lunikeza abazongenela inani le-HTTP proxy elinikezwa ngale metadata noma imvelo.
Qhathanisa izilungiselelo ezilayishiwe kanye ne-middleware esebenzayo yesiqalisi nokugijima komugqa womyalo. Uma zivumelana, qhubeka ne-metadata ye-proxy yesicelo esikhethiwe kanye nephutha langempela lokuthutha.