Load Scrapy proxy settings cikin CrawlerProcess script

Gizo-gizo na iya amfani da proxy ɗin da ake tsammani a ƙarƙashin scrapy crawl kuma ya tafi kai tsaye lokacin da script na Python ya ƙaddamar da shi. Duba yadda wannan script ɗin ke gina tsarinsa kafin ka canza uwar garken proxy.

Don wani aikin Scrapy da yake da shi da gizo-gizon da aka yi rajista:

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

settings = get_project_settings()
process = CrawlerProcess(settings=settings)
process.crawl('catalog')
process.start()

Maye gurbin catalog da sunan gizo-gizon aikin. Dole ne a iya gano aikin ta scrapy.cfg ɗinsa, ko ta SCRAPY_SETTINGS_MODULE da aka saita daidai wanda ke nuni ga module na tsari da za a iya shigo da shi.

Gina CrawlerProcess() kawai ba ya shigo da fayil ɗin tsarin aikin don kawai script ɗin ya shigo da aji na gizo-gizo. Saboda haka middleware na al'ada da DOWNLOADER_MIDDLEWARES ya kunna na iya rasa a wannan gudun.

Loda tsarin yana ba da abubuwan da aka saita; ba ya ƙirƙira adireshin proxy. Middleware na proxy na HTTP na Scrapy har yanzu yana buƙatar proxy ɗin muhalli na yau da kullum ko ƙimar request.meta['proxy'] bayyananna. Duba cewa HTTPPROXY_ENABLED bai kashe shi ba. Don binciken shafukan jama'a da aka yarda, jerin proxy na HTTP yana ba da 'yan takara don ƙimar proxy na HTTP da ake bayarwa ta waɗannan metadata ko muhalli.

Kwatanta tsarin da aka loda da middleware mai aiki na mai ƙaddamarwa da gudun layin umarni. Idan sun dace, ci gaba da metadata na proxy na buƙatar da aka zaɓa da ainihin kuskuren jigilar.