Haɗa da loda jerin wakili a cikin aikin Scrapy da aka tura

Fayil ɗin wakili kusa da kwafin ci gaba ba ya kasancewa kai tsaye a cikin fakitin crawler da aka tura. Akwai ayyuka biyu: haɗa bayanan a cikin rarrabawa, sannan a nemo su a cikin fakitin da aka shigar.

Ga aikin Setuptools da ke da crawl_app/resources/proxies.txt a cikin fakitin Python ɗinsa, shigarwar pyproject.toml na iya haɗa shi:

[tool.setuptools.package-data]
crawl_app = ["resources/*.txt"]

Haɗa wannan da tsarin fakitin da aikin ke da shi. Wani backend na gini yana da nasa hanyar haɗa bayanai.

Ga middleware da zai iya amfani da igiyoyin endpoint, Python 3.11+ na iya karanta albarkatun fakitin kai tsaye:

from importlib.resources import files

text = files('crawl_app').joinpath('resources', 'proxies.txt').read_text(
    encoding='utf-8'
)
proxy_urls = [line.strip() for line in text.splitlines() if line.strip()]

Yi amfani da ainihin sunan fakitin kuma tabbatar da tsarin URL da ake tsammani a cikin mai karatu. Don bayanan endpoint na jama'a da za a rarraba, yi amfani da adireshi daga jerin wakili kyauta a cikin ka'idar da tsarin URL da middleware ke tsammani. Binciken bai dogara da babban fayil ɗin aiki na yanzu na tura ba.

Saitin PROXY_LIST na ɓangare na uku na iya buƙatar hanyar tsarin fayil maimakon waɗannan igiyoyin. Lokacin amfani da importlib.resources.as_file don wannan hanyar, riƙe mahallinsa a buɗe muddin middleware yana buƙatar fayil ɗin da aka ciro. Mayar da hanyar wucin gadi bayan an rufe mahallin na iya rasa albarkatar.

Riƙe jerin samarwa masu ɗauke da takaddun shaida a cikin ajiyar sirri ko tsari da aka nufa don tura. Bayanan fakitin sun dace da bayanan endpoint da za a rarraba, ba zato cewa duk abin da aka haɗa da lambar ya kasance na sirri ba.