Ukufaka nokulayisha uhlu lwe-proxy kuphrojekthi ye-Scrapy esetshenzisiwe

Ifayela le-proxy eduze kwekhophi yokuthuthukisa alikho ngokuzenzakalela kuphakethe le-crawler elisetshenzisiwe. Imisebenzi emibili ikhona: ukufaka amabhayithi ekusabalaliseni, bese uwathola kuphakethe elifakiwe.

Kuphrojekthi ye-Setuptools ene-crawl_app/resources/proxies.txt ngaphakathi kwephakethe layo le-Python, umugqa we-pyproject.toml ungayifaka:

[tool.setuptools.package-data]
crawl_app = ["resources/*.txt"]

Hlanganisa lokho nokucushwa kwephakethe okukhona kokuphrojekthi. Enye i-build backend inesixhumi sayo sokufaka idatha.

Kuma-middleware angasebenzisa izintambo ze-endpoint, i-Python 3.11+ ingafunda insiza yephakethe ngqo:

from importlib.resources import files

text = files('crawl_app').joinpath('resources', 'proxies.txt').read_text(
    encoding='utf-8'
)
proxy_urls = [line.strip() for line in text.splitlines() if line.strip()]

Sebenzisa igama langempela lephakethe futhi uqinisekise ifomethi ye-URL elindelekile kumfundi. Ngedatha ye-endpoint yomphakathi esabalaliswayo, sebenzisa amakheli avela oh lweproxy lwamahhala kufomethi yeprotocol ne-URL elindelwe yi-middleware. Ukusesha akuncikile kudayirekhthori yamanje yomsebenzi yokusetshenziswa.

Isilungiselelo se-PROXY_LIST senkampani yangaphandle singadinga indlela yesistimu yamafayela esikhundleni salezi zintambo. Uma usebenzisa i-importlib.resources.as_file kuleso sixhumi, gcina umongo wayo uvulekile isikhathi eside i-middleware idinga ifayela elikhishiwe. Ukubuyisela indlela yesikhashana ngemva kokuvalwa komongo kungalahlekelwa insiza.

Gcina uhlu lokukhiqiza olunezifakazelo endaweni yokugcina eyimfihlo noma yokucushwa ehlosiwe yokusetshenziswa. Idatha yephakethe ifanele idatha ye-endpoint esabalaliswayo, hhayi ukucabanga ukuthi konke okuboshwe nekhodi kuhlala kuyimfihlo.