Ku darista iyo rarista liiska boroksee ee mashruuc Scrapy la daabacay

Faylka boroksee ee ku xiga nuqul horumarineed si otomaatig ah kama jiro xirmada crawler-ka ee la daabacay. Laba shaqo ayaa jira: ku darista baytasyada qaybinta, ka dibna ka helista xirmada la rakibay.

Mashruuc Setuptools oo leh crawl_app/resources/proxies.txt gudaha xirmada Python, gelinta pyproject.toml ayaa ku dari karta:

[tool.setuptools.package-data]
crawl_app = ["resources/*.txt"]

Ku dar tan habaynta xirmada mashruuca ee jirta. Backend dhismo kale wuxuu leeyahay interface u gaar ah oo xogta lagu daro.

Middleware oo isticmaali kara xargaha endpoint, Python 3.11+ si toos ah ayuu u akhriyi karaa kheyraadka xirmada:

from importlib.resources import files

text = files('crawl_app').joinpath('resources', 'proxies.txt').read_text(
    encoding='utf-8'
)
proxy_urls = [line.strip() for line in text.splitlines() if line.strip()]

Isticmaal magaca xirmada dhabta ah oo xaqiiji qaabka URL ee la filayo akhristaha dhexdiisa. Xogta endpoint dadweynaha ee la qaybin karo, isticmaal cinwaanno ka yimid liiska boroksee ee bilaashka ah qaabka borotokoolka iyo URL ee middleware filayo. Raadinta kuma xirna galka shaqada ee hadda ee daabacaadda.

Dejinta PROXY_LIST ee dhinac saddexaad waxay u baahan kartaa waddo nidaamka faylka halkii xargahaas. Markaad isticmaalayso importlib.resources.as_file interface-kaas, furan hayso macnaha guud inta middleware u baahan yahay faylka la soo saaray. Soo celinta waddo ku meel gaar ah ka dib marka macnaha guud la xiro waxay lumin kartaa kheyraadka.

Kaydi liisyada waxsoosaarka ee qaata aqoonsiga macluumaadka kaydka sirta ama habaynta loogu talagalay daabacaadda. Xogta xirmada waxay ku habboon tahay xogta endpoint ee la qaybin karo, ma aha malo ah in wax kasta oo koodhka lagu duubay ay sii ahaanayaan qarsoodi.