U yeel URL iyo proxy shaqo kasta oo Python ThreadPoolExecutor ah

map-ka barkada waxa uu shaqaalaha u gudbiyaa hal shay mar kasta. Shaygaas ka dhig shaqo buuxda, oo ay ku jiraan URL-ka iyo proxy-ga la doortay, si shaqaaluhu uusan uga qiyaasin jidkiisa liis caalami ah oo isbeddelaya.

Urls xaddidan, bixi liis aan madhanayn oo URL proxy la xaqiijiyay. Musharrixiinta HTTP ee dadweynaha waxaa laga heli karaa liiska proxy HTTP:

from concurrent.futures import ThreadPoolExecutor
from itertools import cycle
import requests

def fetch(job):
    url, proxy_url = job
    proxies = {"http": proxy_url, "https": proxy_url}
    with requests.get(url, proxies=proxies, timeout=(5, 15)) as response:
        response.raise_for_status()
        return url, response.text

if not proxy_urls:
    raise ValueError("No proxy endpoints were supplied")

jobs = zip(urls, cycle(proxy_urls))
with ThreadPoolExecutor(max_workers=5) as executor:
    for url, body in executor.map(fetch, jobs):
        handle_result(url, body)

urls, proxy_urls iyo handle_result waxaa bixiya abka. Isuduwaha waxa uu u qoondeeyaa dhibcaha ugu dambeeya si isdaba joog ah; shaqaale kastaa waxa uu helaa hal lammaane oo go'an. Ku celcelinta natiijooyinka waxa kale oo ay soo bandhigtaa ka-reebitaannada shaqaalaha.

Shan shaqaale ayaa xaddida shaqooyinka firfircoon, ma aha shaqooyinka firfircoon proxy kasta. Haddii liiska uu leeyahay laba dhibcood oo ugu dambeeya, shaqooyin badan ayaa isticmaali kara isla dhibcahaas isku mar. Ku dar qaanuun awood gaar ah dhibcaha ugu dambeeya kasta marka loo baahdo. Hal shaqo firfircoon URL kasta oo dhibcaha ugu dambeeya, kireyn safka la isku waafajiyay waxa uu qabsan karaa oo soo celin karaa gal kasta.

Dib-u-celin waa in ay abuurto isku day cusub oo xaddidan isla URL-kaas. Ka fogow wicitaannada dib-u-celin ee soo noqnoqda ee ka saara galka liis la wadaago: shaqaale kale ayaa laga yaabaa inuu ka dooranayo, liisna dhammaaday weli waxa uu u baahan yahay natiijo cad.