Yabela i-URL ne-proxy kumsebenzi ngamunye we-Python ThreadPoolExecutor

i-map yechibi inikeza umsebenzi into eyodwa ngesikhathi. Yenza leyo nto ibe umsebenzi ophelele, kufaka phakathi i-URL ne-proxy ekhethiwe, ukuze umsebenzi angacabangi indlela yakhe ohlwini lomhlaba oluguqukayo.

Ngama-urls anomkhawulo, nikeza uhlu olungashayi phansi lwama-URL e-proxy aqinisekisiwe. Abazongenela i-HTTP bomphakathi batholakala ku-uhlu lwe-proxy ye-HTTP:

from concurrent.futures import ThreadPoolExecutor
from itertools import cycle
import requests

def fetch(job):
    url, proxy_url = job
    proxies = {"http": proxy_url, "https": proxy_url}
    with requests.get(url, proxies=proxies, timeout=(5, 15)) as response:
        response.raise_for_status()
        return url, response.text

if not proxy_urls:
    raise ValueError("No proxy endpoints were supplied")

jobs = zip(urls, cycle(proxy_urls))
with ThreadPoolExecutor(max_workers=5) as executor:
    for url, body in executor.map(fetch, jobs):
        handle_result(url, body)

urls, proxy_urls ne-handle_result kunikezwa uhlelo lokusebenza. Umxhumanisi wabela amaphoyinti okugcina ngokulandelana okuphindaphindayo; umsebenzi ngamunye uthola ipheya elilodwa elinqunyiwe. Ukuphinda imiphumela kuveza futhi izingqinamba zabasebenzi.

Abasebenzi abahlanu bakhawulela imisebenzi esebenzayo, hhayi imisebenzi esebenzayo nge-proxy ngayinye. Uma uhlu lunamaphoyinti amabili okugcina, imisebenzi eminingi ingasebenzisa iphoyinti elilodwa lokugcina ngesikhathi esisodwa. Engeza umthetho womthamo ohlukile nephoyinti ngalinye lokugcina uma kudingeka. Emsebenzini oyedwa osebenzayo nge-URL yephoyinti ngalinye lokugcina, i-ukuqashwa komugqa okulungisisiwe ingabeka futhi ibuyisele ukufakwa ngakunye.

Ukuphinda kufanele kudale umzamo omusha onomkhawulo we-URL efanayo. Gwema izingcingo zokuphinda eziphindaphindayo ezisusa ukufakwa ohlwini olwabiwe: omunye umsebenzi kungenzeka ukuthi ukhetha kulo, futhi uhlu oluphelile lusadinga umphumela ocacile.