Sanya URL da proxy ga kowane aikin Python ThreadPoolExecutor

map na tafki yana ba ma'aikaci abu ɗaya a lokaci ɗaya. Mai da wannan abu cikakken aiki, tare da URL da zaɓaɓɓen proxy, don ma'aikaci kada ya gano hanyarsa daga jerin duniya mai canzawa.

Don iyakataccen urls, ba da jerin URL na proxy ingantattu marar wofi. Ana samun 'yan takarar HTTP na jama'a a cikin jerin proxy na HTTP:

from concurrent.futures import ThreadPoolExecutor
from itertools import cycle
import requests

def fetch(job):
    url, proxy_url = job
    proxies = {"http": proxy_url, "https": proxy_url}
    with requests.get(url, proxies=proxies, timeout=(5, 15)) as response:
        response.raise_for_status()
        return url, response.text

if not proxy_urls:
    raise ValueError("No proxy endpoints were supplied")

jobs = zip(urls, cycle(proxy_urls))
with ThreadPoolExecutor(max_workers=5) as executor:
    for url, body in executor.map(fetch, jobs):
        handle_result(url, body)

urls, proxy_urls da handle_result aikace-aikacen ne ke bayarwa. Mai daidaitawa yana sanya wuraren ƙarshe a cikin tsari mai maimaitawa; kowane ma'aikaci yana samun biyu tabbatacce. Maimaita sakamakon yana kuma bayyana keɓancewar ma'aikata.

Ma'aikata biyar suna iyakance ayyuka masu aiki, ba ayyuka masu aiki ga kowane proxy ba. Idan jerin yana da wuraren ƙarshe biyu, ayyuka da yawa na iya amfani da wurin ƙarshe ɗaya a lokaci guda. Ƙara wata ka'idar iya aiki daban ga kowane wurin ƙarshe idan an buƙata. Don aiki ɗaya mai aiki ga kowane URL na wurin ƙarshe, hayar layi mai daidaitawa na iya ajiye da dawo da kowace shiga.

Sake gwadawa ya kamata ya ƙirƙiri sabon gwaji mai iyaka don wannan URL. Guji kiran sake gwadawa mai maimaitawa wanda ke cire shiga daga jerin gama gari: wani ma'aikaci na iya zaɓe daga gare shi, kuma jerin da ya ƙare har yanzu yana buƙatar sakamako bayyananne.