Yan URL kan ati proxy fun iṣẹ Python ThreadPoolExecutor kọọkan

map ti adagun kan nfi ohun kan ranṣẹ si oṣiṣẹ ni ẹẹkan. Sọ ohun yẹn di iṣẹ pipe, pẹlu URL ati proxy ti a yan, ki oṣiṣẹ ma ba ṣe iṣiro ọna rẹ lati inu atokọ agbaye ti n yipada.

Fun urls ti o ni opin, pese atokọ URL proxy ti a ti fọwọsi ti ko ṣofo. Awọn oludije HTTP ti gbogbo eniyan wa ninu atokọ proxy HTTP:

from concurrent.futures import ThreadPoolExecutor
from itertools import cycle
import requests

def fetch(job):
    url, proxy_url = job
    proxies = {"http": proxy_url, "https": proxy_url}
    with requests.get(url, proxies=proxies, timeout=(5, 15)) as response:
        response.raise_for_status()
        return url, response.text

if not proxy_urls:
    raise ValueError("No proxy endpoints were supplied")

jobs = zip(urls, cycle(proxy_urls))
with ThreadPoolExecutor(max_workers=5) as executor:
    for url, body in executor.map(fetch, jobs):
        handle_result(url, body)

urls, proxy_urls ati handle_result ni ohun elo n pese. Alakoso n yan awọn aaye ipari ni ilana atunwi; oṣiṣẹ kọọkan gba bata ti o wa titi kan. Yiyi awọn abajade kọja tun fi awọn imukuro oṣiṣẹ han.

Awọn oṣiṣẹ marun n dinku awọn iṣẹ ti n ṣiṣẹ, kii ṣe awọn iṣẹ ti n ṣiṣẹ fun proxy kọọkan. Ti atokọ ba ni awọn aaye ipari meji, ọpọlọpọ awọn iṣẹ le lo aaye ipari kanna ni akoko kanna. Fi ofin agbara lọtọ kun fun aaye ipari kọọkan nigbati o ba yẹ. Fun iṣẹ kan ti n ṣiṣẹ fun URL aaye ipari kọọkan, iyalo isinyi ti a muṣiṣẹpọ le ṣe ipamọ ati da pada gbogbo titẹsi.

Atunṣe yẹ ki o ṣẹda igbiyanju tuntun ti o ni opin fun URL kanna. Yẹra fun awọn ipe atunṣe ti n tẹsiwaju ti o yọ awọn titẹsi kuro ninu atokọ pinpin: oṣiṣẹ miiran le ma n yan lati inu rẹ, ati pe atokọ ti o ti pari tun nilo abajade ti o han gbangba.