Tattara kowane sakamako daga buƙatun wakili na urllib3 a lokaci ɗaya

Ƙirƙirar tafkin ma'aikata ba ya ba mai kira shafukan da aka saukar. Riƙe kuma cinye ƙimar da aikin da aka gabatar ya mayar:

from concurrent.futures import ThreadPoolExecutor
import urllib3

manager = urllib3.ProxyManager(
    'http://proxy.example:8080',
    maxsize=4,
    block=True,
)

def fetch(url):
    response = manager.request(
        'GET', url, timeout=urllib3.Timeout(connect=5, read=15),
    )
    return url, response.status, response.data

try:
    with ThreadPoolExecutor(max_workers=4) as workers:
        results = list(workers.map(fetch, urls))
finally:
    manager.clear()

Wannan ɓangaren yana ɗauka iyakataccen jerin URL da girman amsa na yau da kullum. Don binciken URL na jama'a da aka yarda ba tare da tabbatar da wakili ba, ɗan takara daga jerin wakilan HTTP na iya maye gurbin misalin HTTP ɗin mai gudanarwa. Kowane sakamako ya haɗa da URL ɗinsa don mai kira ya iya danganta matsayi da jiki da buƙatar da ta dace. map yana mayar da sakamako cikin tsarin shigarwa; ana ɗaga keɓancewa a cikin ma'aikaci lokacin da aka karɓi wannan sakamakon.

ProxyManager ɗin da aka saita yana da tafkin haɗi na buƙatun. Tare da yanayin amsa da aka riga aka loda na asali da ake amfani da shi a nan, urllib3 yana karanta jiki kuma yana mayar da haɗin zuwa tafkinsa. Sigar yawo zata buƙaci cinye bayyane da sakin haɗin.

maxsize yawanci yana sarrafa haɗin da aka riƙe da za a sake amfani da su, ba iyaka mai tsauri akan kowace soket da aka buɗe ba. block=True yana canza halin tafkin da ya dace don masu kira su jira haɗi kyauta maimakon buɗe ƙarin sama da iyakar wannan tafkin.

Adadin ma'aikata yana iyakance kira na lokaci ɗaya a wannan misalin. Tare da wurare da yawa, shimfidar tafki da hanyar da ake amfani da ita don ramukan HTTPS suma suna da mahimmanci; bai kamata a kwatanta saitin tafki ɗaya a matsayin ƙididdigar soket ta duniya ga kowace mai masaukin ba.