Ukujikijela ama-proxy ku-httr ye-R nokuhlaziya i-HTML elandwe phansi

Gcina ukukhethwa kwe-proxy emsebenzini wokulanda ngokwawo. Ngemva kokuba i-httr ibuyise ikhasi, hlaziya ama-byte alolo mpendulo esikhundleni sokuthumela i-URL komunye umlandi.

Uma kunikezwe i-vector urls kanye ne-data frame proxies engekho ize enezikhathi host kanye ne-port eyinamba ephelele:

pages <- vector("list", length(urls))
stopifnot(nrow(proxies) > 0L)

for (i in seq_along(urls)) {
  row <- ((i - 1L) %% nrow(proxies)) + 1L
  response <- httr::GET(
    urls[[i]],
    httr::use_proxy(proxies$host[[row]], proxies$port[[row]]),
    httr::timeout(30)
  )
  httr::stop_for_status(response)
  pages[[i]] <- xml2::read_html(httr::content(response, as = "raw"))
}

Ezimweni zezicelo zamakhasi omphakathi ezivunyelwe ngabalamuli bomphakathi, uhlu lwama-proxy e-HTTP lunikeza abaqokelwe izikhathi host ne-port. Inkomba izungeza isuka kumugqa wokuqala we-proxy iye kwowokugcina bese ibuyela ekuqaleni. Ikhwelo ngalinye linokuhlelwa kwalo kwe-proxy; inqubo ye-R ingaqhubeka isebenza.

Amakhasi akhiqizwayo aqukethe imibhalo ye-HTML ehlaziyiwe. Sebenzisa i-rvest::html_elements kanye nemisebenzi yokukhipha edingekayo kuleyo mibhalo. Ukudlulisela i-URL ku-read_html esikhundleni salokho kuzoqala ukulandwa okusha ngokuhlelwa kwenethiwekhi kwayo yaloo ndlela.

Kuma-proxy aqinisekisiwe, dlulisela i-akhawunti ekhethiwe ngama-argument username ne-password e-use_proxy. I-endpoint ehlukile ayiqinisekisi IP yokuphuma yomphakathi ehlukile, ngakho khetha ichibi ngokuziphatha kwangempela kokuphuma kwesevisi.