Rotate Proxies in R’s httr and Parse the Downloaded HTML

Keep proxy selection on the download operation. Once httr has returned the page, parse that response’s bytes rather than passing the URL to another downloader.

Given a vector urls and a nonempty data frame proxies with host and integer port columns:

pages <- vector("list", length(urls))
stopifnot(nrow(proxies) > 0L)

for (i in seq_along(urls)) {
  row <- ((i - 1L) %% nrow(proxies)) + 1L
  response <- httr::GET(
    urls[[i]],
    httr::use_proxy(proxies$host[[row]], proxies$port[[row]]),
    httr::timeout(30)
  )
  httr::stop_for_status(response)
  pages[[i]] <- xml2::read_html(httr::content(response, as = "raw"))
}

For public-page requests permitted through public intermediaries, the HTTP proxy list supplies candidates for the data frame’s host and integer port columns. The index cycles from the first proxy row to the last and wraps around. Each call has its own proxy configuration; the R process can continue running.

The resulting pages contain parsed HTML documents. Apply rvest::html_elements and the needed extraction functions to those documents. Passing a URL to read_html instead would initiate a new retrieval with that path’s own network configuration.

For authenticated proxies, pass the selected account through use_proxy’s username and password arguments. A different endpoint does not guarantee a different public exit IP, so choose the pool according to the service’s actual exit behavior.