Juya wakilai a cikin httr na R da kuma nazarin HTML da aka saukar

Canza layin wakili a kowace buƙatar httr kuma ka tura bayanan da aka saukar zuwa xml2 da rvest.

Ka riƙe zaɓin wakili a aikin saukarwa. Bayan httr ya dawo da shafin, nazarci bayanan wannan amsar maimakon tura URL zuwa wani mai saukarwa.

Idan an ba da vector urls da data frame proxies mara fanko mai ginshiƙan host da port na lamba:

pages <- vector("list", length(urls))
stopifnot(nrow(proxies) > 0L)

for (i in seq_along(urls)) {
  row <- ((i - 1L) %% nrow(proxies)) + 1L
  response <- httr::GET(
    urls[[i]],
    httr::use_proxy(proxies$host[[row]], proxies$port[[row]]),
    httr::timeout(30)
  )
  httr::stop_for_status(response)
  pages[[i]] <- xml2::read_html(httr::content(response, as = "raw"))
}

Don buƙatun shafukan jama'a da aka yarda ta hanyar masu shiga tsakani na jama'a, jerin wakilan HTTP yana samar da 'yan takara don ginshiƙan host da port. Index ɗin yana juya daga layin farko zuwa na ƙarshe sannan ya koma farko. Kowace kira tana da nata saitin wakili; tsarin R na iya ci gaba da aiki.

Shafukan da aka samu suna ɗauke da takardun HTML da aka nazarta. Yi amfani da rvest::html_elements da ayyukan ciro da ake buƙata akan waɗannan takardu. Kai URL zuwa read_html maimakon haka zai fara sabon ɗauko da nata saitin cibiyar sadarwar wannan hanyar.

Don wakilai masu tabbatarwa, tura asusun da aka zaɓa ta hanyar username da password na use_proxy. Wani ƙarshen wuri ba ya tabbatar da wani IP na fita na jama'a daban, don haka zaɓi tafkin bisa ainihin halin fita na sabis ɗin.

Labarai masu alaƙa