Rarraba rikodin CSV na layi da yawa da Python Requests da csv.reader

Rarraba amsa ta hanyar wakili zuwa csv.reader ka kuma adana iyakokin rikodi maimakon raba kowane layin zahiri a mai rabawa.

stream=True yana ba da damar fara aiki kafin an sauke jikin amsar da ke wucewa ta wakili gaba ɗaya. Don CSV, ba da mai maimaita layukan rubutu ga csv.reader domin masu rabawa a cikin waƙafi da rikodin layi da yawa su kasance bayanan CSV.

Don fayil ɗin UTF-8 na misali da ke amfani da waƙafi-tsakiya:

import csv
import requests

routes = {'https': 'http://proxy.example:8080'}
with requests.get(
    'https://data.example/observations.csv',
    proxies=routes,
    stream=True,
    timeout=(5, 30),
) as response:
    response.raise_for_status()
    response.encoding = 'utf-8'
    lines = (
        line + '\n'
        for line in response.iter_lines(
            chunk_size=65536, decode_unicode=True)
    )
    for record in csv.reader(lines, delimiter=';'):
        if record and record[0] == 'wanted':
            print(record)
            break

Zaɓi ainihin rikodin fayil ɗin da yare. Don saukar da CSV na jama'a a kan hanyar wakili na jama'a da aka yarda, jerin wakilan HTTP yana ba da 'yan takara don URL ɗin routes. Wannan sigar tana daidaita ƙarshen layukan zahiri zuwa LF: iter_lines yana cire su, kuma janar ɗin yana sake ƙara sabon layi don mai nazarin CSV. Wannan yana da muhimmanci lokacin da wani fili a cikin waƙafi ya ci gaba a wani layin zahiri.

Don ayyukan da ke buƙatar adana haruffan CRLF da aka saka daidai da byte, yi amfani da tsarin nade rubutu mai gudana wanda ke adana waɗannan ƙarshen layukan. Daidaitawar da aka yi da gangan a misalin ta dace da aiki mai mayar da hankali kan rikodi inda LF shine wakilcin da ake so.

Amsa ɗaya, karatu da yawa

Mai maimaitawa yana cinye guntu daga amsar HTTP ɗaya. Ba ya aika sabuwar buƙata don kowane layi. Buƙatun HTTP Range wata hanya ce daban kuma ba sa gane iyakokin rikodin CSV kai tsaye.

Kira response.text ko tattara duk layuka a cikin jeri zai dawo da dukan abun ciki cikin ƙwaƙwalwar ajiya. Rarraba har yanzu yana buƙatar wuri don buffer ɗin jigila da rikodin na yanzu, wanda shi kansa zai iya ƙunsar babban fili.

Yanayin amsar yana rufe jiki lokacin da madauki ya fita, har da idan an sami daidaituwa da wuri. Ƙayyadaddun lokacin karatu yana iyakance jira don bayanan da ke shigowa, ba dukan rayuwar canja wurin babban fayil ba.

Labarai masu alaƙa