Stream Multiline CSV Records with Python Requests and csv.reader

Stream a proxied response into csv.reader and preserve record boundaries instead of splitting each physical line at the delimiter.

stream=True lets processing begin before a proxied response body has been fully downloaded. For CSV, pass a text-line iterator to csv.reader so quoted delimiters and multiline records remain CSV data.

For an illustrative UTF-8 file using semicolons:

import csv
import requests

routes = {'https': 'http://proxy.example:8080'}
with requests.get(
    'https://data.example/observations.csv',
    proxies=routes,
    stream=True,
    timeout=(5, 30),
) as response:
    response.raise_for_status()
    response.encoding = 'utf-8'
    lines = (
        line + '\n'
        for line in response.iter_lines(
            chunk_size=65536, decode_unicode=True)
    )
    for record in csv.reader(lines, delimiter=';'):
        if record and record[0] == 'wanted':
            print(record)
            break

Choose the actual file’s encoding and dialect. For a public CSV download on a permitted public-proxy route, the HTTP proxy list provides candidates for the routes URL. This version normalizes physical line endings to LF: iter_lines removes them, and the generator adds a newline back for the CSV parser. That matters when a quoted field continues onto another physical line.

For workflows requiring byte-exact preservation of embedded CRLF characters, use a streaming text-wrapper arrangement that preserves those line endings. The example’s deliberate normalization is suited to record-oriented processing where LF is the intended representation.

One response, many reads

The iterator consumes chunks from one HTTP response. It does not issue a new request for every line. HTTP Range requests are a different mechanism and do not naturally identify CSV record boundaries.

Calling response.text or collecting all rows into a list would bring the full content back into memory. Streaming still needs room for transport buffers and the current record, which can itself contain a large field.

The response context closes the body when the loop exits, including an early match. The read timeout bounds a wait for incoming data rather than the total lifetime of a large file transfer.

Allakkat attuumassuteqartut