How to read a large CSV in chunks in Pandas (with Code)

Editorial · Published 8 November 2026

The Short Answer (TL;DR)

Pass chunksize to read_csv and process each returned chunk without collecting the chunks into one DataFrame. Keep only the state needed by the calculation, specify column types and missing-value conventions, and combine sums and observed counts rather than averaging chunk means.

from io import StringIO; import pandas as pd
reader = pd.read_csv(StringIO("value\n10\n20\n30\n"),chunksize=2,dtype={"value":"Int64"})
print(sum(int(chunk["value"].sum()) for chunk in reader))

Detailed Implementation

This example uses synthetic data and does not need a network connection. Install the recorded package versions in a separate Python environment before running it.

python -m pip install numpy==2.5.4 pandas==3.0.6

from pathlib import Path
import pandas as pd

def stream_summary(path, chunksize=50000):
    if isinstance(chunksize,bool) or not isinstance(chunksize,int) or chunksize<1:
        raise ValueError("chunksize must be a positive Python integer.")
    states = {}  # Per region: row count, observed count, exact Python-integer cents.
    with pd.read_csv(path,usecols=["region","amount_cents"],
            dtype={"region":"string","amount_cents":"Int64"},
            keep_default_na=False,na_values={"amount_cents":[""]},
            encoding="utf-8",on_bad_lines="error",chunksize=chunksize) as reader:
        for chunk in reader:
            for region, amount in chunk[["region","amount_cents"]].itertuples(index=False,name=None):
                if pd.isna(region) or not region.strip():
                    raise ValueError("A nonblank region code is required.")
                state = states.setdefault(region,[0,0,0])
                state[0] += 1
                if pd.notna(amount):
                    state[1] += 1
                    state[2] += int(amount)
    rows = [{"region":key,"rows":n,"observed":count,
             "sum_cents":total if count else pd.NA,
             "mean_cents":total/count if count else pd.NA}
            for key,(n,count,total) in sorted(states.items())]
    return pd.DataFrame(rows,columns=["region","rows","observed","sum_cents","mean_cents"])

# A small, self-contained fixture; replace this path with the real CSV.
csv_text = ("region,amount_cents,unused\nA,1000,x\nB,2000,x\nA,,x\n"
            "B,0,x\nNA,3000,x\nA,-500,x\nNA,4500,x\nD,,x\n")
path = Path("chunked-payments-fixture.csv")
path.write_text(csv_text,encoding="utf-8")
result = stream_summary(path,chunksize=2)
report = result.to_string(index=False,na_rep="NA")+"\n"
Path("chunked-csv-summary.txt").write_text(report,encoding="utf-8")
print(report)

The output has regions A, B, D and NA. Row counts total eight, observed counts total six and observed sums total 10,000 cents. A has mean 250 cents; D has no observed amount and retains missing sum and mean.

What is kept in memory?

Each iteration holds a parsed chunk and a dictionary with three totals per region. The helper does not append chunks to a list or concatenate the original rows. With C rows in the current chunk and K distinct region codes, retained data grows with C and K rather than the total row count alone. This is not a measured memory benchmark or a guarantee that any chosen chunk fits. Long text fields, parser buffers and a growing number of distinct groups still require memory.

Why specify column types and missing-value conventions?

The example reads only region and amount_cents. Region codes remain text, including the literal code NA, because default missing-token recognition is disabled. Only an empty amount field is declared missing. The amount column uses nullable signed 64-bit integers, and invalid integer text stops the calculation. Choose these rules from the source contract; a blank value, a missing region code and a measured zero have different meanings. No rows are silently skipped to make parsing succeed.

How are summaries combined across chunk boundaries?

Each row increases the region row count. An observed amount also increases the observed count and adds its integer cents to the running sum. Python integers keep the accumulated total from overflowing a fixed-width integer reduction, although each input amount must fit the declared CSV dtype. The final mean divides the combined sum by the combined observed count. Region A has three rows but only two observed amounts, so its mean is 250 cents.

Why is an average of chunk means unreliable?

Chunks can have different numbers of observed values, even when their row counts match. Averaging their means gives each chunk equal weight and can disagree with the mean of the observations. Carry a sum and count instead. The eight-row fixture contains six observed amounts totalling 10,000 cents, so the observed overall mean is about 1,666.667 cents. An all-missing region retains a missing sum and mean; a measured zero remains an observation.

Which operations need more than this small running state?

This implementation is suited to a grouped sum, row count and observed mean. Exact global duplicate removal, median calculation, sorting and joins can need coordination or state that grows across chunks. Removing duplicates separately inside each chunk would miss repeated keys in different chunks. Use an algorithm or storage engine designed for the required global operation. The helper also keeps every distinct region in memory, so a nearly unique grouping field defeats the small-state assumption.

How should the example be adapted and checked?

The fixture is deliberately small and writes a separate output report without overwriting its input. Replace the input path and start with a chunk size suited to available memory. The context manager closes the reader when processing finishes or raises an exception. Verify totals against an independently processed sample and include a header-only file in checks. A completely empty file is an error here. No speed improvement or maximum file size is claimed from this synthetic run.

Mathematical Context

For region g and chunk j, retain a sum S_gj, an observed count n_gj and a row count r_gj. Combining chunks gives S_g as the sum of S_gj and n_g as the sum of n_gj. The observed mean is S_g/n_g only when n_g is positive. Equivalently, chunk means can be combined with observed-count weights; an unweighted mean generally answers a different question. Sums and means are in cents, while counts are dimensionless. This is descriptive aggregation with an explicit missing-value policy, not evidence that missing amounts are zero or missing at random.

\[ \bar{x}_g=\frac{\sum_j S_{gj}}{\sum_j n_{gj}},\qquad \sum_j n_{gj}>0 \]

Reproducibility

Executed with Python 3.12.14; numpy 2.5.4, pandas 3.0.6. Automated checks confirm the stated output properties.