How to fill missing values within groups in Pandas (with Code)

Editorial · Published 2 November 2026

The Short Answer (TL;DR)

Use groupby with transform("median") to return each group median to its original rows, then fill only missing numeric values with that aligned Series. Preserve the original column and a missingness flag, and leave groups with no observed values unresolved instead of inventing a fallback.

import pandas as pd
df = pd.DataFrame({"group":["A","A","B","B"],"value":[2.,None,10.,None]})
df["value_filled"] = df["value"].fillna(df.groupby("group",dropna=False,observed=True)["value"].transform("median")); print(df)

Detailed Implementation

This example uses synthetic data and does not need a network connection. Install the recorded package versions in a separate Python environment before running it.

python -m pip install numpy==2.5.4 pandas==3.0.6

from pathlib import Path
import numpy as np
import pandas as pd
from pandas.api.types import is_numeric_dtype, is_bool_dtype, is_complex_dtype

def fill_group_medians(frame):
    if not frame.columns.is_unique or not {"group","value"}.issubset(frame.columns):
        raise ValueError("Require unique columns including group and value.")
    reserved = {"group_median","value_was_missing","value_filled"}
    if reserved.intersection(frame.columns):
        raise ValueError("Output column names already exist; rename them explicitly.")
    values = frame["value"]
    if not is_numeric_dtype(values) or is_bool_dtype(values) or is_complex_dtype(values):
        raise TypeError("Use a real numeric measurement column, not strings or booleans.")
    numeric = values.astype("float64")
    if np.isinf(numeric.to_numpy()).any():
        raise ValueError("Resolve infinite measurements before imputation.")
    result = frame.copy()
    result["value_was_missing"] = numeric.isna()
    # Here missing group labels form one explicit group; no rows are dropped.
    result["group_median"] = numeric.groupby(
        result["group"],dropna=False,observed=True,sort=False
    ).transform("median")
    result["value_filled"] = numeric.fillna(result["group_median"])
    return result

# Synthetic measurements; duplicate index labels are intentional.
data = pd.DataFrame({
    "row_id": range(10),
    "group": pd.Categorical(["A","A","A","B","B","B","C","C",None,None],
                            categories=["A","B","C","unused"]),
    "value": [2.,np.nan,4.,10.,np.nan,14.,np.nan,np.nan,7.,np.nan],
},index=[12,7,12,4,4,9,1,5,8,8])
result = fill_group_medians(data)
missing_before = int(result["value_was_missing"].sum())
filled_count = int((result["value_was_missing"] & result["value_filled"].notna()).sum())
unresolved = int(result["value_filled"].isna().sum())
report = result.to_string(index=True,na_rep="NA") + (
    f"\nmissing before={missing_before}; filled={filled_count}; unresolved={unresolved}\n"
)
Path("group-median-imputation.txt").write_text(report,encoding="utf-8")
print(report)

The filled values in original row order are 2, 3, 4, 10, 12, 14, NA, NA, 7 and 7. The report ends with missing before=5; filled=3; unresolved=2. Raw measurements, row order and duplicate index labels are preserved.

Why use transform instead of a grouped aggregate?

An aggregate produces one value per group. Its group labels do not match the measurement rows. transform broadcasts each median back to the original row index, so fillna receives a Series aligned with the measurement column. The example intentionally includes duplicate index labels and verifies that row order, index labels and every observed value remain unchanged. Avoid assigning a grouped median directly to the raw column.

What happens to empty groups and missing group keys?

Group A has observed values 2 and 4, so its replacement is 3. Group B uses 12, while both Group C measurements remain missing because that group has no observations. This example explicitly groups missing labels together with dropna=False; its observed value 7 fills the other missing-label row. That policy is appropriate only when unknown labels may share a reference group. If they should not, keep those rows unresolved and audit them separately. observed=True avoids materializing unused categorical groups and does not manufacture observations.

Which values and dtypes does the helper accept?

The helper accepts one real numeric measurement column and converts its working copy to float64 so a median such as 1.5 can be represented even when observed values are integers. It preserves the original value column, group column and index. It rejects strings, booleans, complex values, infinite measurements, duplicate column names and pre-existing output columns. Large integers can lose exactness when converted to floating point; use a precision-preserving approach if exact large-integer arithmetic is required. isna records the missingness mask before replacement.

What does the audit count establish?

The full example starts with five missing measurements, fills three from observed group medians and leaves two unresolved. It writes a reproducible text report without replacing the raw measurements. The separate missingness flag records which input rows lacked values; it is not evidence that those rows were missing at random. A deterministic replacement is a data-processing choice, not a recovered observation, a confidence interval or an additional independent sample.

How should this be used before model evaluation?

The example describes one analysis table. In predictive work, estimate group medians using training rows only, then apply that saved mapping to validation or test rows. Recomputing medians after combining the splits leaks held-out information. Specify an explicit policy for unseen groups and training groups with no observed measurements. Fit preprocessing separately inside each training fold when using cross-validation; the scikit-learn common-pitfalls documentation explains this principle. This example does not fit or evaluate a predictive model.

Mathematical Context

For each group g, form the ordered observed measurements and let m_g be their sample median. With an even number of observations, use the average of the two central values; with an odd number, use the central value. Keep y_i=x_i when x_i is observed, and set y_i=m_g only when x_i is missing and that group has a defined median. The median and filled values have the same measurement units as x. For Group A, m_A=(2+4)/2=3; for Group B, m_B=(10+14)/2=12. The sum of absolute deviations from observed group values is minimized by a sample median, but that property does not prove that median imputation preserves a population distribution, relationships between variables or inferential uncertainty. An all-missing group supplies no information for estimating its median, so its replacements remain missing.

\[ m_g\in\operatorname{argmin}_{m}\sum_{i\in O_g}|x_i-m| \]

Reproducibility

Executed with Python 3.12.14; numpy 2.5.4, pandas 3.0.6. Automated checks confirm the stated output properties.