How to flag missing values before imputation in Pandas (with Code)

Editorial · Published 3 November 2026

The Short Answer (TL;DR)

Capture an isna mask before filling values, then store it as a Boolean or zero-one indicator beside the original column. This preserves the distinction between an observed value and an identical replacement value after imputation.

import pandas as pd
df = pd.DataFrame({"value":pd.array([0,None,3],dtype="Float64")})
df["value_was_missing"] = df["value"].isna(); df["value_filled"] = df["value"].fillna(0); print(df)

Detailed Implementation

This example uses synthetic data and does not need a network connection. Install the recorded package versions in a separate Python environment before running it.

python -m pip install numpy==2.5.4 pandas==3.0.6

from pathlib import Path
import numpy as np
import pandas as pd

def capture_missingness(frame, features):
    features = list(features)
    if not frame.columns.is_unique or not features:
        raise ValueError("Use unique input columns and at least one selected feature.")
    if any(not isinstance(c,str) for c in features) or len(set(features)) != len(features):
        raise ValueError("Select distinct string column names.")
    if not set(features).issubset(frame.columns):
        raise ValueError("A selected feature is absent from the input.")
    names = [c + "_was_missing" for c in features]
    if set(names + ["row_has_missing"]).intersection(frame.columns):
        raise ValueError("An output name already exists; rename it explicitly.")
    # Capture once, before any fill operation.
    mask = frame[features].isna()
    result = frame.copy()
    for feature,name in zip(features,names):
        result[name] = mask[feature].astype("uint8")
    result["row_has_missing"] = mask.any(axis=1).astype("uint8")
    audit = pd.DataFrame({"missing":mask.sum(),"rows":len(frame),
                          "fraction_missing":mask.mean()})
    audit.index.name = "feature"
    return result,audit

# Synthetic data; empty strings and observed zeroes are intentional.
data = pd.DataFrame({
    "row_id": range(6),
    "value": pd.array([0,None,2.5,np.nan,0,5],dtype="Float64"),
    "label": pd.array(["",None,"A","B","",None],dtype="string"),
},index=[9,3,9,1,1,7])
result,audit = capture_missingness(data,["value","label"])
# Zero is a demonstration replacement, not a recommended default policy.
result["value_filled"] = result["value"].fillna(0)
missing_cells = int(audit["missing"].sum())
rows_with_missing = int(result["row_has_missing"].sum())
report = result.to_string(index=True,na_rep="NA") + "\n\n" + audit.to_string() + (
    f"\nmissing cells={missing_cells}/12; rows with missing={rows_with_missing}/6\n"
)
Path("missingness-flags.txt").write_text(report,encoding="utf-8")
print(report)

The value flags are 0, 1, 0, 1, 0, 0; label flags are 0, 1, 0, 0, 0, 1. The report counts four missing selected cells and three rows with any missing feature. The filled value column has no missing entries, while its saved indicator still sums to two.

Why must the mask be captured before replacement?

The first row contains a measured zero. The second row lacks a measurement and receives zero only as a demonstration replacement. Their filled values are identical, but the saved flags are zero and one. Calling isna after filling would report neither value as missing and lose that distinction. Keep the raw column and choose replacement rules from the application; an indicator does not make zero replacement appropriate.

Which entries count as missing?

The mask recognizes None, NumPy NaN, pandas NA and datetime NaT. It does not classify an empty string, an observed zero or positive infinity as missing. The two empty labels therefore receive zero flags. Define any sentinel normalization policy before capturing the mask, preserve the original data, and document that policy. If a blank code means unrecorded in a particular source, normalize that code explicitly rather than assuming isna will recognize it.

How are rows and feature counts preserved?

The helper works with an explicit list of distinct string feature names, creates one uint8 indicator per selected feature and adds a row-level any-missing indicator. It copies the input and rejects output-name collisions, missing features and duplicate column names. Duplicate index labels remain in their original order; indicators are assigned from an identically indexed mask. The audit reports missing counts and the row denominator for each selected feature. A flag value of one always refers to the input state, even after a later fill.

What does the synthetic audit show?

Both value and label have two missing entries among six rows, so each feature fraction is one third. Four of the twelve selected cells are missing, while three of six rows contain at least one missing selected feature. These are different denominators: the row fraction is one half. A row with two missing cells contributes twice to the cell count but once to the row count. An empty input has zero missing counts and undefined fractions, represented as NaN rather than a fabricated zero-percent estimate.

Can indicators be used in a predictive model?

An indicator records an observed data-availability pattern, not why a value is absent or whether that absence is random. It does not guarantee improved prediction or unbiased statistical inference. Keep the indicator feature list stable across training and inference. Learn any data-dependent selection or imputation parameters from training rows only and repeat that fitting inside each cross-validation training fold. The example records audit flags and demonstrates a fill; it does not train a model or compare its performance.

Mathematical Context

For row i and selected feature j, let M_ij equal one when the original entry is missing and zero otherwise. The feature missingness count is the sum of M_ij over rows; divide by n to obtain its observed fraction when n is positive. The any-missing row flag is one if at least one selected M_ij is one. Here each feature count is 2 and n=6, while the total is 4 missing cells among 6 times 2 selected cells. There are 3 rows with any missing feature. All these fractions are dimensionless descriptive summaries, not probabilities that the missing entries have a particular value. A saved mask cannot reconstruct an unavailable measurement or identify a missingness mechanism by itself.

\[ r_j=\frac{1}{n}\sum_{i=1}^{n}M_{ij},\quad M_{ij}\in\{0,1\},\quad n>0 \]

Reproducibility

Executed with Python 3.12.14; numpy 2.5.4, pandas 3.0.6. Automated checks confirm the stated output properties.