Skip to content

How to calculate a weighted mean with missing values in NumPy (with Code)

The Short Answer (TL;DR)

Select value-weight pairs with a finite value and a positive finite weight before calling np.average. Use the weights from those same selected pairs in the denominator, reject invalid weights, and report an undefined result when no positive-weight pair remains.

import numpy as np
x=np.array([10.,20.,np.nan,40.]); w=np.array([1.,3.,2.,0.]); keep=np.isfinite(x)&np.isfinite(w)&(w>0)
print(np.average(x[keep],weights=w[keep]) if keep.any() else np.nan)

Detailed Implementation

This example uses synthetic data and does not need a network connection. Install the recorded package versions in a separate Python environment before running it.

python -m pip install numpy==2.5.4
from pathlib import Path
import numpy as np

def weighted_mean_audit(values,weights):
    if np.iscomplexobj(values) or np.iscomplexobj(weights):
        raise ValueError("Require real measurements and weights.")
    x=np.asarray(values,dtype=np.float64)
    w=np.asarray(weights,dtype=np.float64)
    if x.ndim!=1 or w.ndim!=1 or x.shape!=w.shape or x.size>1_000_000:
        raise ValueError("Require equal one-dimensional arrays with at most one million rows.")
    # Stated measurement contract: NaN means missing; infinity is invalid.
    if np.isinf(x).any() or np.isinf(w).any() or np.any(w<0):
        raise ValueError("Reject infinity and negative weights, even beside missing values.")
    if np.any(np.abs(x[np.isfinite(x)])>1e9) or np.any(w[np.isfinite(w)]>1e9):
        raise ValueError("This bounded example permits magnitudes and weights up to 1e9.")
    eligible=np.isfinite(x)&np.isfinite(w)&(w>0)
    paired_weight=float(w[eligible].sum())
    numerator=float(np.sum(x[eligible]*w[eligible]))
    mean=float(np.average(x[eligible],weights=w[eligible])) if paired_weight>0 else np.nan
    audit={"input_rows":int(x.size),"retained_pairs":int(eligible.sum()),
           "missing_values":int(np.isnan(x).sum()),"missing_weights":int(np.isnan(w).sum()),
           "zero_weights":int(np.count_nonzero(w==0)),
           "paired_weight":paired_weight,"weighted_sum":numerator,"mean":mean}
    return mean,eligible,audit

x=np.array([10.,20.,np.nan,40.,50.,0.,np.nan,100.])
w=np.array([1.,3.,2.,np.nan,0.,2.,np.nan,4.])
mean,eligible,audit=weighted_mean_audit(x,w)
text="\n".join(f"{key}={value}" for key,value in audit.items())+"\nretained_positions="+str(np.flatnonzero(eligible).tolist())+"\n"
Path("weighted-mean-missing-audit.txt").write_text(text,encoding="utf-8")
print(text)

The weighted mean is 47 with weighted sum 470, retained weight 10 and four retained positions [0, 1, 5, 7]. Two missing values, two missing weights and one zero weight are reported separately, with overlapping diagnostic counts.

Which pairs enter the weighted mean?

A row contributes only if its value is finite, its weight is finite and its weight is positive. Missing values use NaN in this example. A measured zero remains valid and enters with its positive weight. A zero weight contributes nothing and is excluded from the retained-pair count. The same Boolean mask selects both arrays, so the numerator and denominator describe identical observations.

Why must missing values remove their weights too?

The missing value at position 2 has weight 2, but no observed measurement to contribute to the numerator. Retaining that weight in the denominator would pull the result toward zero without any measured zero. Here the retained weighted sum is 470 and retained weight is 10, giving 47. Summing every known weight would give a denominator of 12 and a different quantity. Missing weights cannot be inferred from the values.

What does the audit count?

Four of eight pairs are retained, two source values are missing, two weights are missing and one weight is zero. Those last three counts overlap: position 6 lacks both fields, so they are diagnostics rather than a disjoint partition of rows. Retained positions [0, 1, 5, 7] make the actual selection explicit. Save the source arrays and this mask when downstream users need to trace exclusions.

How are negative weights, infinity and empty results handled?

This helper rejects negative weights and either sign of infinity, including invalid weights next to missing values. The policy suits nonnegative importance or frequency weights; signed coefficients need a different definition and interpretation. With no positive-weight observed pair, it returns NaN rather than zero and avoids calling np.average on a zero denominator. A single observed zero with positive weight correctly returns a mean of zero.

What numerical assumptions does the example make?

The demonstration accepts at most one million one-dimensional rows, finite measurement magnitudes up to one billion and nonnegative finite weights up to one billion, with NaN allowed as missing. Those explicit bounds keep sums and products far below float64 overflow. Equal shapes prevent accidental weight broadcasting. Binary floating point still introduces rounding, so the independent math.fsum reference and scaling checks use tolerances where needed. Exact monetary accounting needs a separate representation.

Does removing missing pairs correct missing-data bias?

The result describes the retained weighted observations. Pairwise exclusion does not show that missingness is random, restore excluded population weight or turn weights into an uncertainty estimate. If missingness depends on the measurement, the retained mean may differ from the target population mean. The mean uses the units of the measurement; a standard error or confidence interval also needs a justified sampling and weighting model.

Mathematical Context

Let I contain indices with a finite observed value and a positive finite weight. The weighted mean is the sum of w_i x_i over I divided by the sum of w_i over I, provided that denominator is positive. The synthetic numerator is 1 times 10 + 3 times 20 + 2 times 0 + 4 times 100 = 470, and the denominator is 10. Multiplying every retained weight by the same positive constant leaves the mean unchanged. This invariance does not imply that standard errors remain unchanged under every interpretation of weights.

\[ \bar{x}_w=\frac{\sum_{i\in I}w_i x_i}{\sum_{i\in I}w_i},\qquad \sum_{i\in I}w_i>0 \]

Reproducibility

Executed with Python 3.12.14; numpy 2.5.4. Automated checks confirm the stated output properties.

Primary documentation

Related guides