How to standardize columns with an explicit degrees-of-freedom choice in NumPy (with Code)
The Short Answer (TL;DR)
Subtract each column mean and divide by its standard deviation along axis=0, passing ddof explicitly. Use ddof=0 for the empirical population denominator or ddof=1 for the sample-variance denominator, and handle constant columns before division.
import numpy as np
x = np.array([[1.,10.],[3.,20.],[5.,30.],[7.,40.]])
print((x-x.mean(axis=0,keepdims=True))/x.std(axis=0,ddof=1,keepdims=True))
Detailed Implementation
This example uses synthetic data and does not need a network connection. Install the recorded package versions in a separate Python environment before running it.
python -m pip install numpy==2.5.4
from pathlib import Path
import numpy as np
def checked_matrix(values):
raw=np.asarray(values)
if raw.ndim!=2 or raw.shape[1]==0 or raw.dtype.kind not in "iuf":
raise ValueError("Require a two-dimensional real numeric matrix with at least one column.")
x=raw.astype(np.float64,copy=False)
if not np.isfinite(x).all() or np.any(np.abs(x)>1e12):
raise ValueError("Require finite measurements with magnitudes at most 1e12.")
return x
def fit_standardizer(values,ddof):
x=checked_matrix(values)
if isinstance(ddof,(bool,np.bool_)) or not isinstance(ddof,(int,np.integer)) or ddof not in (0,1):
raise ValueError("Choose ddof=0 or ddof=1 explicitly.")
if x.shape[0]<=ddof:
raise ValueError("The row count must exceed ddof.")
center=x.mean(axis=0,dtype=np.float64)
scale=x.std(axis=0,ddof=ddof,dtype=np.float64)
return center,scale
def transform(values,center,scale):
x=checked_matrix(values)
if x.shape[1]!=len(center) or center.shape!=scale.shape:
raise ValueError("The matrix must match the fitted column layout.")
result=np.full(x.shape,np.nan,dtype=np.float64)
# Constant training columns have no defined standardized coordinate.
np.divide(x-center,scale,out=result,where=scale>0)
return result
train=np.array([[1.,10.,7.],[3.,20.,7.],[5.,30.,7.],[7.,40.,7.]])
center,scale=fit_standardizer(train,ddof=1)
z=transform(train,center,scale)
future=np.array([[9.,50.,100.]])
future_z=transform(future,center,scale)
text=("CENTER\n"+np.array2string(center)+"\nSAMPLE SCALE\n"+
np.array2string(scale)+"\nSTANDARDIZED TRAINING ROWS\n"+np.array2string(z)+
"\nFUTURE ROW USING THE SAME FIT\n"+np.array2string(future_z)+
"\nconstant_columns="+str(np.flatnonzero(scale==0).tolist())+"\n")
Path("explicit-ddof-standardization.txt").write_text(text,encoding="utf-8")
print(text)
The fitted center is [4, 25, 7]. Sample scales are approximately [2.581989, 12.909944, 0]. The first two standardized columns have sample standard deviation one; the constant third column stays NaN. The future row scores approximately [1.936492, 1.936492, NaN].
Which axis and denominator define the standardization?
Rows are observations and columns are measurements. axis=0 fits one mean and one scale per column. With four rows, ddof=0 divides squared deviations by four and ddof=1 divides by three. The short answer uses keepdims=True so the fitted vectors retain a one-row shape during broadcasting. The full implementation stores one-dimensional fitted vectors and relies on their explicit column alignment.
Does ddof=1 make the standard deviation unbiased?
The N minus 1 denominator gives the usual sample variance estimate under appropriate independent, identically distributed sampling assumptions. Its square root is not itself an unbiased estimate of the population standard deviation. Standardizing a table does not establish those sampling assumptions. Choose the convention required by the analysis or downstream software, and record it so a comparison does not silently mix different scales.
What happens to a constant column?
The third training column is always seven, giving a scale of zero. Dividing its centered values by zero would not define a standardized coordinate. The helper leaves that entire output column as NaN and reports its index, rather than inventing zero scores. The future value of 100 in that column also remains undefined under this training fit. A model that needs a different constant-feature policy must state that policy separately.
Why retain fitted statistics for later rows?
The future row is transformed using training means [4, 25, 7] and training scales. It is not used to recalculate them. Refitting on a future or evaluation table changes the coordinate system and can allow information from that table to influence preprocessing. Preserve the fitted vectors together with the column order, units and ddof. The example validates a matching column count; a production table also needs a reliable feature-name and unit contract.
How can the numerical output be checked independently?
Python statistics.mean, stdev and pstdev provide scalar references for every column. The two nonconstant standardized training columns have mean zero and sample standard deviation one. Their population variance is 3/4, which is consistent with the chosen denominator rather than a failed scaling step. Reordering observations preserves fitted means and scales, and the check confirms that the original source matrix remains unchanged.
Which input and precision limits are explicit?
The helper accepts a real numeric matrix, rejects missing and infinite observations, bounds magnitudes at one trillion and accumulates in float64. It rejects invalid ddof values and a training row count that does not exceed ddof. This complete-data example does not silently drop different rows per column or fill missing values. Nearly constant columns can still give unstable practical scales, so choose a domain-specific tolerance when the measurement resolution requires it. Standardization does not make data normal or remove outliers.
Mathematical Context
For column j with N observations, the scale s_j is the square root of the sum of squared deviations from its mean divided by N minus d, where d is the chosen ddof. A standardized value is (x_ij minus the mean) divided by s_j when the scale is positive. The scale retains the measurement units and the resulting standardized value is dimensionless. For four complete training rows, sample-based standardized values have sum of squared scores equal to three; their population variance is therefore three divided by four.
\[ s_j=\sqrt{\frac{\sum_{i=1}^{N}(x_{ij}-\bar{x}_j)^2}{N-d}},\qquad z_{ij}=\frac{x_{ij}-\bar{x}_j}{s_j},\quad s_j>0 \]
Reproducibility
Executed with Python 3.12.14; numpy 2.5.4. Automated checks confirm the stated output properties.
Primary documentation
- NumPy std: axis, ddof and numerical precision
- Python statistics: scalar variance and standard-deviation references