Stability monitoring

Viability testing is a fit-time guardrail: it rejects groupings that don’t hold up on a dev sample or CV folds. Once carving is done, the same question comes back in production — do these carved features still hold on data the model has never seen?

evaluate_stability() answers it without any training data: each carved feature already stores its own train reference (see The reference), so a fitted — or reloaded — carver can score a new sample on its own.

from AutoCarver import BinaryCarver

carver = BinaryCarver.load("carver.json")

report = carver.evaluate_stability(X_prod, y_prod)
report.summary           # one row per feature
report.per_modality      # one row per (feature, modality)
report.unstable_features # features flagged by any check

The target is optional. Labels usually lag in production, so carver.evaluate_stability(X_prod) computes the population-side metrics alone (PSI and the chi-square goodness-of-fit); the target-side ones are reported as NaN.

AutoCarver.stability.evaluate_stability(carver: BaseCarver, X: DataFrame, y: Series | None = None, *, alpha: float = 0.05) → StabilityReport

Evaluates carved features on a new sample against their train reference.

The reference is statistics, persisted on each feature at carving time, so no training data is needed at monitoring time. Production rates are recomputed with the carver’s own aggregator and target rate, which makes them directly comparable, and are then run through test_viability() — the same rank-inversion / Wilson min_freq / distinct-rates suite the carver used to accept the combination at fit time.

Parameters:

carver (BaseCarver) – A fitted carver (in memory or reloaded from JSON).
X (pd.DataFrame) – New (production) sample, with the carver’s raw feature columns.
y (pd.Series, optional) – Target for X. Without it only PSI and the chi-square goodness-of-fit on counts are computed, by default None.
alpha (float, optional) – Significance level for every test, by default 0.05.

Return type:

StabilityReport

Notes

Ordinal and multiclass carvers get PSI, the chi-square test and the full viability block, but no per-modality target-drift test: their rate is a ridit / correspondence-analysis score whose sampling variance is not recoverable from the three stored columns. The rate delta is still reported.

class AutoCarver.stability.StabilityReport(per_modality: DataFrame, per_feature: DataFrame, alpha: float, has_target: bool)

Comparison of carved features between their train reference and a new sample.

per_modality

One row per (feature, label): reference and new count / frequency / target rate, the modality’s PSI contribution, the rate delta and — when a target-drift test applies — its p-value.

Type:: pd.DataFrame

per_feature

One row per feature: psi and its flag, the chi-square goodness-of-fit test, and (with a target) the viability verdict produced by the carver’s own fit-time robustness tests.

Type:: pd.DataFrame

alpha

Significance level used by every test.

Type:: float

has_target

Whether a target was provided — without one, only the frequency-based metrics are computed.

Type:: bool

property summary: DataFrame: Per-feature verdicts, indexed by feature.

to_json() → dict[str, Any]

Converts to a JSON-serializable dict (for json.dump or MCP transport).

Frames round-trip through pandas’ own JSON writer so numpy scalars become base types and NaN becomes null.

property unstable_features: list[str]

Features needing attention.

Flagged when the population shifted (PSI above 0.25), when the chi-square test is both significant and carries a non-negligible effect size (chi2_cramerv at or above 0.1 — significance alone would flag nearly everything on a large extract), when the carver’s viability filter no longer passes, or when the reference is too incomplete to judge (psi_flag == "unknown").

The reference

At carving time the winning combination’s per-modality statistics are stored on the feature itself, as statistics — a frame indexed by final label carrying:

count and frequency — the population reference;
the evaluator’s target rate (target_mean, woe, odds_ratio, target_median, target_mean_ridit, target_mean_level or ca_score) — the target reference;
std for continuous targets — the dispersion needed to test a mean shift.

These survive save() / load(), together with any per-feature state the target rate was fit on (the ridit reference marginal, the correspondence-analysis axis). Features that were never carved — discretized only — carry no statistics and are skipped with a warning.

Production statistics are recomputed with the carver’s own aggregator and target rate, so both sides of every comparison are like-for-like by construction. Comparing a carver against the very sample it was fitted on therefore returns a PSI of exactly zero and no drift anywhere — a useful sanity check.

Population drift

Two complementary readings of the same shift in modality frequencies.

PSI (Population Stability Index) is the industry-standard magnitude:

\[\text{PSI} \;=\; \sum_{i} (f^{\text{new}}_i - f^{\text{ref}}_i) \, \log \frac{f^{\text{new}}_i}{f^{\text{ref}}_i}\]

reported per feature (psi) and per modality (psi_contribution), with the conventional verdict in psi_flag: stable below 0.1, moderate up to 0.25, shifted above. Both frequencies are floored so a modality that emptied out contributes a large but finite amount instead of infinity.

A fourth verdict, unknown, means the reference itself was incomplete — a manual split() leaves the affected bins’ statistics unknowable (NaN), and dropping them would silently renormalize the comparison onto a different support. The index is then NaN rather than a plausible-looking number, and the feature is reported as needing attention: unverifiable must never read as verified-stable.

Chi-square homogeneity is the significance-based counterpart: a two-sample test on the 2 x k table of reference and production counts (chi2, chi2_pvalue, chi2_significant). It is deliberately not a goodness-of-fit against the reference frequencies — the reference is itself an estimate from a finite train sample, and treating it as known truth would understate the p-value. It stays on scipy.stats.chi2_contingency() rather than Pearson’s \chi^2 because it needs the p-value and degrees of freedom, which AutoCarver.stats.pearson_chi2() doesn’t return.

Because any chi-square grows with sample size, Cramér’s V is reported beside it (chi2_cramerv) as the sample-size-independent effect size: V = sqrt(chi2 / N) for a 2 x k table, conventionally negligible below 0.1. unstable_features requires both significance and a non-negligible V, so a 300k-row extract doesn’t flag every feature over a shift too small to act on.

AutoCarver.stability.population_stability_index(ref_freq: Series, new_freq: Series, *, epsilon: float = 1e-06) → tuple[float, Series]

Population Stability Index and its per-modality contributions.

Both frequencies are floored at epsilon and renormalized so a modality that emptied out on either side yields a large-but-finite contribution instead of inf. Conventional reading: below 0.1 stable, 0.1 to 0.25 moderate shift, above 0.25 significant shift.

A reference carrying any NaN bin (a manual split leaves the affected bins’ statistics unknowable) makes the index undefined: dropping those bins would silently renormalize the comparison onto a different support. Both the total and every contribution are then NaN.

Parameters:

ref_freq (pd.Series) – Reference (train) frequency per modality — feature.statistics["frequency"].
new_freq (pd.Series) – Frequency per modality observed on the new sample.
epsilon (float, optional) – Floor applied to both frequencies, by default 1e-6.

Returns:

The PSI, and its per-modality contributions (indexed like ref_freq).

Return type:

tuple[float, pd.Series]

AutoCarver.stability.chi2_homogeneity(ref_count: Series, new_count: Series) → tuple[float, float, int, float]

Chi-square test of homogeneity between the reference and the new sample.

A two-sample test on the 2 x k table of per-modality counts, not a goodness-of-fit against fixed frequencies: the reference is itself an estimate from a finite train sample, and treating its frequencies as known truth would understate the p-value. Expected counts come from the table’s own margins, so no modality can be compared against a mis-scaled expectation. Modalities empty in both samples carry no information and are dropped.

The statistic grows with sample size, so a large production extract will flag shifts that are real but negligible. Cramér’s V is returned alongside it as the sample-size-independent effect size — for a 2 x k table V = sqrt(chi2 / N), bounded in [0, 1], conventionally read as negligible below 0.1, small to 0.3, moderate to 0.5, large above.

Returns:: Statistic, two-sided p-value, degrees of freedom and Cramér’s V. All nan / 0 when the table is degenerate (an incomplete reference, fewer than two informative modalities, or an empty sample).
Return type:: tuple[float, float, int, float]

Target drift

Every modality’s rate delta is reported (rate_delta). Whether it comes with a significance test depends on the target:

Carver	Test	p-value
Binary, one-vs-rest (`target_mean`, `woe`, `odds_ratio`)	Pooled two-proportion z-test	`drift_pvalue`
Continuous `target_mean`	Welch t-test (uses the stored `std`)	`drift_pvalue`
Continuous `target_median`	none	`NaN`
Ordinal, multiclass	none	`NaN`

Only two rates admit a test from the stored statistics. target_median does not: the stored std describes the spread of values, so feeding it to a standard-error-of-the-mean formula would test the wrong quantity. Ordinal and multiclass rates are bounded ridit / correspondence-analysis scores whose sampling variance cannot be recovered from the three stored columns either. In every such case the rate delta is still reported and the viability block below still runs — only the p-value is withheld.

A multiclass target carrying a class unseen at fit time raises: the correspondence-analysis axis is fixed at carving time and cannot project a class it never saw.

AutoCarver.stability.two_proportion_test(ref_rate: Series, ref_count: Series, new_rate: Series, new_count: Series) → Series

Per-modality two-sided p-value for a change in a binary target rate.

Pooled-proportion z-test. Both rates must already be probabilities — pass them through to_probability() first when the carver’s target rate is woe or odds_ratio.

AutoCarver.stability.welch_test(ref_mean: Series, ref_std: Series, ref_count: Series, new_mean: Series, new_std: Series, new_count: Series) → Series

Per-modality two-sided Welch p-value for a change in a continuous target mean.

Returns nan wherever an input is nan — notably for carvers fitted before the std column was persisted, and for singleton modalities.

Re-running the viability filter

The most direct question is also the cheapest: would this combination still have been accepted, had production been the dev sample?

test_viability() is called with the stored train rates as the reference and the production rates as the candidate, so the report’s viable / info columns are produced by exactly the machinery described in Viability testing — rank inversion, Wilson min_freq, distinct target rates — with the same human-readable failure messages (“Inversion of target rates per modality”, “Non-representative modality for min_freq=…”, “Non-distinct target rates per consecutive modalities”).

A rank inversion here is the strongest possible signal: the carved ordering that the whole model rests on no longer holds.

MCP

The same evaluation is exposed as an MCP tool (see LLM & MCP Integration):

evaluate_stability(path="holdout.csv", target="y")

It returns the report’s JSON form: unstable_features, per_feature and per_modality.