{
"cells": [
{
"cell_type": "markdown",
"id": "intro-md",
"metadata": {},
"source": [
"# Benchmark: AutoCarver vs. optbinning vs. KBinsDiscretizer\n",
"\n",
"This notebook runs the three binning libraries side-by-side on two public datasets:\n",
"\n",
"1. **German Credit** — binary classification, mixed numeric / categorical features, 1,000 rows.\n",
"2. **California Housing** — regression, all-numeric features, 20,640 rows.\n",
"\n",
"For each library and dataset, we report:\n",
"\n",
"- **`fit` and `transform` wall-clock** (seconds)\n",
"- **Downstream-model score** — AUC for binary, R² for regression — using a linear model (logistic regression / ridge) on the one-hot-encoded bin output\n",
"- **`train` → `test` score drop** as a coarse proxy for drift sensitivity\n",
"\n",
"All three libraries see the same `train + dev` data and are evaluated on the same held-out `test`. AutoCarver uses the dev sample for its built-in robustness veto; optbinning and KBinsDiscretizer don't have a dev-set concept and so treat the union of train + dev as one pooled training set — which is the comparison practitioners actually run.\n",
"\n",
"**This is not an IV / Tschuprow's T leaderboard.** Those metrics structurally favour the library whose objective they are. The downstream-model score is the metric a real scorecard team would use to pick a binner.\n",
"\n",
"Numbers come from a single run on a single machine with a fixed seed; treat them as illustrative, not as authoritative benchmark figures. Re-run on your own data before drawing conclusions."
]
},
{
"cell_type": "markdown",
"id": "setup-md",
"metadata": {},
"source": [
"## Setup"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "imports",
"metadata": {},
"outputs": [],
"source": [
"import time\n",
"import warnings\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"from sklearn.datasets import fetch_california_housing, fetch_openml\n",
"from sklearn.linear_model import LogisticRegression, Ridge\n",
"from sklearn.metrics import r2_score, roc_auc_score\n",
"from sklearn.model_selection import train_test_split\n",
"from sklearn.preprocessing import KBinsDiscretizer\n",
"\n",
"from AutoCarver import BinaryCarver, ContinuousCarver, Features\n",
"from AutoCarver.discretizers.utils.base_discretizer import ProcessingConfig\n",
"\n",
"try:\n",
" from optbinning import ContinuousOptimalBinning, OptimalBinning\n",
"\n",
" HAS_OPTBINNING = True\n",
"except ImportError:\n",
" HAS_OPTBINNING = False\n",
" print('optbinning is not installed \\u2014 its rows will be skipped.')\n",
"\n",
"SEED = 42\n",
"warnings.filterwarnings('ignore')\n",
"plt.rcParams['figure.figsize'] = (10, 3.5)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "helpers",
"metadata": {},
"outputs": [],
"source": [
"def one_hot(df):\n",
" \"\"\"Treat every bin label as a categorical level and one-hot encode it.\n",
"\n",
" Lets a linear downstream model consume any of the three libraries' outputs\n",
" uniformly, without us computing WoE per bin.\n",
" \"\"\"\n",
" return pd.get_dummies(df.astype(str), drop_first=True).astype(float)\n",
"\n",
"\n",
"def fit_eval_binary(X_train, X_test, y_train, y_test):\n",
" Xtr = one_hot(X_train)\n",
" Xte = one_hot(X_test).reindex(columns=Xtr.columns, fill_value=0.0)\n",
" model = LogisticRegression(max_iter=1000, random_state=SEED).fit(Xtr, y_train)\n",
" return {\n",
" 'train_auc': roc_auc_score(y_train, model.predict_proba(Xtr)[:, 1]),\n",
" 'test_auc': roc_auc_score(y_test, model.predict_proba(Xte)[:, 1]),\n",
" }\n",
"\n",
"\n",
"def fit_eval_regression(X_train, X_test, y_train, y_test):\n",
" Xtr = one_hot(X_train)\n",
" Xte = one_hot(X_test).reindex(columns=Xtr.columns, fill_value=0.0)\n",
" model = Ridge(random_state=SEED).fit(Xtr, y_train)\n",
" return {\n",
" 'train_r2': r2_score(y_train, model.predict(Xtr)),\n",
" 'test_r2': r2_score(y_test, model.predict(Xte)),\n",
" }\n",
"\n",
"\n",
"def plot_bars(results_df, score_cols, title):\n",
" fig, axes = plt.subplots(1, len(score_cols), figsize=(4 * len(score_cols), 3.5))\n",
" if len(score_cols) == 1:\n",
" axes = [axes]\n",
" for ax, col in zip(axes, score_cols):\n",
" results_df.plot.bar(x='library', y=col, ax=ax, legend=False, color='#4C72B0')\n",
" ax.set_title(col)\n",
" ax.set_xlabel('')\n",
" ax.tick_params(axis='x', rotation=0)\n",
" fig.suptitle(title)\n",
" fig.tight_layout()\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "binners",
"metadata": {},
"outputs": [],
"source": [
"from AutoCarver.combinations.binary import CramervCombinations\n",
"\n",
"MAX_N_MOD = 5\n",
"MIN_FREQ = 0.05\n",
"\n",
"def bin_with_autocarver(X_train, y_train, X_dev, y_dev, X_test, categoricals, quantitatives, kind):\n",
" Carver = BinaryCarver if kind == 'binary' else ContinuousCarver\n",
" features = Features(categoricals=categoricals, numericals=quantitatives)\n",
" config = ProcessingConfig(verbose=False, dropna=False, ordinal_encoding=False) # showing statistics\n",
" combination_evaluator = CramervCombinations() if kind == 'binary' else None\n",
" carver = Carver(features=features, min_freq=MIN_FREQ, max_n_mod=MAX_N_MOD, config=config,combination_evaluator=combination_evaluator)\n",
"\n",
" t0 = time.perf_counter()\n",
" X_tr = carver.fit_transform(X_train.copy(), y_train, X_dev=X_dev.copy(), y_dev=y_dev)\n",
" fit_t = time.perf_counter() - t0\n",
"\n",
" X_dv = carver.transform(X_dev.copy())\n",
" t1 = time.perf_counter()\n",
" X_te = carver.transform(X_test.copy())\n",
" transform_t = time.perf_counter() - t1\n",
" return pd.concat([X_tr, X_dv]), X_te, fit_t, transform_t, carver\n",
"\n",
"\n",
"def bin_with_optbinning(X_train, y_train, X_dev, y_dev, X_test, categoricals, quantitatives, kind):\n",
" Cls = OptimalBinning if kind == 'binary' else ContinuousOptimalBinning\n",
" X_all = pd.concat([X_train, X_dev])\n",
" y_all = pd.concat([y_train, y_dev])\n",
" binners = {}\n",
" train_binned = pd.DataFrame(index=X_all.index)\n",
" test_binned = pd.DataFrame(index=X_test.index)\n",
"\n",
" t0 = time.perf_counter()\n",
" for col in X_all.columns:\n",
" dtype = 'categorical' if col in categoricals else 'numerical'\n",
" binner = Cls(name=col, dtype=dtype, min_prebin_size=MIN_FREQ/2, max_n_bins=MAX_N_MOD)\n",
" binner.fit(X_all[col].to_numpy(), y_all.to_numpy())\n",
" binners[col] = binner\n",
" train_binned[col] = binner.transform(X_all[col].to_numpy(), metric='bins')\n",
" fit_t = time.perf_counter() - t0\n",
"\n",
" t1 = time.perf_counter()\n",
" for col, b in binners.items():\n",
" test_binned[col] = b.transform(X_test[col].to_numpy(), metric='bins')\n",
" transform_t = time.perf_counter() - t1\n",
" return train_binned, test_binned, fit_t, transform_t, binners\n",
"\n",
"\n",
"def bin_with_kbins(X_train, X_dev, X_test, categoricals, quantitatives, n_bins=5):\n",
" X_all = pd.concat([X_train, X_dev])\n",
" num_train = X_all[quantitatives].apply(lambda c: c.fillna(c.median()))\n",
" num_test = X_test[quantitatives].apply(lambda c: c.fillna(c.median()))\n",
" kbd = KBinsDiscretizer(n_bins=n_bins, encode='ordinal', strategy='quantile')\n",
"\n",
" t0 = time.perf_counter()\n",
" binned_num_train = pd.DataFrame(\n",
" kbd.fit_transform(num_train), columns=quantitatives, index=X_all.index\n",
" )\n",
" fit_t = time.perf_counter() - t0\n",
"\n",
" t1 = time.perf_counter()\n",
" binned_num_test = pd.DataFrame(\n",
" kbd.transform(num_test), columns=quantitatives, index=X_test.index\n",
" )\n",
" transform_t = time.perf_counter() - t1\n",
"\n",
" # KBins has no opinion on categoricals — pass them through as labels\n",
" train = pd.concat([binned_num_train, X_all[categoricals].astype(str)], axis=1)\n",
" test = pd.concat([binned_num_test, X_test[categoricals].astype(str)], axis=1)\n",
" return train, test, fit_t, transform_t, kbd"
]
},
{
"cell_type": "markdown",
"id": "binary-md",
"metadata": {},
"source": [
"## Binary classification — German Credit\n",
"\n",
"20 features (numeric + categorical), 1,000 rows, target = `class == 'bad'`. Train / dev / test split = 60 / 20 / 20 %."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "381a7051",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train=600, dev=200, test=200\n",
"categoricals=13, numericals=7\n",
"bad rate (train)=0.300, (test)=0.300\n"
]
}
],
"source": [
"credit = fetch_openml(data_id=31, as_frame=True)\n",
"df = credit.frame.copy()\n",
"\n",
"y_binary = (df['class'] == 'bad').astype(int)\n",
"X_binary = df.drop(columns=['class'])\n",
"\n",
"X_train, X_rest, y_train, y_rest = train_test_split(\n",
" X_binary, y_binary, test_size=0.4, random_state=SEED, stratify=y_binary,\n",
")\n",
"X_dev, X_test, y_dev, y_test = train_test_split(\n",
" X_rest, y_rest, test_size=0.5, random_state=SEED, stratify=y_rest,\n",
")\n",
"\n",
"categoricals = [c for c in X_binary.columns if X_binary[c].dtype == object or isinstance(X_binary[c].dtype, pd.CategoricalDtype)]\n",
"quantitatives = [c for c in X_binary.columns if c not in categoricals]\n",
"\n",
"print(f'train={len(X_train)}, dev={len(X_dev)}, test={len(X_test)}')\n",
"print(f'categoricals={len(categoricals)}, numericals={len(quantitatives)}')\n",
"print(f'bad rate (train)={y_train.mean():.3f}, (test)={y_test.mean():.3f}')"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "run-binary",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"WARNING: No robust combination for Categorical('own_telephone'). Consider increasing the size of X_dev or dropping the feature (X not representative of X_dev for this feature).\n",
"WARNING: No robust combination for Categorical('foreign_worker'). Consider increasing the size of X_dev or dropping the feature (X not representative of X_dev for this feature).\n",
"WARNING: No robust combination for Numerical('residence_since'). Consider increasing the size of X_dev or dropping the feature (X not representative of X_dev for this feature).\n",
"WARNING: No robust combination for Numerical('existing_credits'). Consider increasing the size of X_dev or dropping the feature (X not representative of X_dev for this feature).\n",
"WARNING: No robust combination for Numerical('num_dependents'). Consider increasing the size of X_dev or dropping the feature (X not representative of X_dev for this feature).\n"
]
},
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" library | \n",
" fit_s | \n",
" transform_s | \n",
" train_auc | \n",
" test_auc | \n",
" auc_drop | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" AutoCarver | \n",
" 1.102 | \n",
" 0.0109 | \n",
" 0.8496 | \n",
" 0.8044 | \n",
" 0.0451 | \n",
"
\n",
" \n",
" | 1 | \n",
" optbinning | \n",
" 1.099 | \n",
" 0.0114 | \n",
" 0.8523 | \n",
" 0.7931 | \n",
" 0.0592 | \n",
"
\n",
" \n",
" | 2 | \n",
" KBinsDiscretizer | \n",
" 0.003 | \n",
" 0.0016 | \n",
" 0.8401 | \n",
" 0.7943 | \n",
" 0.0458 | \n",
"
\n",
" \n",
"
\n",
"
"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"plot_bars(binary_results, ['fit_s', 'test_auc', 'auc_drop'], 'German Credit \\u2014 binary classification')"
]
},
{
"cell_type": "markdown",
"id": "regression-md",
"metadata": {},
"source": [
"## Regression — California Housing\n",
"\n",
"6 numeric demographic features (Latitude / Longitude dropped — see comment in the next cell), 20,640 rows, target = median house value. Same 60 / 20 / 20 split."
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "load-regression",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"train=12384, dev=4128, test=4128\n",
"numericals=8 (['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude'])\n"
]
}
],
"source": [
"housing = fetch_california_housing(as_frame=True)\n",
"X_reg = housing.frame.drop(columns=['MedHouseVal'])\n",
"y_reg = housing.frame['MedHouseVal']\n",
"\n",
"X_train, X_rest, y_train, y_rest = train_test_split(X_reg, y_reg, test_size=0.4, random_state=SEED)\n",
"X_dev, X_test, y_dev, y_test = train_test_split(X_rest, y_rest, test_size=0.5, random_state=SEED)\n",
"\n",
"quantitatives = list(X_reg.columns)\n",
"categoricals = []\n",
"\n",
"print(f'train={len(X_train)}, dev={len(X_dev)}, test={len(X_test)}')\n",
"print(f'numericals={len(quantitatives)} ({quantitatives})')"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "adebc1c4",
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" library | \n",
" fit_s | \n",
" transform_s | \n",
" train_r2 | \n",
" test_r2 | \n",
" r2_drop | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" AutoCarver | \n",
" 2.870 | \n",
" 0.0093 | \n",
" 0.6652 | \n",
" 0.6595 | \n",
" 0.0057 | \n",
"
\n",
" \n",
" | 1 | \n",
" optbinning | \n",
" 2.274 | \n",
" 0.0089 | \n",
" 0.5145 | \n",
" 0.5077 | \n",
" 0.0068 | \n",
"
\n",
" \n",
" | 2 | \n",
" KBinsDiscretizer | \n",
" 0.008 | \n",
" 0.0015 | \n",
" 0.6181 | \n",
" 0.6192 | \n",
" -0.0011 | \n",
"
\n",
" \n",
"
\n",
"