Setting things up
About this notebook
In this notebook, we embark on a journey to elevate the predictive capabilities of the California Housing Prices Dataset through advanced preprocessing using the ContinuousCarver pipeline. Renowned for its association-maximizing discretization, ContinuousCarver is a powerful Python tool designed to handle diverse data types—whether they be quantitative or qualitative. Our specific goal is to prepare the dataset for continuous regression tasks, such as predicting housing prices.
The California Housing Prices Dataset is a treasure trove of features, encompassing information on factors like square footage, bedrooms, location, and more. By employing ContinuousCarver, we aim to seamlessly discretize both quantitative and qualitative features, tailoring them for optimal representation in our continuous regression models.
Throughout this notebook, we’ll explore the intricacies of ContinuousCarver’s discretization pipeline, witnessing its adaptability to a variety of data types. Whether it involves transforming square footage or encoding location information, ContinuousCarver ensures that each feature is finely tuned for our regression tasks.
Join us in this exploration as we leverage the power of ContinuousCarver to preprocess the California Housing Prices Dataset. Through effective feature engineering and discretization, our aim is to create a dataset that captures the nuanced relationships within the housing market, setting the stage for the development of accurate and impactful continuous regression models.
Let’s dive in and uncover the potential of ContinuousCarver in transforming the California Housing Prices Dataset for optimal predictive modeling.
Installation
[1]:
# %pip install AutoCarver[jupyter]
Califorinia Housing Prices Data
In this example notebook, we will use the California Housing Prices dataset.
The California Housing Prices dataset is a well-known dataset in the field of machine learning and statistics. It provides information about housing districts in California and is frequently used for regression analysis and predictive modeling tasks.
Comprising housing-related metrics for various districts in California, such as median house value, median income, housing median age, average rooms, average bedrooms, population, households, and more, the California Housing Prices dataset is a valuable resource for exploring the relationships between different features and predicting the median house values (continuous regression).
[2]:
from sklearn import datasets
# Load dataset directly from sklearn
housing = datasets.fetch_california_housing(as_frame=True)
# conversion to pandas
housing_data = housing["data"]
housing_data[housing["target_names"][0]] = housing["target"]
# Display the first few rows of the dataset
housing_data.head()
[2]:
| MedInc | HouseAge | AveRooms | AveBedrms | Population | AveOccup | Latitude | Longitude | MedHouseVal | |
|---|---|---|---|---|---|---|---|---|---|
| 0 | 8.3252 | 41.0 | 6.984127 | 1.023810 | 322.0 | 2.555556 | 37.88 | -122.23 | 4.526 |
| 1 | 8.3014 | 21.0 | 6.238137 | 0.971880 | 2401.0 | 2.109842 | 37.86 | -122.22 | 3.585 |
| 2 | 7.2574 | 52.0 | 8.288136 | 1.073446 | 496.0 | 2.802260 | 37.85 | -122.24 | 3.521 |
| 3 | 5.6431 | 52.0 | 5.817352 | 1.073059 | 558.0 | 2.547945 | 37.85 | -122.25 | 3.413 |
| 4 | 3.8462 | 52.0 | 6.281853 | 1.081081 | 565.0 | 2.181467 | 37.85 | -122.25 | 3.422 |
Target type and Carver selection
[3]:
target = "MedHouseVal"
housing_data[target].describe()
[3]:
count 20640.000000
mean 2.068558
std 1.153956
min 0.149990
25% 1.196000
50% 1.797000
75% 2.647250
max 5.000010
Name: MedHouseVal, dtype: float64
The target "MedHouseVal" is a continuous target of type float64 used in a regression task. Hence we will use AutoCarver.ContinuousCarver and AutoCarver.selectors.RegressionSelector in following code blocks.
Data Sampling
[4]:
from sklearn.model_selection import train_test_split
# stratified sampling by target
train_set, dev_set = train_test_split(housing_data, test_size=0.33, random_state=42)
# checking target rate per dataset
train_set[target].mean(), dev_set[target].mean()
[4]:
(np.float64(2.0666362048018514), np.float64(2.072459655020552))
Picking up columns to Carve
[5]:
train_set.head()
[5]:
| MedInc | HouseAge | AveRooms | AveBedrms | Population | AveOccup | Latitude | Longitude | MedHouseVal | |
|---|---|---|---|---|---|---|---|---|---|
| 5088 | 0.9809 | 19.0 | 3.187726 | 1.129964 | 726.0 | 2.620939 | 33.98 | -118.28 | 1.214 |
| 17096 | 4.2232 | 33.0 | 6.189696 | 1.086651 | 1015.0 | 2.377049 | 37.46 | -122.23 | 3.637 |
| 5617 | 3.5488 | 42.0 | 4.821577 | 1.095436 | 1044.0 | 4.331950 | 33.79 | -118.26 | 2.056 |
| 20060 | 1.6469 | 24.0 | 4.274194 | 1.048387 | 1686.0 | 4.532258 | 35.87 | -119.26 | 0.476 |
| 895 | 3.9909 | 14.0 | 4.608303 | 1.089350 | 2738.0 | 2.471119 | 37.54 | -121.96 | 2.360 |
[6]:
# column data types
train_set.dtypes
[6]:
MedInc float64
HouseAge float64
AveRooms float64
AveBedrms float64
Population float64
AveOccup float64
Latitude float64
Longitude float64
MedHouseVal float64
dtype: object
All features are quantitative continuous features at the exception of Latitude and Longitude which are geographical featues (not supported by AutoCarver as is). All other features will be added to the list of quantitative_features.
[7]:
from AutoCarver import Features
# lists of features per data type
features = Features(quantitatives=["MedInc", "HouseAge", "AveRooms", "AveBedrms", "Population", "AveOccup"])
C:\Users\defra\Desktop\git\PROJECTS\AutoCarver\AutoCarver\combinations\utils\combination_evaluator.py:10: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from tqdm.autonotebook import tqdm
Using AutoCarver
AutoCarver settings
Representativness of modalities
The attribute min_freq allows one to choose the minimum frequency per basic modalities. It is used:
For quantitative features, to define the number of quantiles to initialy discretize the features with.
For qualitative features, to define the threshold under which a modality is grouped to either a default value or its closest modality.
[8]:
min_freq = 0.1
Tip: should be set between 0.01 (slower, preciser, less robust) and 0.2 (faster, more robust)
Desired number of modalities
The attribute max_n_mod allows one to choose the maximum number of modalities per carved feature. It is used by Carvers has the upper limit of number of modalities per consecutive combination of modalities.
[9]:
max_n_mod = 4
Tip: should be set between 3 (faster, more robust) and 7 (slower, preciser, less robust)
Grouping NaNs
The attribute dropna allows one to choose whether or not nan should be grouped with another modality. If set to True, Carvers will first find the most suitable combination of non-nan values, and then test out all possible combinations with nan.
[10]:
dropna = False # anyway, there are no nan in this dataset
Type of output carved features
The attribute ordinal_encoding allows one to choose the output type:
Use
Truefor integer output of ranked modalities (default)Use
Falsefor string output of modalities
[11]:
ordinal_encoding = True
Fitting AutoCarver
First, all quantitative features are discretized:
Using
ContinuousDiscretizerfor quantile discretization that keeps track of over-represented values (more frequent thanmin_freq)Using
OrdinalDiscretizerfor any remaining under-represented values (less frequent thanmin_freq/2) to be grouped with its closest modality
Second, all features are carved following this recipe, for all classes of
train_set[target](except one):The raw distribution is printed out on provided
train_setanddev_set. It’s the output of the discretization stepGrouping modalities: all consecutive combinations of modalities are applied to
train_setComputing associations: the association metric (Krsuskal-Wallis’ statistic, by default) is computed with the provided
train_set[target]Combinations are sorted in descending order by association value
Testing robustness: finds the first combination that checks the following:
Representativness of modalities on
train_setanddev_set(all should be more frequent thanmin_freq/2)Distinct target rates per consecutive modalities on
train_setanddev_setNo inversion of target rates between
train_setanddev_set(same ordering of modalities by target rate)
(Optional) If requested via
dropna=True, and if any, all combinations of modalities withnanare applied totrain_setand steps 3. and 4. are runThe carved distribution is printed out on provided
train_setanddev_set. It’s the output of the carving step
[12]:
from AutoCarver import ContinuousCarver
# intiating AutoCarver
auto_carver = ContinuousCarver(
features=features,
min_freq=min_freq,
max_n_mod=max_n_mod,
dropna=dropna,
ordinal_encoding=ordinal_encoding,
verbose=True, # showing statistics
copy=True, # whether or not to return a copy of the input dataset
)
# fitting on training sample, a dev sample can be specified to evaluate carving robustness
train_set_processed = auto_carver.fit_transform(train_set, train_set[target], X_dev=dev_set, y_dev=dev_set[target])
------
--- [QuantitativeDiscretizer] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])
- [ContinuousDiscretizer] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])
- [OrdinalDiscretizer] Fit Features(['HouseAge'])
------
---------
------ [ContinuousCarver] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])
--- [ContinuousCarver] Fit Quantitative('MedInc') (1/6)
[ContinuousCarver] Raw distribution
| target_rate | frequency | |
|---|---|---|
| x <= 1.6e+00 | 1.1102 | 0.0500 |
| 1.6e+00 < x <= 1.9e+00 | 1.1285 | 0.0500 |
| 1.9e+00 < x <= 2.2e+00 | 1.2198 | 0.0500 |
| 2.2e+00 < x <= 2.4e+00 | 1.3171 | 0.0500 |
| 2.4e+00 < x <= 2.6e+00 | 1.3817 | 0.0500 |
| 2.6e+00 < x <= 2.7e+00 | 1.5409 | 0.0500 |
| 2.7e+00 < x <= 3.0e+00 | 1.6159 | 0.0500 |
| 3.0e+00 < x <= 3.1e+00 | 1.6906 | 0.0499 |
| 3.1e+00 < x <= 3.3e+00 | 1.8232 | 0.0500 |
| 3.3e+00 < x <= 3.5e+00 | 1.9059 | 0.0500 |
| 3.5e+00 < x <= 3.7e+00 | 2.0076 | 0.0502 |
| 3.7e+00 < x <= 4.0e+00 | 2.0271 | 0.0498 |
| 4.0e+00 < x <= 4.2e+00 | 2.1456 | 0.0500 |
| 4.2e+00 < x <= 4.5e+00 | 2.2433 | 0.0500 |
| 4.5e+00 < x <= 4.8e+00 | 2.3621 | 0.0501 |
| 4.8e+00 < x <= 5.1e+00 | 2.3986 | 0.0499 |
| 5.1e+00 < x <= 5.5e+00 | 2.6438 | 0.0500 |
| 5.5e+00 < x <= 6.2e+00 | 2.9324 | 0.0500 |
| 6.2e+00 < x <= 7.3e+00 | 3.4592 | 0.0500 |
| 7.3e+00 < x | 4.3784 | 0.0500 |
| target_rate | frequency |
|---|---|
| 1.1017 | 0.0509 |
| 1.0410 | 0.0502 |
| 1.2407 | 0.0501 |
| 1.2919 | 0.0506 |
| 1.4676 | 0.0536 |
| 1.5605 | 0.0417 |
| 1.6280 | 0.0584 |
| 1.7519 | 0.0471 |
| 1.8443 | 0.0504 |
| 1.8500 | 0.0498 |
| 2.0040 | 0.0533 |
| 2.0890 | 0.0502 |
| 2.1641 | 0.0505 |
| 2.2700 | 0.0540 |
| 2.3768 | 0.0439 |
| 2.5087 | 0.0479 |
| 2.6814 | 0.0483 |
| 2.9805 | 0.0479 |
| 3.3748 | 0.0530 |
| 4.3748 | 0.0483 |
Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1390.66it/s]
Computing associations: 100%|██████████| 1159/1159 [00:02<00:00, 434.10it/s]
Testing robustness : 0%| | 0/1159 [00:00<?, ?it/s]
[ContinuousCarver] Carved distribution
| target_rate | frequency | |
|---|---|---|
| x <= 2.6e+00 | 1.2314 | 0.2500 |
| 2.6e+00 < x <= 4.0e+00 | 1.8016 | 0.3500 |
| 4.0e+00 < x <= 5.5e+00 | 2.3587 | 0.2499 |
| 5.5e+00 < x | 3.5900 | 0.1501 |
| target_rate | frequency |
|---|---|
| 1.2315 | 0.2554 |
| 1.8222 | 0.3509 |
| 2.3953 | 0.2446 |
| 3.5721 | 0.1491 |
--- [ContinuousCarver] Fit Quantitative('HouseAge') (2/6)
[ContinuousCarver] Raw distribution
| target_rate | frequency | |
|---|---|---|
| x <= 8.0e+00 | 2.1158 | 0.0537 |
| 8.0e+00 < x <= 1.2e+01 | 1.8220 | 0.0477 |
| 1.2e+01 < x <= 1.5e+01 | 1.8590 | 0.0613 |
| 1.5e+01 < x <= 1.6e+01 | 2.0358 | 0.0393 |
| 1.6e+01 < x <= 1.8e+01 | 1.9013 | 0.0596 |
| 1.8e+01 < x <= 2.0e+01 | 1.9399 | 0.0468 |
| 2.0e+01 < x <= 2.2e+01 | 2.0134 | 0.0404 |
| 2.2e+01 < x <= 2.5e+01 | 2.1055 | 0.0705 |
| 2.5e+01 < x <= 2.6e+01 | 2.0977 | 0.0300 |
| 2.6e+01 < x <= 2.8e+01 | 2.0218 | 0.0475 |
| 2.8e+01 < x <= 3.1e+01 | 2.0439 | 0.0682 |
| 3.1e+01 < x <= 3.3e+01 | 2.0275 | 0.0575 |
| 3.3e+01 < x <= 3.4e+01 | 2.1189 | 0.0328 |
| 3.4e+01 < x <= 3.5e+01 | 2.0204 | 0.0395 |
| 3.5e+01 < x <= 3.7e+01 | 2.0750 | 0.0687 |
| 3.7e+01 < x <= 3.9e+01 | 2.0212 | 0.0361 |
| 3.9e+01 < x <= 4.2e+01 | 2.0013 | 0.0450 |
| 4.2e+01 < x <= 4.5e+01 | 2.1301 | 0.0485 |
| 4.5e+01 < x | 2.4785 | 0.1072 |
| target_rate | frequency |
|---|---|
| 2.0205 | 0.0526 |
| 1.7827 | 0.0443 |
| 1.8780 | 0.0556 |
| 1.9208 | 0.0335 |
| 1.9484 | 0.0652 |
| 1.9517 | 0.0470 |
| 2.1141 | 0.0421 |
| 2.1179 | 0.0759 |
| 2.0888 | 0.0299 |
| 2.2138 | 0.0443 |
| 1.9546 | 0.0664 |
| 2.0512 | 0.0565 |
| 2.1979 | 0.0346 |
| 2.1762 | 0.0408 |
| 2.0747 | 0.0659 |
| 1.9885 | 0.0388 |
| 2.0394 | 0.0508 |
| 2.0015 | 0.0489 |
| 2.4651 | 0.1069 |
Grouping modalities : 100%|█████████▉| 986/987 [00:00<00:00, 1105.48it/s]
Computing associations: 100%|██████████| 987/987 [00:02<00:00, 440.98it/s]
Testing robustness : 1%| | 6/987 [00:00<00:02, 356.01it/s]
[ContinuousCarver] Carved distribution
| target_rate | frequency | |
|---|---|---|
| x <= 2.2e+01 | 1.9494 | 0.3486 |
| 2.2e+01 < x <= 2.6e+01 | 2.1032 | 0.1005 |
| 2.6e+01 < x <= 4.5e+01 | 2.0509 | 0.4437 |
| 4.5e+01 < x | 2.4785 | 0.1072 |
| target_rate | frequency |
|---|---|
| 1.9447 | 0.3403 |
| 2.1097 | 0.1058 |
| 2.0670 | 0.4470 |
| 2.4651 | 0.1069 |
--- [ContinuousCarver] Fit Quantitative('AveRooms') (3/6)
[ContinuousCarver] Raw distribution
| target_rate | frequency | |
|---|---|---|
| x <= 3.4e+00 | 1.9126 | 0.0500 |
| 3.4e+00 < x <= 3.8e+00 | 1.8286 | 0.0500 |
| 3.8e+00 < x <= 4.1e+00 | 1.8169 | 0.0500 |
| 4.1e+00 < x <= 4.3e+00 | 1.8418 | 0.0500 |
| 4.3e+00 < x <= 4.5e+00 | 1.7529 | 0.0500 |
| 4.5e+00 < x <= 4.6e+00 | 1.7915 | 0.0500 |
| 4.6e+00 < x <= 4.8e+00 | 1.8214 | 0.0500 |
| 4.8e+00 < x <= 4.9e+00 | 1.7685 | 0.0500 |
| 4.9e+00 < x <= 5.1e+00 | 1.7466 | 0.0500 |
| 5.1e+00 < x <= 5.2e+00 | 1.7717 | 0.0500 |
| 5.2e+00 < x <= 5.4e+00 | 1.8664 | 0.0500 |
| 5.4e+00 < x <= 5.5e+00 | 1.8472 | 0.0500 |
| 5.5e+00 < x <= 5.7e+00 | 1.9199 | 0.0500 |
| 5.7e+00 < x <= 5.9e+00 | 1.9910 | 0.0500 |
| 5.9e+00 < x <= 6.1e+00 | 2.0870 | 0.0500 |
| 6.1e+00 < x <= 6.3e+00 | 2.1908 | 0.0500 |
| 6.3e+00 < x <= 6.5e+00 | 2.4050 | 0.0500 |
| 6.5e+00 < x <= 6.9e+00 | 2.6874 | 0.0500 |
| 6.9e+00 < x <= 7.7e+00 | 3.1129 | 0.0500 |
| 7.7e+00 < x | 3.1718 | 0.0500 |
| target_rate | frequency |
|---|---|
| 1.8659 | 0.0518 |
| 1.8728 | 0.0505 |
| 1.7627 | 0.0524 |
| 1.8020 | 0.0543 |
| 1.7223 | 0.0552 |
| 1.6802 | 0.0452 |
| 1.7707 | 0.0530 |
| 1.8030 | 0.0443 |
| 1.8209 | 0.0523 |
| 1.8326 | 0.0437 |
| 1.7923 | 0.0550 |
| 1.9388 | 0.0514 |
| 1.9465 | 0.0501 |
| 2.0248 | 0.0468 |
| 2.1049 | 0.0483 |
| 2.2239 | 0.0490 |
| 2.4339 | 0.0467 |
| 2.7667 | 0.0468 |
| 3.1001 | 0.0548 |
| 3.2429 | 0.0483 |
Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1307.90it/s]
Computing associations: 100%|██████████| 1159/1159 [00:02<00:00, 431.03it/s]
Testing robustness : 1%| | 7/1159 [00:00<00:04, 280.25it/s]
[ContinuousCarver] Carved distribution
| target_rate | frequency | |
|---|---|---|
| x <= 5.2e+00 | 1.8053 | 0.5000 |
| 5.2e+00 < x <= 5.9e+00 | 1.9061 | 0.2000 |
| 5.9e+00 < x <= 6.5e+00 | 2.2275 | 0.1500 |
| 6.5e+00 < x | 2.9907 | 0.1501 |
| target_rate | frequency |
|---|---|
| 1.7933 | 0.5028 |
| 1.9208 | 0.2033 |
| 2.2521 | 0.1440 |
| 3.0420 | 0.1499 |
--- [ContinuousCarver] Fit Quantitative('AveBedrms') (4/6)
[ContinuousCarver] Raw distribution
| target_rate | frequency | |
|---|---|---|
| x <= 9.400e-01 | 2.0684 | 0.0500 |
| 9.400e-01 < x <= 9.672e-01 | 2.0735 | 0.0500 |
| 9.672e-01 < x <= 9.832e-01 | 2.2167 | 0.0501 |
| 9.832e-01 < x <= 9.958e-01 | 2.1706 | 0.0499 |
| 9.958e-01 < x <= 1.007e+00 | 2.1310 | 0.0500 |
| 1.007e+00 < x <= 1.015e+00 | 2.2358 | 0.0500 |
| 1.015e+00 < x <= 1.025e+00 | 2.1668 | 0.0500 |
| 1.025e+00 < x <= 1.033e+00 | 2.2102 | 0.0500 |
| 1.033e+00 < x <= 1.041e+00 | 2.1295 | 0.0500 |
| 1.041e+00 < x <= 1.050e+00 | 2.1548 | 0.0500 |
| 1.050e+00 < x <= 1.058e+00 | 2.1238 | 0.0500 |
| 1.058e+00 < x <= 1.067e+00 | 2.1025 | 0.0500 |
| 1.067e+00 < x <= 1.077e+00 | 2.0704 | 0.0500 |
| 1.077e+00 < x <= 1.088e+00 | 2.0664 | 0.0501 |
| 1.088e+00 < x <= 1.100e+00 | 2.1118 | 0.0499 |
| 1.100e+00 < x <= 1.116e+00 | 1.9937 | 0.0500 |
| 1.116e+00 < x <= 1.138e+00 | 1.9405 | 0.0500 |
| 1.138e+00 < x <= 1.174e+00 | 1.7990 | 0.0500 |
| 1.174e+00 < x <= 1.273e+00 | 1.9162 | 0.0500 |
| 1.273e+00 < x | 1.6515 | 0.0500 |
| target_rate | frequency |
|---|---|
| 2.0416 | 0.0539 |
| 2.2043 | 0.0527 |
| 2.0997 | 0.0482 |
| 2.1835 | 0.0487 |
| 2.2628 | 0.0552 |
| 2.1619 | 0.0480 |
| 2.2295 | 0.0567 |
| 2.1690 | 0.0493 |
| 2.1581 | 0.0528 |
| 2.1202 | 0.0476 |
| 2.1039 | 0.0452 |
| 2.1595 | 0.0509 |
| 2.1037 | 0.0521 |
| 2.0662 | 0.0484 |
| 2.0487 | 0.0489 |
| 1.9543 | 0.0467 |
| 1.8871 | 0.0484 |
| 1.8680 | 0.0499 |
| 1.8371 | 0.0465 |
| 1.7182 | 0.0498 |
Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1292.68it/s]
Computing associations: 100%|██████████| 1159/1159 [00:03<00:00, 368.26it/s]
Testing robustness : 3%|▎ | 35/1159 [00:00<00:02, 476.85it/s]
[ContinuousCarver] Carved distribution
| target_rate | frequency | |
|---|---|---|
| x <= 9.67e-01 | 2.0709 | 0.1000 |
| 9.67e-01 < x <= 1.06e+00 | 2.1710 | 0.4500 |
| 1.06e+00 < x <= 1.14e+00 | 2.0475 | 0.2999 |
| 1.14e+00 < x | 1.7888 | 0.1501 |
| target_rate | frequency |
|---|---|
| 2.1221 | 0.1066 |
| 2.1685 | 0.4517 |
| 2.0390 | 0.2955 |
| 1.8072 | 0.1462 |
--- [ContinuousCarver] Fit Quantitative('Population') (5/6)
[ContinuousCarver] Raw distribution
| target_rate | frequency | |
|---|---|---|
| x <= 3.5e+02 | 1.9859 | 0.0501 |
| 3.5e+02 < x <= 5.1e+02 | 2.1616 | 0.0501 |
| 5.1e+02 < x <= 6.3e+02 | 2.1117 | 0.0501 |
| 6.3e+02 < x <= 7.2e+02 | 2.2819 | 0.0497 |
| 7.2e+02 < x <= 7.9e+02 | 2.0335 | 0.0509 |
| 7.9e+02 < x <= 8.6e+02 | 2.2113 | 0.0492 |
| 8.6e+02 < x <= 9.4e+02 | 2.0772 | 0.0498 |
| 9.4e+02 < x <= 1.0e+03 | 2.1386 | 0.0500 |
| 1.0e+03 < x <= 1.1e+03 | 2.0430 | 0.0503 |
| 1.1e+03 < x <= 1.2e+03 | 2.0506 | 0.0496 |
| 1.2e+03 < x <= 1.3e+03 | 2.0870 | 0.0505 |
| 1.3e+03 < x <= 1.4e+03 | 2.0195 | 0.0497 |
| 1.4e+03 < x <= 1.5e+03 | 2.0004 | 0.0502 |
| 1.5e+03 < x <= 1.6e+03 | 2.1102 | 0.0498 |
| 1.6e+03 < x <= 1.7e+03 | 2.0346 | 0.0500 |
| 1.7e+03 < x <= 1.9e+03 | 1.9139 | 0.0499 |
| 1.9e+03 < x <= 2.2e+03 | 2.0006 | 0.0500 |
| 2.2e+03 < x <= 2.6e+03 | 2.0707 | 0.0500 |
| 2.6e+03 < x <= 3.3e+03 | 1.9614 | 0.0500 |
| 3.3e+03 < x | 2.0428 | 0.0500 |
| target_rate | frequency |
|---|---|
| 1.9012 | 0.0530 |
| 2.1915 | 0.0520 |
| 2.1706 | 0.0523 |
| 2.1062 | 0.0514 |
| 2.2019 | 0.0531 |
| 2.1765 | 0.0490 |
| 2.2025 | 0.0506 |
| 2.1329 | 0.0553 |
| 2.1744 | 0.0437 |
| 2.1319 | 0.0480 |
| 1.9939 | 0.0534 |
| 2.0096 | 0.0465 |
| 1.9569 | 0.0465 |
| 1.9756 | 0.0504 |
| 2.0815 | 0.0496 |
| 2.0272 | 0.0461 |
| 1.9789 | 0.0487 |
| 1.9355 | 0.0496 |
| 2.0714 | 0.0518 |
| 2.0157 | 0.0487 |
Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1369.87it/s]
Computing associations: 100%|██████████| 1159/1159 [00:02<00:00, 431.67it/s]
Testing robustness : 16%|█▋ | 191/1159 [00:00<00:01, 611.18it/s]
[ContinuousCarver] Carved distribution
| target_rate | frequency | |
|---|---|---|
| x <= 6.3e+02 | 2.0864 | 0.1503 |
| 6.3e+02 < x <= 8.6e+02 | 2.1743 | 0.1498 |
| 8.6e+02 < x <= 2.2e+03 | 2.0433 | 0.5498 |
| 2.2e+03 < x | 2.0250 | 0.1501 |
| target_rate | frequency |
|---|---|
| 2.0867 | 0.1572 |
| 2.1618 | 0.1536 |
| 2.0607 | 0.5390 |
| 2.0084 | 0.1502 |
--- [ContinuousCarver] Fit Quantitative('AveOccup') (6/6)
[ContinuousCarver] Raw distribution
| target_rate | frequency | |
|---|---|---|
| x <= 1.87e+00 | 2.7122 | 0.0500 |
| 1.87e+00 < x <= 2.07e+00 | 2.6633 | 0.0500 |
| 2.07e+00 < x <= 2.22e+00 | 2.3373 | 0.0500 |
| 2.22e+00 < x <= 2.34e+00 | 2.3080 | 0.0500 |
| 2.34e+00 < x <= 2.43e+00 | 2.1976 | 0.0500 |
| 2.43e+00 < x <= 2.51e+00 | 2.2064 | 0.0500 |
| 2.51e+00 < x <= 2.60e+00 | 2.1736 | 0.0500 |
| 2.60e+00 < x <= 2.67e+00 | 2.1862 | 0.0500 |
| 2.67e+00 < x <= 2.74e+00 | 2.1378 | 0.0500 |
| 2.74e+00 < x <= 2.82e+00 | 2.1902 | 0.0500 |
| 2.82e+00 < x <= 2.90e+00 | 2.1824 | 0.0500 |
| 2.90e+00 < x <= 2.98e+00 | 2.0741 | 0.0500 |
| 2.98e+00 < x <= 3.07e+00 | 2.0255 | 0.0501 |
| 3.07e+00 < x <= 3.17e+00 | 1.9914 | 0.0498 |
| 3.17e+00 < x <= 3.28e+00 | 1.8992 | 0.0500 |
| 3.28e+00 < x <= 3.43e+00 | 1.8926 | 0.0500 |
| 3.43e+00 < x <= 3.61e+00 | 1.7085 | 0.0500 |
| 3.61e+00 < x <= 3.88e+00 | 1.5666 | 0.0500 |
| 3.88e+00 < x <= 4.32e+00 | 1.4505 | 0.0500 |
| 4.32e+00 < x | 1.4294 | 0.0500 |
| target_rate | frequency |
|---|---|
| 2.7684 | 0.0484 |
| 2.5334 | 0.0435 |
| 2.3989 | 0.0542 |
| 2.3641 | 0.0533 |
| 2.2272 | 0.0546 |
| 2.2969 | 0.0489 |
| 2.3179 | 0.0508 |
| 2.0793 | 0.0467 |
| 2.1847 | 0.0521 |
| 2.1752 | 0.0504 |
| 2.0762 | 0.0533 |
| 2.0535 | 0.0501 |
| 2.0535 | 0.0528 |
| 1.9477 | 0.0458 |
| 1.8397 | 0.0449 |
| 1.8861 | 0.0514 |
| 1.7301 | 0.0448 |
| 1.6200 | 0.0499 |
| 1.4423 | 0.0527 |
| 1.4596 | 0.0515 |
Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1387.85it/s]
Computing associations: 100%|██████████| 1159/1159 [00:02<00:00, 483.00it/s]
Testing robustness : 0%| | 3/1159 [00:00<00:06, 178.53it/s]
[ContinuousCarver] Carved distribution
| target_rate | frequency | |
|---|---|---|
| x <= 2.2e+00 | 2.5709 | 0.1501 |
| 2.2e+00 < x <= 3.1e+00 | 2.1681 | 0.5001 |
| 3.1e+00 < x <= 3.6e+00 | 1.8729 | 0.1998 |
| 3.6e+00 < x | 1.4822 | 0.1501 |
| target_rate | frequency |
|---|---|
| 2.5615 | 0.1461 |
| 2.1836 | 0.5129 |
| 1.8527 | 0.1869 |
| 1.5056 | 0.1541 |
AutoCarver analysis
Carving Summary
[13]:
auto_carver.summary
[13]:
| content | target_rate | frequency | ||||
|---|---|---|---|---|---|---|
| feature | kruskal | n_mod | label | |||
| Quantitative('MedInc') | 6037.182135 | 4 | 0 | x <= 2.6e+00 | 1.231421 | 0.250000 |
| 1 | 2.6e+00 < x <= 4.0e+00 | 1.801562 | 0.350014 | |||
| 2 | 4.0e+00 < x <= 5.5e+00 | 2.358660 | 0.249928 | |||
| 3 | 5.5e+00 < x | 3.590040 | 0.150058 | |||
| Quantitative('HouseAge') | 163.527841 | 4 | 0 | x <= 2.2e+01 | 1.949361 | 0.348568 |
| 1 | 2.2e+01 < x <= 2.6e+01 | 2.103173 | 0.100521 | |||
| 2 | 2.6e+01 < x <= 4.5e+01 | 2.050927 | 0.443665 | |||
| 3 | 4.5e+01 < x | 2.478542 | 0.107246 | |||
| Quantitative('AveRooms') | 1391.586489 | 4 | 0 | x <= 5.2e+00 | 1.805255 | 0.500000 |
| 1 | 5.2e+00 < x <= 5.9e+00 | 1.906098 | 0.199957 | |||
| 2 | 5.9e+00 < x <= 6.5e+00 | 2.227531 | 0.149986 | |||
| 3 | 6.5e+00 < x | 2.990676 | 0.150058 | |||
| Quantitative('AveBedrms') | 315.794350 | 4 | 0 | x <= 9.67e-01 | 2.070937 | 0.100014 |
| 1 | 9.67e-01 < x <= 1.06e+00 | 2.171033 | 0.450029 | |||
| 2 | 1.06e+00 < x <= 1.14e+00 | 2.047547 | 0.299899 | |||
| 3 | 1.14e+00 < x | 1.788831 | 0.150058 | |||
| Quantitative('Population') | 16.109709 | 4 | 0 | x <= 6.3e+02 | 2.086394 | 0.150347 |
| 1 | 6.3e+02 < x <= 8.6e+02 | 2.174297 | 0.149841 | |||
| 2 | 8.6e+02 < x <= 2.2e+03 | 2.043255 | 0.549754 | |||
| 3 | 2.2e+03 < x | 2.024995 | 0.150058 | |||
| Quantitative('AveOccup') | 991.408301 | 4 | 0 | x <= 2.2e+00 | 2.570888 | 0.150058 |
| 1 | 2.2e+00 < x <= 3.1e+00 | 2.168126 | 0.500072 | |||
| 2 | 3.1e+00 < x <= 3.6e+00 | 1.872867 | 0.199812 | |||
| 3 | 3.6e+00 < x | 1.482183 | 0.150058 |
As requested with
ordinal_encoding=True, output labels are integers of modalitiesFor quantitative feature
Population, the selected combination of modalities groups populations as follows:label
0: lower or equal to 630 people (content="x <= 6.3e+02")label
1: greater than 630 people and lower or equal to 860 people (content="6.3e+02 < x <= 8.6e+02")label
2: greater than 860 people and lower or equal to 2200 people (content="8.6e+02 < x <= 2.2e+03")label
3: higher than 2200 people (content="2.2e+03 < x")
Detailed overview of tested combinations
[14]:
features["AveOccup"].history.head(7)
[14]:
| info | kruskal | combination | n_mod | dropna | train | viable | dev | |
|---|---|---|---|---|---|---|---|---|
| 0 | Raw distribution (n_mod=20>max_n_mod=4) | 1062.072498 | {'x <= 1.87e+00': 'x <= 1.87e+00', '1.87e+00 <... | 20 | False | NaN | NaN | NaN |
| 1 | Not viable | 994.514410 | {'x <= 1.87e+00': 'x <= 1.87e+00', '1.87e+00 <... | 4 | False | {'viable': True, 'info': ''} | False | {'viable': False, 'info': 'Non-representative ... |
| 2 | Not viable | 994.504665 | {'x <= 1.87e+00': 'x <= 1.87e+00', '1.87e+00 <... | 4 | False | {'viable': True, 'info': ''} | False | {'viable': False, 'info': 'Non-representative ... |
| 3 | Not viable | 991.504255 | {'x <= 1.87e+00': 'x <= 1.87e+00', '1.87e+00 <... | 4 | False | {'viable': True, 'info': ''} | False | {'viable': False, 'info': 'Non-representative ... |
| 4 | Best for kruskal and max_n_mod=4 | 991.408301 | {'x <= 1.87e+00': 'x <= 1.87e+00', '1.87e+00 <... | 4 | False | {'viable': True, 'info': ''} | True | {'viable': True, 'info': ''} |
| 5 | Not checked | 991.308986 | {'x <= 1.87e+00': 'x <= 1.87e+00', '1.87e+00 <... | 4 | False | NaN | NaN | NaN |
| 6 | Not checked | 988.666983 | {'x <= 1.87e+00': 'x <= 1.87e+00', '1.87e+00 <... | 4 | False | NaN | NaN | NaN |
[15]:
features["AveOccup"].history.dev[1]
[15]:
{'viable': False, 'info': 'Non-representative modality for min_freq=10.00%'}
The most associated combination of feature
AveOccup(the first tested out, whereinfo!="Raw distribution") did not pass the viability tests. When looking inhistory.dev:"Non-representative modality for min_freq=10.00%": tells us that a modality is unstable betweentrain_setanddev_set
For feature feature
AveOccup, the 4th combination is the first to pass tests:viabe=Trueinfo="Best for kruskal and max_n_mod=4"Kruskal-Wallis’ H with
MedHouseValis991.408301for this combinationFollowing combinations (less associated with the target) where not tested:
info="Not checked"
For all combinations
dropna=Falsemeans that it is not a combination in whichnans are being grouped with other modalities (as requested withdropna=False)
Saving and Loading AutoCarver
Saving
All Carvers can safely be stored as a .json file.
[16]:
auto_carver.save("continuous_carver.json")
Loading
Carvers can safely be loaded from a .json file.
[17]:
from AutoCarver import ContinuousCarver
# loading json file
auto_carver = ContinuousCarver.load('continuous_carver.json')
Applying AutoCarver
[18]:
dev_set_processed = auto_carver.transform(dev_set)
[19]:
dev_set_processed[auto_carver.features].apply(lambda u: u.value_counts(dropna=False, normalize=True))
[19]:
| MedInc | HouseAge | AveRooms | AveBedrms | Population | AveOccup | |
|---|---|---|---|---|---|---|
| 0.0 | 0.255432 | 0.340282 | 0.502789 | 0.106577 | 0.157223 | 0.146066 |
| 1.0 | 0.350851 | 0.105843 | 0.203318 | 0.451703 | 0.153553 | 0.512918 |
| 2.0 | 0.244568 | 0.447005 | 0.144011 | 0.295508 | 0.539049 | 0.186876 |
| 3.0 | 0.149149 | 0.106870 | 0.149883 | 0.146213 | 0.150176 | 0.154140 |
Feature Selection
Selectors settings
Features to select from
Here all features have been carved using ContinuousCarver, hence all features are qualitative.
Number of features to select
The attribute n_best_per_type allows one to choose the number of features to be selected per data type (quantitative and qualitative).
[20]:
n_best_per_type = 6
Using Selectors
[21]:
from AutoCarver.selectors import RegressionSelector
# select the most target associated qualitative features
feature_selector = RegressionSelector(
features=features,
n_best_per_type=n_best_per_type,
verbose=True, # displays statistics
)
best_features = feature_selector.select(train_set_processed, train_set_processed[target])
best_features
[RegressionSelector] Selected Features
| feature | NanMeasure | ModeMeasure | KruskalMeasure | KruskalRank | TschuprowtFilter | TschuprowtWith | |
|---|---|---|---|---|---|---|---|
| 0 | Quantitative('MedInc') | 0.0000 | 0.3500 | 6037.1821 | 0 | 0.0000 | itself |
| 2 | Quantitative('AveRooms') | 0.0000 | 0.5000 | 1391.5865 | 1 | 0.4015 | MedInc |
| 5 | Quantitative('AveOccup') | 0.0000 | 0.5001 | 991.4083 | 2 | 0.1864 | AveRooms |
| 3 | Quantitative('AveBedrms') | 0.0000 | 0.4500 | 315.7944 | 3 | 0.1392 | MedInc |
| 1 | Quantitative('HouseAge') | 0.0000 | 0.4437 | 163.5278 | 4 | 0.1362 | AveRooms |
| 4 | Quantitative('Population') | 0.0000 | 0.5498 | 16.1097 | 5 | 0.1517 | AveBedrms |
[21]:
Features(['MedInc', 'AveRooms', 'AveOccup', 'AveBedrms', 'HouseAge', 'Population'])
[22]:
train_set_processed[best_features].head()
[22]:
| MedInc | AveRooms | AveOccup | AveBedrms | HouseAge | Population | |
|---|---|---|---|---|---|---|
| 5088 | 0.0 | 0.0 | 1.0 | 2.0 | 0.0 | 1.0 |
| 17096 | 2.0 | 2.0 | 1.0 | 2.0 | 2.0 | 2.0 |
| 5617 | 1.0 | 0.0 | 3.0 | 2.0 | 2.0 | 2.0 |
| 20060 | 0.0 | 0.0 | 3.0 | 1.0 | 1.0 | 2.0 |
| 895 | 2.0 | 0.0 | 1.0 | 2.0 | 0.0 | 3.0 |
Feature
MedIncis the most associated with the targetMedHouseVal:Kruskal-Wallis’ H value is
KruskalMeasure=6037.1821It has 0 % of NaNs (
NanMeasure=0.0000)Its mode represents 35 % of observed data (
ModeMeasure=0.3500)
Feature
AveRoomsis strongly associated to featureMedInc:Tschuprow’s T value is
TschuprowtFilter=0.4015forTschuprowtWith=MedInc
Here, no feature where filtered out for there inter-feature association or over-represented values (no thresholds were set)
Modeling
Fitting model on train data
[23]:
from xgboost import XGBRegressor
model = XGBRegressor()
model.fit(train_set_processed[best_features], train_set_processed[target])
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\IPython\core\formatters.py:974, in MimeBundleFormatter.__call__(self, obj, include, exclude)
971 method = get_real_method(obj, self.print_method)
973 if method is not None:
--> 974 return method(include=include, exclude=exclude)
975 return None
976 else:
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\base.py:469, in BaseEstimator._repr_mimebundle_(self, **kwargs)
467 output = {"text/plain": repr(self)}
468 if get_config()["display"] == "diagram":
--> 469 output["text/html"] = estimator_html_repr(self)
470 return output
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\utils\_estimator_html_repr.py:387, in estimator_html_repr(estimator)
385 else:
386 try:
--> 387 check_is_fitted(estimator)
388 status_label = "<span>Fitted</span>"
389 is_fitted_css_class = "fitted"
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\utils\validation.py:1751, in check_is_fitted(estimator, attributes, msg, all_or_any)
1748 if not hasattr(estimator, "fit"):
1749 raise TypeError("%s is not an estimator instance." % (estimator))
-> 1751 tags = get_tags(estimator)
1753 if not tags.requires_fit and attributes is None:
1754 return
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\utils\_tags.py:430, in get_tags(estimator)
428 for klass in reversed(type(estimator).mro()):
429 if "__sklearn_tags__" in vars(klass):
--> 430 sklearn_tags_provider[klass] = klass.__sklearn_tags__(estimator) # type: ignore[attr-defined]
431 class_order.append(klass)
432 elif "_more_tags" in vars(klass):
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\base.py:613, in RegressorMixin.__sklearn_tags__(self)
612 def __sklearn_tags__(self):
--> 613 tags = super().__sklearn_tags__()
614 tags.estimator_type = "regressor"
615 tags.regressor_tags = RegressorTags()
AttributeError: 'super' object has no attribute '__sklearn_tags__'
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\IPython\core\formatters.py:344, in BaseFormatter.__call__(self, obj)
342 method = get_real_method(obj, self.print_method)
343 if method is not None:
--> 344 return method()
345 return None
346 else:
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\base.py:463, in BaseEstimator._repr_html_inner(self)
458 def _repr_html_inner(self):
459 """This function is returned by the @property `_repr_html_` to make
460 `hasattr(estimator, "_repr_html_") return `True` or `False` depending
461 on `get_config()["display"]`.
462 """
--> 463 return estimator_html_repr(self)
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\utils\_estimator_html_repr.py:387, in estimator_html_repr(estimator)
385 else:
386 try:
--> 387 check_is_fitted(estimator)
388 status_label = "<span>Fitted</span>"
389 is_fitted_css_class = "fitted"
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\utils\validation.py:1751, in check_is_fitted(estimator, attributes, msg, all_or_any)
1748 if not hasattr(estimator, "fit"):
1749 raise TypeError("%s is not an estimator instance." % (estimator))
-> 1751 tags = get_tags(estimator)
1753 if not tags.requires_fit and attributes is None:
1754 return
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\utils\_tags.py:430, in get_tags(estimator)
428 for klass in reversed(type(estimator).mro()):
429 if "__sklearn_tags__" in vars(klass):
--> 430 sklearn_tags_provider[klass] = klass.__sklearn_tags__(estimator) # type: ignore[attr-defined]
431 class_order.append(klass)
432 elif "_more_tags" in vars(klass):
File c:\Users\defra\AppData\Local\pypoetry\Cache\virtualenvs\autocarver-i96ERKJw-py3.9\lib\site-packages\sklearn\base.py:613, in RegressorMixin.__sklearn_tags__(self)
612 def __sklearn_tags__(self):
--> 613 tags = super().__sklearn_tags__()
614 tags.estimator_type = "regressor"
615 tags.regressor_tags = RegressorTags()
AttributeError: 'super' object has no attribute '__sklearn_tags__'
[23]:
XGBRegressor(base_score=None, booster=None, callbacks=None,
colsample_bylevel=None, colsample_bynode=None,
colsample_bytree=None, device=None, early_stopping_rounds=None,
enable_categorical=False, eval_metric=None, feature_types=None,
gamma=None, grow_policy=None, importance_type=None,
interaction_constraints=None, learning_rate=None, max_bin=None,
max_cat_threshold=None, max_cat_to_onehot=None,
max_delta_step=None, max_depth=None, max_leaves=None,
min_child_weight=None, missing=nan, monotone_constraints=None,
multi_strategy=None, n_estimators=None, n_jobs=None,
num_parallel_tree=None, random_state=None, ...)
Saving model
[24]:
model.save_model("regression_xgboost.json")
Prediction on dev dataset and performance
[25]:
from sklearn.metrics import root_mean_squared_error
dev_pred = model.predict(dev_set_processed[best_features])
root_mean_squared_error(dev_set_processed[target], dev_pred)
[25]:
0.7773564029114313
What’s next?
Thanks to Carvers all of your features are now optimally processed for your regression task!
As a final step towards your model, Selectors can prove to be handy tools to operate target optimal Data Pre-Selection, so make sure to check out Selectors Examples!
Well done!
Your commitment to achieving optimal results in continuous regression tasks shines through in your meticulous use of AutoCarver’s ContinuousCarver for data preprocessing. By fine-tuning and optimizing your dataset, you have set the stage for robust and accurate machine learning models.
The ContinuousCarver has proven to be a valuable ally in your pursuit of excellence, carving out a path toward enhanced feature representation and model interpretability. Your dedication to refining the data preprocessing steps reflects a commitment to extracting the maximum value from your datasets.
We extend our sincere appreciation for choosing AutoCarver as your companion in the data preprocessing journey. Your use of AutoCarver demonstrates a dedication to leveraging cutting-edge tools for achieving excellence in continuous regression tasks.
As you transition to the modeling phase, may the carefully crafted features and preprocessing steps contribute to the success of your predictive models. We’re excited to see the impact of your work and are grateful for the opportunity to be part of your data science endeavors.
Thank you for trusting AutoCarver, and we wish you continued success in your data-driven ventures.