{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Setting things up\n", "\n", "## About this notebook\n", "\n", "In this notebook, we embark on a journey to elevate the predictive capabilities of the California Housing Prices Dataset through advanced preprocessing using the ``ContinuousCarver`` pipeline. Renowned for its association-maximizing discretization, ``ContinuousCarver`` is a powerful Python tool designed to handle diverse data types—whether they be quantitative or qualitative. Our specific goal is to prepare the dataset for continuous regression tasks, such as predicting housing prices.\n", "\n", "The California Housing Prices Dataset is a treasure trove of features, encompassing information on factors like square footage, bedrooms, location, and more. By employing ``ContinuousCarver``, we aim to seamlessly discretize both quantitative and qualitative features, tailoring them for optimal representation in our continuous regression models.\n", "\n", "Throughout this notebook, we'll explore the intricacies of ``ContinuousCarver``'s discretization pipeline, witnessing its adaptability to a variety of data types. Whether it involves transforming square footage or encoding location information, ``ContinuousCarver`` ensures that each feature is finely tuned for our regression tasks.\n", "\n", "Join us in this exploration as we leverage the power of ``ContinuousCarver`` to preprocess the California Housing Prices Dataset. Through effective feature engineering and discretization, our aim is to create a dataset that captures the nuanced relationships within the housing market, setting the stage for the development of accurate and impactful continuous regression models.\n", "\n", "Let's dive in and uncover the potential of ``ContinuousCarver`` in transforming the California Housing Prices Dataset for optimal predictive modeling.\n", "\n", "\n", "## Installation" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# %pip install AutoCarver[jupyter]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Califorinia Housing Prices Data\n", "\n", "In this example notebook, we will use the California Housing Prices dataset.\n", "\n", "The California Housing Prices dataset is a well-known dataset in the field of machine learning and statistics. It provides information about housing districts in California and is frequently used for regression analysis and predictive modeling tasks.\n", "\n", "Comprising housing-related metrics for various districts in California, such as median house value, median income, housing median age, average rooms, average bedrooms, population, households, and more, the California Housing Prices dataset is a valuable resource for exploring the relationships between different features and predicting the median house values (continuous regression)." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncHouseAgeAveRoomsAveBedrmsPopulationAveOccupLatitudeLongitudeMedHouseVal
08.325241.06.9841271.023810322.02.55555637.88-122.234.526
18.301421.06.2381370.9718802401.02.10984237.86-122.223.585
27.257452.08.2881361.073446496.02.80226037.85-122.243.521
35.643152.05.8173521.073059558.02.54794537.85-122.253.413
43.846252.06.2818531.081081565.02.18146737.85-122.253.422
\n", "
" ], "text/plain": [ " MedInc HouseAge AveRooms AveBedrms Population AveOccup Latitude \\\n", "0 8.3252 41.0 6.984127 1.023810 322.0 2.555556 37.88 \n", "1 8.3014 21.0 6.238137 0.971880 2401.0 2.109842 37.86 \n", "2 7.2574 52.0 8.288136 1.073446 496.0 2.802260 37.85 \n", "3 5.6431 52.0 5.817352 1.073059 558.0 2.547945 37.85 \n", "4 3.8462 52.0 6.281853 1.081081 565.0 2.181467 37.85 \n", "\n", " Longitude MedHouseVal \n", "0 -122.23 4.526 \n", "1 -122.22 3.585 \n", "2 -122.24 3.521 \n", "3 -122.25 3.413 \n", "4 -122.25 3.422 " ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn import datasets\n", "\n", "# Load dataset directly from sklearn\n", "housing = datasets.fetch_california_housing(as_frame=True)\n", "\n", "# conversion to pandas\n", "housing_data = housing[\"data\"]\n", "housing_data[housing[\"target_names\"][0]] = housing[\"target\"]\n", "\n", "# Display the first few rows of the dataset\n", "housing_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Target type and Carver selection" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 20640.000000\n", "mean 2.068558\n", "std 1.153956\n", "min 0.149990\n", "25% 1.196000\n", "50% 1.797000\n", "75% 2.647250\n", "max 5.000010\n", "Name: MedHouseVal, dtype: float64" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "target = \"MedHouseVal\"\n", "\n", "housing_data[target].describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The target ``\"MedHouseVal\"`` is a continuous target of type ``float64`` used in a regression task. Hence we will use ``AutoCarver.ContinuousCarver`` and ``AutoCarver.selectors.RegressionSelector`` in following code blocks." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Sampling" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(2.0666362048018514, 2.072459655020552)" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "# stratified sampling by target\n", "train_set, dev_set = train_test_split(housing_data, test_size=0.33, random_state=42)\n", "\n", "# checking target rate per dataset\n", "train_set[target].mean(), dev_set[target].mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Picking up columns to Carve" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncHouseAgeAveRoomsAveBedrmsPopulationAveOccupLatitudeLongitudeMedHouseVal
50880.980919.03.1877261.129964726.02.62093933.98-118.281.214
170964.223233.06.1896961.0866511015.02.37704937.46-122.233.637
56173.548842.04.8215771.0954361044.04.33195033.79-118.262.056
200601.646924.04.2741941.0483871686.04.53225835.87-119.260.476
8953.990914.04.6083031.0893502738.02.47111937.54-121.962.360
\n", "
" ], "text/plain": [ " MedInc HouseAge AveRooms AveBedrms Population AveOccup Latitude \\\n", "5088 0.9809 19.0 3.187726 1.129964 726.0 2.620939 33.98 \n", "17096 4.2232 33.0 6.189696 1.086651 1015.0 2.377049 37.46 \n", "5617 3.5488 42.0 4.821577 1.095436 1044.0 4.331950 33.79 \n", "20060 1.6469 24.0 4.274194 1.048387 1686.0 4.532258 35.87 \n", "895 3.9909 14.0 4.608303 1.089350 2738.0 2.471119 37.54 \n", "\n", " Longitude MedHouseVal \n", "5088 -118.28 1.214 \n", "17096 -122.23 3.637 \n", "5617 -118.26 2.056 \n", "20060 -119.26 0.476 \n", "895 -121.96 2.360 " ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_set.head()" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "MedInc float64\n", "HouseAge float64\n", "AveRooms float64\n", "AveBedrms float64\n", "Population float64\n", "AveOccup float64\n", "Latitude float64\n", "Longitude float64\n", "MedHouseVal float64\n", "dtype: object" ] }, "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# column data types\n", "train_set.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All features are quantitative continuous features at the exception of ``Latitude`` and ``Longitude`` which are geographical featues (not supported by ``AutoCarver`` as is). All other features will be added to the list of ``quantitative_features``." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "from AutoCarver import Features\n", "\n", "# lists of features per data type\n", "features = Features(quantitatives=[\"MedInc\", \"HouseAge\", \"AveRooms\", \"AveBedrms\", \"Population\", \"AveOccup\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Using AutoCarver" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## AutoCarver settings\n", "\n", "### Representativness of modalities\n", "\n", "The attribute ``min_freq`` allows one to choose the minimum frequency per basic modalities. It is used:\n", "\n", "- For quantitative features, to define the number of quantiles to initialy discretize the features with.\n", "\n", "- For qualitative features, to define the threshold under which a modality is grouped to either a default value or its closest modality." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "min_freq = 0.1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Tip:** should be set between ``0.01`` (slower, preciser, less robust) and ``0.2`` (faster, more robust)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Desired number of modalities\n", "\n", "The attribute ``max_n_mod`` allows one to choose the maximum number of modalities per carved feature. It is used by **Carvers** has the upper limit of number of modalities per consecutive combination of modalities." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "max_n_mod = 4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Tip:** should be set between ``3`` (faster, more robust) and ``7`` (slower, preciser, less robust)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Grouping NaNs\n", "\n", "The attribute ``dropna`` allows one to choose whether or not ``nan`` should be grouped with another modality. If set to ``True``, **Carvers** will first find the most suitable combination of non-``nan`` values, and then test out all possible combinations with ``nan``." ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "dropna = False # anyway, there are no nan in this dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Type of output carved features\n", "\n", "The attribute ``ordinal_encoding`` allows one to choose the output type:\n", "\n", "* Use ``True`` for integer output of ranked modalities (default)\n", "* Use ``False`` for string output of modalities" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "ordinal_encoding = True" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fitting AutoCarver\n", "\n", "* First, all quantitative features are discretized:\n", " 1. Using ``ContinuousDiscretizer`` for quantile discretization that keeps track of over-represented values (more frequent than ``min_freq``)\n", " 2. Using ``OrdinalDiscretizer`` for any remaining under-represented values (less frequent than ``min_freq/2``) to be grouped with its closest modality\n", "\n", "* Second, all features are carved following this recipe, for all classes of ``train_set[target]`` (except one):\n", " 1. The raw distribution is printed out on provided ``train_set`` and ``dev_set``. It's the output of the discretization step\n", " 2. Grouping modalities: all consecutive combinations of modalities are applied to ``train_set``\n", " 3. Computing associations: the association metric (Krsuskal-Wallis' statistic, by default) is computed with the provided ``train_set[target]``\n", " 4. Combinations are sorted in descending order by association value\n", " 5. Testing robustness: finds the first combination that checks the following:\n", " - Representativness of modalities on ``train_set`` and ``dev_set`` (all should be more frequent than ``min_freq/2``)\n", " - Distinct target rates per consecutive modalities on ``train_set`` and ``dev_set`` \n", " - No inversion of target rates between ``train_set`` and ``dev_set`` (same ordering of modalities by target rate)\n", " 6. (Optional) If requested via ``dropna=True``, and if any, all combinations of modalities with ``nan`` are applied to ``train_set`` and steps 3. and 4. are run\n", " 7. The carved distribution is printed out on provided ``train_set`` and ``dev_set``. It's the output of the carving step" ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "------\n", "--- [QuantitativeDiscretizer] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])\n", " - [ContinuousDiscretizer] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])\n", " - [OrdinalDiscretizer] Fit Features(['HouseAge'])\n", "------\n", "\n", "---------\n", "------ [ContinuousCarver] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])\n", "--- [ContinuousCarver] Fit Quantitative('MedInc') (1/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 1.60e+001.11020.0500
1.60e+00 < x <= 1.91e+001.12850.0500
1.91e+00 < x <= 2.15e+001.21980.0500
2.15e+00 < x <= 2.35e+001.31710.0500
2.35e+00 < x <= 2.57e+001.38170.0500
2.57e+00 < x <= 2.74e+001.54090.0500
2.74e+00 < x <= 2.98e+001.61590.0500
2.98e+00 < x <= 3.14e+001.69060.0499
3.14e+00 < x <= 3.32e+001.82320.0500
3.32e+00 < x <= 3.54e+001.90590.0500
3.54e+00 < x <= 3.73e+002.00760.0502
3.73e+00 < x <= 3.97e+002.02710.0498
3.97e+00 < x <= 4.18e+002.14560.0500
4.18e+00 < x <= 4.46e+002.24330.0500
4.46e+00 < x <= 4.76e+002.36210.0501
4.76e+00 < x <= 5.12e+002.39860.0499
5.12e+00 < x <= 5.54e+002.64380.0500
5.54e+00 < x <= 6.16e+002.93240.0500
6.16e+00 < x <= 7.32e+003.45920.0500
7.32e+00 < x4.37840.0500
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
1.10170.0509
1.04100.0502
1.24070.0501
1.29190.0506
1.46760.0536
1.56050.0417
1.62800.0584
1.75190.0471
1.84430.0504
1.85000.0498
2.00400.0533
2.08900.0502
2.16410.0505
2.27000.0540
2.37680.0439
2.50870.0479
2.68140.0483
2.98050.0479
3.37480.0530
4.37480.0483
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1300.95it/s]\n", "Computing associations: 100%|██████████| 1159/1159 [00:04<00:00, 280.31it/s]\n", "Testing robustness : 0%| | 0/1159 [00:00\n", "#T_78027_row0_col0, #T_78027_row3_col1 {\n", " background-color: #3b4cc0;\n", " color: #f1f1f1;\n", "}\n", "#T_78027_row0_col1, #T_78027_row2_col1 {\n", " background-color: #dcdddd;\n", " color: #000000;\n", "}\n", "#T_78027_row1_col0 {\n", " background-color: #89acfd;\n", " color: #000000;\n", "}\n", "#T_78027_row1_col1, #T_78027_row3_col0 {\n", " background-color: #b40426;\n", " color: #f1f1f1;\n", "}\n", "#T_78027_row2_col0 {\n", " background-color: #d7dce3;\n", " color: #000000;\n", "}\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 2.57e+001.23140.2500
2.57e+00 < x <= 3.97e+001.80160.3500
3.97e+00 < x <= 5.54e+002.35870.2499
5.54e+00 < x3.59000.1501
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
1.23150.2554
1.82220.3509
2.39530.2446
3.57210.1491
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('HouseAge') (2/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 8.00e+002.11580.0537
8.00e+00 < x <= 1.20e+011.82200.0477
1.20e+01 < x <= 1.50e+011.85900.0613
1.50e+01 < x <= 1.60e+012.03580.0393
1.60e+01 < x <= 1.80e+011.90130.0596
1.80e+01 < x <= 2.00e+011.93990.0468
2.00e+01 < x <= 2.20e+012.01340.0404
2.20e+01 < x <= 2.50e+012.10550.0705
2.50e+01 < x <= 2.60e+012.09770.0300
2.60e+01 < x <= 2.80e+012.02180.0475
2.80e+01 < x <= 3.10e+012.04390.0682
3.10e+01 < x <= 3.30e+012.02750.0575
3.30e+01 < x <= 3.40e+012.11890.0328
3.40e+01 < x <= 3.50e+012.02040.0395
3.50e+01 < x <= 3.70e+012.07500.0687
3.70e+01 < x <= 3.90e+012.02120.0361
3.90e+01 < x <= 4.20e+012.00130.0450
4.20e+01 < x <= 4.50e+012.13010.0485
4.50e+01 < x2.47850.1072
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
2.02050.0526
1.78270.0443
1.87800.0556
1.92080.0335
1.94840.0652
1.95170.0470
2.11410.0421
2.11790.0759
2.08880.0299
2.21380.0443
1.95460.0664
2.05120.0565
2.19790.0346
2.17620.0408
2.07470.0659
1.98850.0388
2.03940.0508
2.00150.0489
2.46510.1069
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Grouping modalities : 100%|█████████▉| 986/987 [00:00<00:00, 1312.66it/s]\n", "Computing associations: 100%|██████████| 987/987 [00:03<00:00, 288.78it/s]\n", "Testing robustness : 1%| | 6/987 [00:00<00:02, 335.58it/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 2.20e+011.94940.3486
2.20e+01 < x <= 2.60e+012.10320.1005
2.60e+01 < x <= 4.50e+012.05090.4437
4.50e+01 < x2.47850.1072
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
1.94470.3403
2.10970.1058
2.06700.4470
2.46510.1069
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('AveRooms') (3/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 3.44e+001.91260.0500
3.44e+00 < x <= 3.79e+001.82860.0500
3.79e+00 < x <= 4.06e+001.81690.0500
4.06e+00 < x <= 4.28e+001.84180.0500
4.28e+00 < x <= 4.46e+001.75290.0500
4.46e+00 < x <= 4.62e+001.79150.0500
4.62e+00 < x <= 4.79e+001.82140.0500
4.79e+00 < x <= 4.94e+001.76850.0500
4.94e+00 < x <= 5.09e+001.74660.0500
5.09e+00 < x <= 5.23e+001.77170.0500
5.23e+00 < x <= 5.38e+001.86640.0500
5.38e+00 < x <= 5.53e+001.84720.0500
5.53e+00 < x <= 5.69e+001.91990.0500
5.69e+00 < x <= 5.86e+001.99100.0500
5.86e+00 < x <= 6.06e+002.08700.0500
6.06e+00 < x <= 6.27e+002.19080.0500
6.27e+00 < x <= 6.54e+002.40500.0500
6.54e+00 < x <= 6.95e+002.68740.0500
6.95e+00 < x <= 7.65e+003.11290.0500
7.65e+00 < x3.17180.0500
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
1.86590.0518
1.87280.0505
1.76270.0524
1.80200.0543
1.72230.0552
1.68020.0452
1.77070.0530
1.80300.0443
1.82090.0523
1.83260.0437
1.79230.0550
1.93880.0514
1.94650.0501
2.02480.0468
2.10490.0483
2.22390.0490
2.43390.0467
2.76670.0468
3.10010.0548
3.24290.0483
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1378.37it/s]\n", "Computing associations: 100%|██████████| 1159/1159 [00:03<00:00, 289.89it/s]\n", "Testing robustness : 1%| | 7/1159 [00:00<00:03, 304.14it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " [ContinuousCarver] Carved distribution\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 5.23e+001.80530.5000
5.23e+00 < x <= 5.86e+001.90610.2000
5.86e+00 < x <= 6.54e+002.22750.1500
6.54e+00 < x2.99070.1501
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
1.79330.5028
1.92080.2033
2.25210.1440
3.04200.1499
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('AveBedrms') (4/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 9.4000e-012.06840.0500
9.4000e-01 < x <= 9.6724e-012.07350.0500
9.6724e-01 < x <= 9.8319e-012.21670.0501
9.8319e-01 < x <= 9.9576e-012.17060.0499
9.9576e-01 < x <= 1.0066e+002.13100.0500
1.0066e+00 < x <= 1.0154e+002.23580.0500
1.0154e+00 < x <= 1.0247e+002.16680.0500
1.0247e+00 < x <= 1.0331e+002.21020.0500
1.0331e+00 < x <= 1.0412e+002.12950.0500
1.0412e+00 < x <= 1.0495e+002.15480.0500
1.0495e+00 < x <= 1.0576e+002.12380.0500
1.0576e+00 < x <= 1.0665e+002.10250.0500
1.0665e+00 < x <= 1.0768e+002.07040.0500
1.0768e+00 < x <= 1.0878e+002.06640.0501
1.0878e+00 < x <= 1.1003e+002.11180.0499
1.1003e+00 < x <= 1.1161e+001.99370.0500
1.1161e+00 < x <= 1.1382e+001.94050.0500
1.1382e+00 < x <= 1.1738e+001.79900.0500
1.1738e+00 < x <= 1.2732e+001.91620.0500
1.2732e+00 < x1.65150.0500
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
2.04160.0539
2.20430.0527
2.09970.0482
2.18350.0487
2.26280.0552
2.16190.0480
2.22950.0567
2.16900.0493
2.15810.0528
2.12020.0476
2.10390.0452
2.15950.0509
2.10370.0521
2.06620.0484
2.04870.0489
1.95430.0467
1.88710.0484
1.86800.0499
1.83710.0465
1.71820.0498
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1278.14it/s]\n", "Computing associations: 100%|██████████| 1159/1159 [00:04<00:00, 277.88it/s]\n", "Testing robustness : 3%|▎ | 35/1159 [00:00<00:02, 512.40it/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 9.672e-012.07090.1000
9.672e-01 < x <= 1.058e+002.17100.4500
1.058e+00 < x <= 1.138e+002.04750.2999
1.138e+00 < x1.78880.1501
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
2.12210.1066
2.16850.4517
2.03900.2955
1.80720.1462
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('Population') (5/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 3.53e+021.98590.0501
3.53e+02 < x <= 5.14e+022.16160.0501
5.14e+02 < x <= 6.27e+022.11170.0501
6.27e+02 < x <= 7.15e+022.28190.0497
7.15e+02 < x <= 7.93e+022.03350.0509
7.93e+02 < x <= 8.64e+022.21130.0492
8.64e+02 < x <= 9.38e+022.07720.0498
9.38e+02 < x <= 1.02e+032.13860.0500
1.02e+03 < x <= 1.09e+032.04300.0503
1.09e+03 < x <= 1.17e+032.05060.0496
1.17e+03 < x <= 1.26e+032.08700.0505
1.26e+03 < x <= 1.35e+032.01950.0497
1.35e+03 < x <= 1.46e+032.00040.0502
1.46e+03 < x <= 1.58e+032.11020.0498
1.58e+03 < x <= 1.73e+032.03460.0500
1.73e+03 < x <= 1.91e+031.91390.0499
1.91e+03 < x <= 2.15e+032.00060.0500
2.15e+03 < x <= 2.56e+032.07070.0500
2.56e+03 < x <= 3.30e+031.96140.0500
3.30e+03 < x2.04280.0500
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
1.90120.0530
2.19150.0520
2.17060.0523
2.10620.0514
2.20190.0531
2.17650.0490
2.20250.0506
2.13290.0553
2.17440.0437
2.13190.0480
1.99390.0534
2.00960.0465
1.95690.0465
1.97560.0504
2.08150.0496
2.02720.0461
1.97890.0487
1.93550.0496
2.07140.0518
2.01570.0487
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Grouping modalities : 100%|█████████▉| 1158/1159 [00:00<00:00, 1364.07it/s]\n", "Computing associations: 100%|██████████| 1159/1159 [00:03<00:00, 294.36it/s]\n", "Testing robustness : 16%|█▋ | 191/1159 [00:00<00:01, 626.53it/s]" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " [ContinuousCarver] Carved distribution\n" ] }, { "name": "stderr", "output_type": "stream", "text": [ "\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 6.27e+022.08640.1503
6.27e+02 < x <= 8.64e+022.17430.1498
8.64e+02 < x <= 2.15e+032.04330.5498
2.15e+03 < x2.02500.1501
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
2.08670.1572
2.16180.1536
2.06070.5390
2.00840.1502
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('AveOccup') (6/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 1.870e+002.71220.0500
1.870e+00 < x <= 2.067e+002.66330.0500
2.067e+00 < x <= 2.225e+002.33730.0500
2.225e+00 < x <= 2.338e+002.30800.0500
2.338e+00 < x <= 2.432e+002.19760.0500
2.432e+00 < x <= 2.513e+002.20640.0500
2.513e+00 < x <= 2.595e+002.17360.0500
2.595e+00 < x <= 2.668e+002.18620.0500
2.668e+00 < x <= 2.743e+002.13780.0500
2.743e+00 < x <= 2.820e+002.19020.0500
2.820e+00 < x <= 2.898e+002.18240.0500
2.898e+00 < x <= 2.984e+002.07410.0500
2.984e+00 < x <= 3.073e+002.02550.0501
3.073e+00 < x <= 3.171e+001.99140.0498
3.171e+00 < x <= 3.282e+001.89920.0500
3.282e+00 < x <= 3.425e+001.89260.0500
3.425e+00 < x <= 3.607e+001.70850.0500
3.607e+00 < x <= 3.877e+001.56660.0500
3.877e+00 < x <= 4.325e+001.45050.0500
4.325e+00 < x1.42940.0500
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
2.76840.0484
2.53340.0435
2.39890.0542
2.36410.0533
2.22720.0546
2.29690.0489
2.31790.0508
2.07930.0467
2.18470.0521
2.17520.0504
2.07620.0533
2.05350.0501
2.05350.0528
1.94770.0458
1.83970.0449
1.88610.0514
1.73010.0448
1.62000.0499
1.44230.0527
1.45960.0515
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stderr", "output_type": "stream", "text": [ "Grouping modalities : 100%|█████████▉| 1158/1159 [00:01<00:00, 1152.95it/s]\n", "Computing associations: 100%|██████████| 1159/1159 [00:03<00:00, 297.60it/s]\n", "Testing robustness : 0%| | 3/1159 [00:00<00:06, 184.97it/s]\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "\n", " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequency
x <= 2.22e+002.57090.1501
2.22e+00 < x <= 3.07e+002.16810.5001
3.07e+00 < x <= 3.61e+001.87290.1998
3.61e+00 < x1.48220.1501
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequency
2.56150.1461
2.18360.5129
1.85270.1869
1.50560.1541
\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from AutoCarver import ContinuousCarver\n", "\n", "# intiating AutoCarver\n", "auto_carver = ContinuousCarver(\n", " features=features,\n", " min_freq=min_freq,\n", " max_n_mod=max_n_mod,\n", " dropna=dropna,\n", " ordinal_encoding=ordinal_encoding,\n", " verbose=True, # showing statistics\n", " copy=True, # whether or not to return a copy of the input dataset\n", ")\n", "\n", "# fitting on training sample, a dev sample can be specified to evaluate carving robustness\n", "train_set_processed = auto_carver.fit_transform(train_set, train_set[target], X_dev=dev_set, y_dev=dev_set[target])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## AutoCarver analysis\n", "\n", "### Carving Summary" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
contentfrequency
featuretarget_meankruskaln_modlabel
Quantitative('MedInc')1.2314216037.18213540x <= 2.57e+000.250000
1.8015626037.182135412.57e+00 < x <= 3.97e+000.350014
2.3586606037.182135423.97e+00 < x <= 5.54e+000.249928
3.5900406037.182135435.54e+00 < x0.150058
Quantitative('HouseAge')1.949361163.52784140x <= 2.20e+010.348568
2.103173163.527841412.20e+01 < x <= 2.60e+010.100521
2.050927163.527841422.60e+01 < x <= 4.50e+010.443665
2.478542163.527841434.50e+01 < x0.107246
Quantitative('AveRooms')1.8052551391.58648940x <= 5.23e+000.500000
1.9060981391.586489415.23e+00 < x <= 5.86e+000.199957
2.2275311391.586489425.86e+00 < x <= 6.54e+000.149986
2.9906761391.586489436.54e+00 < x0.150058
Quantitative('AveBedrms')2.070937315.79435040x <= 9.672e-010.100014
2.171033315.794350419.672e-01 < x <= 1.058e+000.450029
2.047547315.794350421.058e+00 < x <= 1.138e+000.299899
1.788831315.794350431.138e+00 < x0.150058
Quantitative('Population')2.08639416.10970940x <= 6.27e+020.150347
2.17429716.109709416.27e+02 < x <= 8.64e+020.149841
2.04325516.109709428.64e+02 < x <= 2.15e+030.549754
2.02499516.109709432.15e+03 < x0.150058
Quantitative('AveOccup')2.570888991.40830140x <= 2.22e+000.150058
2.168126991.408301412.22e+00 < x <= 3.07e+000.500072
1.872867991.408301423.07e+00 < x <= 3.61e+000.199812
1.482183991.408301433.61e+00 < x0.150058
\n", "
" ], "text/plain": [ " content \\\n", "feature target_mean kruskal n_mod label \n", "Quantitative('MedInc') 1.231421 6037.182135 4 0 x <= 2.57e+00 \n", " 1.801562 6037.182135 4 1 2.57e+00 < x <= 3.97e+00 \n", " 2.358660 6037.182135 4 2 3.97e+00 < x <= 5.54e+00 \n", " 3.590040 6037.182135 4 3 5.54e+00 < x \n", "Quantitative('HouseAge') 1.949361 163.527841 4 0 x <= 2.20e+01 \n", " 2.103173 163.527841 4 1 2.20e+01 < x <= 2.60e+01 \n", " 2.050927 163.527841 4 2 2.60e+01 < x <= 4.50e+01 \n", " 2.478542 163.527841 4 3 4.50e+01 < x \n", "Quantitative('AveRooms') 1.805255 1391.586489 4 0 x <= 5.23e+00 \n", " 1.906098 1391.586489 4 1 5.23e+00 < x <= 5.86e+00 \n", " 2.227531 1391.586489 4 2 5.86e+00 < x <= 6.54e+00 \n", " 2.990676 1391.586489 4 3 6.54e+00 < x \n", "Quantitative('AveBedrms') 2.070937 315.794350 4 0 x <= 9.672e-01 \n", " 2.171033 315.794350 4 1 9.672e-01 < x <= 1.058e+00 \n", " 2.047547 315.794350 4 2 1.058e+00 < x <= 1.138e+00 \n", " 1.788831 315.794350 4 3 1.138e+00 < x \n", "Quantitative('Population') 2.086394 16.109709 4 0 x <= 6.27e+02 \n", " 2.174297 16.109709 4 1 6.27e+02 < x <= 8.64e+02 \n", " 2.043255 16.109709 4 2 8.64e+02 < x <= 2.15e+03 \n", " 2.024995 16.109709 4 3 2.15e+03 < x \n", "Quantitative('AveOccup') 2.570888 991.408301 4 0 x <= 2.22e+00 \n", " 2.168126 991.408301 4 1 2.22e+00 < x <= 3.07e+00 \n", " 1.872867 991.408301 4 2 3.07e+00 < x <= 3.61e+00 \n", " 1.482183 991.408301 4 3 3.61e+00 < x \n", "\n", " frequency \n", "feature target_mean kruskal n_mod label \n", "Quantitative('MedInc') 1.231421 6037.182135 4 0 0.250000 \n", " 1.801562 6037.182135 4 1 0.350014 \n", " 2.358660 6037.182135 4 2 0.249928 \n", " 3.590040 6037.182135 4 3 0.150058 \n", "Quantitative('HouseAge') 1.949361 163.527841 4 0 0.348568 \n", " 2.103173 163.527841 4 1 0.100521 \n", " 2.050927 163.527841 4 2 0.443665 \n", " 2.478542 163.527841 4 3 0.107246 \n", "Quantitative('AveRooms') 1.805255 1391.586489 4 0 0.500000 \n", " 1.906098 1391.586489 4 1 0.199957 \n", " 2.227531 1391.586489 4 2 0.149986 \n", " 2.990676 1391.586489 4 3 0.150058 \n", "Quantitative('AveBedrms') 2.070937 315.794350 4 0 0.100014 \n", " 2.171033 315.794350 4 1 0.450029 \n", " 2.047547 315.794350 4 2 0.299899 \n", " 1.788831 315.794350 4 3 0.150058 \n", "Quantitative('Population') 2.086394 16.109709 4 0 0.150347 \n", " 2.174297 16.109709 4 1 0.149841 \n", " 2.043255 16.109709 4 2 0.549754 \n", " 2.024995 16.109709 4 3 0.150058 \n", "Quantitative('AveOccup') 2.570888 991.408301 4 0 0.150058 \n", " 2.168126 991.408301 4 1 0.500072 \n", " 1.872867 991.408301 4 2 0.199812 \n", " 1.482183 991.408301 4 3 0.150058 " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "auto_carver.summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* As requested with ``ordinal_encoding=True``, output labels are integers of modalities\n", "\n", "* For quantitative feature ``Population``, the selected combination of modalities groups populations as follows:\n", " * label ``0``: lower or equal to 630 people (``content=\"x <= 6.3e+02\"``)\n", " * label ``1``: greater than 630 people and lower or equal to 860 people (``content=\"6.3e+02 < x <= 8.6e+02\"``)\n", " * label ``2``: greater than 860 people and lower or equal to 2200 people (``content=\"8.6e+02 < x <= 2.2e+03\"``)\n", " * label ``3``: higher than 2200 people (``content=\"2.2e+03 < x\"``)\n", "\n", "### Detailed overview of tested combinations" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
infokruskalcombinationn_moddropnatrainviabledev
0Raw distribution (n_mod=20>max_n_mod=4)1062.072498{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...20FalseNaNNaNNaN
1Not viable994.514410{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}False{'viable': False, 'info': 'Non-representative ...
2Not viable994.504665{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}False{'viable': False, 'info': 'Non-representative ...
3Not viable991.504255{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}False{'viable': False, 'info': 'Non-representative ...
4Best for kruskal and max_n_mod=4991.408301{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}True{'viable': True, 'info': ''}
5Not checked991.308986{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4FalseNaNNaNNaN
6Not checked988.666983{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4FalseNaNNaNNaN
\n", "
" ], "text/plain": [ " info kruskal \\\n", "0 Raw distribution (n_mod=20>max_n_mod=4) 1062.072498 \n", "1 Not viable 994.514410 \n", "2 Not viable 994.504665 \n", "3 Not viable 991.504255 \n", "4 Best for kruskal and max_n_mod=4 991.408301 \n", "5 Not checked 991.308986 \n", "6 Not checked 988.666983 \n", "\n", " combination n_mod dropna \\\n", "0 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 20 False \n", "1 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "2 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "3 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "4 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "5 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "6 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "\n", " train viable \\\n", "0 NaN NaN \n", "1 {'viable': True, 'info': ''} False \n", "2 {'viable': True, 'info': ''} False \n", "3 {'viable': True, 'info': ''} False \n", "4 {'viable': True, 'info': ''} True \n", "5 NaN NaN \n", "6 NaN NaN \n", "\n", " dev \n", "0 NaN \n", "1 {'viable': False, 'info': 'Non-representative ... \n", "2 {'viable': False, 'info': 'Non-representative ... \n", "3 {'viable': False, 'info': 'Non-representative ... \n", "4 {'viable': True, 'info': ''} \n", "5 NaN \n", "6 NaN " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"AveOccup\"].history.head(7)" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'viable': False, 'info': 'Non-representative modality for min_freq=10.00%'}" ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"AveOccup\"].history.dev[1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* The most associated combination of feature ``AveOccup`` (the first tested out, where ``info!=\"Raw distribution\"``) did not pass the viability tests. When looking in ``history.dev``:\n", " * ``\"Non-representative modality for min_freq=10.00%\"``: tells us that a modality is unstable between ``train_set`` and ``dev_set``\n", "\n", "* For feature feature ``AveOccup``, the 4th combination is the first to pass tests:\n", " - ``viabe=True``\n", " - ``info=\"Best for kruskal and max_n_mod=4\"``\n", " - Kruskal-Wallis' H with ``MedHouseVal`` is ``991.408301`` for this combination\n", " - Following combinations (less associated with the target) where not tested: ``info=\"Not checked\"``\n", "\n", "* For all combinations ``dropna=False`` means that it is not a combination in which ``nan``s are being grouped with other modalities (as requested with ``dropna=False``)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Saving and Loading AutoCarver" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Saving\n", "\n", "All **Carvers** can safely be stored as a .json file." ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [], "source": [ "auto_carver.save(\"continuous_carver.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Loading\n", "\n", "**Carvers** can safely be loaded from a .json file." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "from AutoCarver import ContinuousCarver\n", "\n", "# loading json file\n", "auto_carver = ContinuousCarver.load('continuous_carver.json')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Applying AutoCarver" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [], "source": [ "dev_set_processed = auto_carver.transform(dev_set)" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncHouseAgeAveRoomsAveBedrmsPopulationAveOccup
0.00.2554320.3402820.5027890.1065770.1572230.146066
1.00.3508510.1058430.2033180.4517030.1535530.512918
2.00.2445680.4470050.1440110.2955080.5390490.186876
3.00.1491490.1068700.1498830.1462130.1501760.154140
\n", "
" ], "text/plain": [ " MedInc HouseAge AveRooms AveBedrms Population AveOccup\n", "0.0 0.255432 0.340282 0.502789 0.106577 0.157223 0.146066\n", "1.0 0.350851 0.105843 0.203318 0.451703 0.153553 0.512918\n", "2.0 0.244568 0.447005 0.144011 0.295508 0.539049 0.186876\n", "3.0 0.149149 0.106870 0.149883 0.146213 0.150176 0.154140" ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dev_set_processed[auto_carver.features].apply(lambda u: u.value_counts(dropna=False, normalize=True))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Feature Selection\n", "## Selectors settings\n", "### Features to select from\n", "\n", "Here all features have been carved using ``ContinuousCarver``, hence all features are qualitative.\n", "\n", "### Number of features to select\n", "\n", "The attribute ``n_best_per_type`` allows one to choose the number of features to be selected per data type (quantitative and qualitative)." ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [], "source": [ "n_best_per_type = 6 " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using Selectors" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " [RegressionSelector] Selected Qualitative Features \n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 featureNanModeKruskalMeasureKruskalRankTschuprowtFilterTschuprowtWith
0Quantitative('MedInc')0.00000.35006037.18210.00000.0000itself
2Quantitative('AveRooms')0.00000.50001391.58651.00000.4015MedInc
5Quantitative('AveOccup')0.00000.5001991.40832.00000.1864AveRooms
3Quantitative('AveBedrms')0.00000.4500315.79443.00000.1392MedInc
1Quantitative('HouseAge')0.00000.4437163.52784.00000.1362AveRooms
4Quantitative('Population')0.00000.549816.10975.00000.1517AveBedrms
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "Features(['MedInc', 'AveRooms', 'AveOccup', 'AveBedrms', 'HouseAge', 'Population'])" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from AutoCarver.selectors import RegressionSelector\n", "\n", "# select the most target associated qualitative features\n", "feature_selector = RegressionSelector(\n", " features=features,\n", " n_best_per_type=n_best_per_type,\n", " verbose=True, # displays statistics\n", ")\n", "best_features = feature_selector.select(train_set_processed, train_set_processed[target])\n", "best_features" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncAveRoomsAveOccupAveBedrmsHouseAgePopulation
50880.00.01.02.00.01.0
170962.02.01.02.02.02.0
56171.00.03.02.02.02.0
200600.00.03.01.01.02.0
8952.00.01.02.00.03.0
\n", "
" ], "text/plain": [ " MedInc AveRooms AveOccup AveBedrms HouseAge Population\n", "5088 0.0 0.0 1.0 2.0 0.0 1.0\n", "17096 2.0 2.0 1.0 2.0 2.0 2.0\n", "5617 1.0 0.0 3.0 2.0 2.0 2.0\n", "20060 0.0 0.0 3.0 1.0 1.0 2.0\n", "895 2.0 0.0 1.0 2.0 0.0 3.0" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_set_processed[best_features].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Feature ``MedInc`` is the most associated with the target ``MedHouseVal``:\n", " - Kruskal-Wallis' H value is ``KruskalMeasure=6037.1821``\n", " - It has 0 % of NaNs (``NanMeasure=0.0000``) \n", " - Its mode represents 35 % of observed data (``ModeMeasure=0.3500``)\n", "\n", "* Feature ``AveRooms`` is strongly associated to feature ``MedInc``:\n", " - Tschuprow's T value is ``TschuprowtFilter=0.4015`` for ``TschuprowtWith=MedInc``\n", "\n", "* Here, no feature where filtered out for there inter-feature association or over-represented values (no thresholds were set)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Modeling\n", "Fitting model on train data" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "ename": "AttributeError", "evalue": "'super' object has no attribute '__sklearn_tags__'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\IPython\\core\\formatters.py:974\u001b[0m, in \u001b[0;36mMimeBundleFormatter.__call__\u001b[1;34m(self, obj, include, exclude)\u001b[0m\n\u001b[0;32m 971\u001b[0m method \u001b[38;5;241m=\u001b[39m get_real_method(obj, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mprint_method)\n\u001b[0;32m 973\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m method \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m--> 974\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mmethod\u001b[49m\u001b[43m(\u001b[49m\u001b[43minclude\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minclude\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mexclude\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mexclude\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 975\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m 976\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\base.py:469\u001b[0m, in \u001b[0;36mBaseEstimator._repr_mimebundle_\u001b[1;34m(self, **kwargs)\u001b[0m\n\u001b[0;32m 467\u001b[0m output \u001b[38;5;241m=\u001b[39m {\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtext/plain\u001b[39m\u001b[38;5;124m\"\u001b[39m: \u001b[38;5;28mrepr\u001b[39m(\u001b[38;5;28mself\u001b[39m)}\n\u001b[0;32m 468\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m get_config()[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdisplay\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdiagram\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[1;32m--> 469\u001b[0m output[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mtext/html\u001b[39m\u001b[38;5;124m\"\u001b[39m] \u001b[38;5;241m=\u001b[39m \u001b[43mestimator_html_repr\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[0;32m 470\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m output\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\utils\\_estimator_html_repr.py:387\u001b[0m, in \u001b[0;36mestimator_html_repr\u001b[1;34m(estimator)\u001b[0m\n\u001b[0;32m 385\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m 386\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 387\u001b[0m \u001b[43mcheck_is_fitted\u001b[49m\u001b[43m(\u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 388\u001b[0m status_label \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFitted\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 389\u001b[0m is_fitted_css_class \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfitted\u001b[39m\u001b[38;5;124m\"\u001b[39m\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\utils\\validation.py:1751\u001b[0m, in \u001b[0;36mcheck_is_fitted\u001b[1;34m(estimator, attributes, msg, all_or_any)\u001b[0m\n\u001b[0;32m 1748\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(estimator, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfit\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n\u001b[0;32m 1749\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m is not an estimator instance.\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m (estimator))\n\u001b[1;32m-> 1751\u001b[0m tags \u001b[38;5;241m=\u001b[39m \u001b[43mget_tags\u001b[49m\u001b[43m(\u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 1753\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m tags\u001b[38;5;241m.\u001b[39mrequires_fit \u001b[38;5;129;01mand\u001b[39;00m attributes \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m 1754\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\utils\\_tags.py:430\u001b[0m, in \u001b[0;36mget_tags\u001b[1;34m(estimator)\u001b[0m\n\u001b[0;32m 428\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m klass \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mreversed\u001b[39m(\u001b[38;5;28mtype\u001b[39m(estimator)\u001b[38;5;241m.\u001b[39mmro()):\n\u001b[0;32m 429\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m__sklearn_tags__\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mvars\u001b[39m(klass):\n\u001b[1;32m--> 430\u001b[0m sklearn_tags_provider[klass] \u001b[38;5;241m=\u001b[39m \u001b[43mklass\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__sklearn_tags__\u001b[49m\u001b[43m(\u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# type: ignore[attr-defined]\u001b[39;00m\n\u001b[0;32m 431\u001b[0m class_order\u001b[38;5;241m.\u001b[39mappend(klass)\n\u001b[0;32m 432\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_more_tags\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mvars\u001b[39m(klass):\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\base.py:613\u001b[0m, in \u001b[0;36mRegressorMixin.__sklearn_tags__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m 612\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21m__sklearn_tags__\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m--> 613\u001b[0m tags \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__sklearn_tags__\u001b[49m()\n\u001b[0;32m 614\u001b[0m tags\u001b[38;5;241m.\u001b[39mestimator_type \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mregressor\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 615\u001b[0m tags\u001b[38;5;241m.\u001b[39mregressor_tags \u001b[38;5;241m=\u001b[39m RegressorTags()\n", "\u001b[1;31mAttributeError\u001b[0m: 'super' object has no attribute '__sklearn_tags__'" ] }, { "ename": "AttributeError", "evalue": "'super' object has no attribute '__sklearn_tags__'", "output_type": "error", "traceback": [ "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m", "\u001b[1;31mAttributeError\u001b[0m Traceback (most recent call last)", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\IPython\\core\\formatters.py:344\u001b[0m, in \u001b[0;36mBaseFormatter.__call__\u001b[1;34m(self, obj)\u001b[0m\n\u001b[0;32m 342\u001b[0m method \u001b[38;5;241m=\u001b[39m get_real_method(obj, \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mprint_method)\n\u001b[0;32m 343\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m method \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m--> 344\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mmethod\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 345\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m\n\u001b[0;32m 346\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\base.py:463\u001b[0m, in \u001b[0;36mBaseEstimator._repr_html_inner\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m 458\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21m_repr_html_inner\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[0;32m 459\u001b[0m \u001b[38;5;250m \u001b[39m\u001b[38;5;124;03m\"\"\"This function is returned by the @property `_repr_html_` to make\u001b[39;00m\n\u001b[0;32m 460\u001b[0m \u001b[38;5;124;03m `hasattr(estimator, \"_repr_html_\") return `True` or `False` depending\u001b[39;00m\n\u001b[0;32m 461\u001b[0m \u001b[38;5;124;03m on `get_config()[\"display\"]`.\u001b[39;00m\n\u001b[0;32m 462\u001b[0m \u001b[38;5;124;03m \"\"\"\u001b[39;00m\n\u001b[1;32m--> 463\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mestimator_html_repr\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43mself\u001b[39;49m\u001b[43m)\u001b[49m\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\utils\\_estimator_html_repr.py:387\u001b[0m, in \u001b[0;36mestimator_html_repr\u001b[1;34m(estimator)\u001b[0m\n\u001b[0;32m 385\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[0;32m 386\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m--> 387\u001b[0m \u001b[43mcheck_is_fitted\u001b[49m\u001b[43m(\u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 388\u001b[0m status_label \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mFitted\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 389\u001b[0m is_fitted_css_class \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfitted\u001b[39m\u001b[38;5;124m\"\u001b[39m\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\utils\\validation.py:1751\u001b[0m, in \u001b[0;36mcheck_is_fitted\u001b[1;34m(estimator, attributes, msg, all_or_any)\u001b[0m\n\u001b[0;32m 1748\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28mhasattr\u001b[39m(estimator, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mfit\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n\u001b[0;32m 1749\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m%s\u001b[39;00m\u001b[38;5;124m is not an estimator instance.\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;241m%\u001b[39m (estimator))\n\u001b[1;32m-> 1751\u001b[0m tags \u001b[38;5;241m=\u001b[39m \u001b[43mget_tags\u001b[49m\u001b[43m(\u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m 1753\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m tags\u001b[38;5;241m.\u001b[39mrequires_fit \u001b[38;5;129;01mand\u001b[39;00m attributes \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m 1754\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\utils\\_tags.py:430\u001b[0m, in \u001b[0;36mget_tags\u001b[1;34m(estimator)\u001b[0m\n\u001b[0;32m 428\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m klass \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mreversed\u001b[39m(\u001b[38;5;28mtype\u001b[39m(estimator)\u001b[38;5;241m.\u001b[39mmro()):\n\u001b[0;32m 429\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m__sklearn_tags__\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mvars\u001b[39m(klass):\n\u001b[1;32m--> 430\u001b[0m sklearn_tags_provider[klass] \u001b[38;5;241m=\u001b[39m \u001b[43mklass\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__sklearn_tags__\u001b[49m\u001b[43m(\u001b[49m\u001b[43mestimator\u001b[49m\u001b[43m)\u001b[49m \u001b[38;5;66;03m# type: ignore[attr-defined]\u001b[39;00m\n\u001b[0;32m 431\u001b[0m class_order\u001b[38;5;241m.\u001b[39mappend(klass)\n\u001b[0;32m 432\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m_more_tags\u001b[39m\u001b[38;5;124m\"\u001b[39m \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mvars\u001b[39m(klass):\n", "File \u001b[1;32mc:\\Users\\defra\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\autocarver-i96ERKJw-py3.9\\lib\\site-packages\\sklearn\\base.py:613\u001b[0m, in \u001b[0;36mRegressorMixin.__sklearn_tags__\u001b[1;34m(self)\u001b[0m\n\u001b[0;32m 612\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;21m__sklearn_tags__\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m--> 613\u001b[0m tags \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m__sklearn_tags__\u001b[49m()\n\u001b[0;32m 614\u001b[0m tags\u001b[38;5;241m.\u001b[39mestimator_type \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mregressor\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m 615\u001b[0m tags\u001b[38;5;241m.\u001b[39mregressor_tags \u001b[38;5;241m=\u001b[39m RegressorTags()\n", "\u001b[1;31mAttributeError\u001b[0m: 'super' object has no attribute '__sklearn_tags__'" ] }, { "data": { "text/plain": [ "XGBRegressor(base_score=None, booster=None, callbacks=None,\n", " colsample_bylevel=None, colsample_bynode=None,\n", " colsample_bytree=None, device=None, early_stopping_rounds=None,\n", " enable_categorical=False, eval_metric=None, feature_types=None,\n", " gamma=None, grow_policy=None, importance_type=None,\n", " interaction_constraints=None, learning_rate=None, max_bin=None,\n", " max_cat_threshold=None, max_cat_to_onehot=None,\n", " max_delta_step=None, max_depth=None, max_leaves=None,\n", " min_child_weight=None, missing=nan, monotone_constraints=None,\n", " multi_strategy=None, n_estimators=None, n_jobs=None,\n", " num_parallel_tree=None, random_state=None, ...)" ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from xgboost import XGBRegressor\n", "\n", "model = XGBRegressor()\n", "model.fit(train_set_processed[best_features], train_set_processed[target])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Saving model" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "model.save_model(\"regression_xgboost.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Prediction on dev dataset and performance" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.7773564029114313" ] }, "execution_count": 25, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.metrics import root_mean_squared_error\n", "\n", "dev_pred = model.predict(dev_set_processed[best_features])\n", "root_mean_squared_error(dev_set_processed[target], dev_pred)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What's next?\n", "\n", "* Thanks to **Carvers** all of your features are now optimally processed for your regression task!\n", "* As a final step towards your model, **Selectors** can prove to be handy tools to operate target optimal Data Pre-Selection, so make sure to check out [Selectors Examples](https://autocarver.readthedocs.io/en/latest/selectors_examples.html)!\n", "\n", "## Well done!\n", "\n", "Your commitment to achieving optimal results in continuous regression tasks shines through in your meticulous use of **AutoCarver**'s ``ContinuousCarver`` for data preprocessing. By fine-tuning and optimizing your dataset, you have set the stage for robust and accurate machine learning models.\n", "\n", "The ``ContinuousCarver`` has proven to be a valuable ally in your pursuit of excellence, carving out a path toward enhanced feature representation and model interpretability. Your dedication to refining the data preprocessing steps reflects a commitment to extracting the maximum value from your datasets.\n", "\n", "We extend our sincere appreciation for choosing **AutoCarver** as your companion in the data preprocessing journey. Your use of **AutoCarver** demonstrates a dedication to leveraging cutting-edge tools for achieving excellence in continuous regression tasks.\n", "\n", "As you transition to the modeling phase, may the carefully crafted features and preprocessing steps contribute to the success of your predictive models. We're excited to see the impact of your work and are grateful for the opportunity to be part of your data science endeavors.\n", "\n", "Thank you for trusting **AutoCarver**, and we wish you continued success in your data-driven ventures." ] } ], "metadata": { "kernelspec": { "display_name": "autocarver-i96ERKJw-py3.9", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 2 }