{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Setting things up\n", "\n", "## About this notebook\n", "\n", "In this notebook, we embark on a journey to elevate the predictive capabilities of the California Housing Prices Dataset through advanced preprocessing using the ``ContinuousCarver`` pipeline. Renowned for its association-maximizing discretization, ``ContinuousCarver`` is a powerful Python tool designed to handle diverse data types—whether they be quantitative or qualitative. Our specific goal is to prepare the dataset for continuous regression tasks, such as predicting housing prices.\n", "\n", "The California Housing Prices Dataset is a treasure trove of features, encompassing information on factors like square footage, bedrooms, location, and more. By employing ``ContinuousCarver``, we aim to seamlessly discretize both quantitative and qualitative features, tailoring them for optimal representation in our continuous regression models.\n", "\n", "Throughout this notebook, we'll explore the intricacies of ``ContinuousCarver``'s discretization pipeline, witnessing its adaptability to a variety of data types. Whether it involves transforming square footage or encoding location information, ``ContinuousCarver`` ensures that each feature is finely tuned for our regression tasks.\n", "\n", "Join us in this exploration as we leverage the power of ``ContinuousCarver`` to preprocess the California Housing Prices Dataset. Through effective feature engineering and discretization, our aim is to create a dataset that captures the nuanced relationships within the housing market, setting the stage for the development of accurate and impactful continuous regression models.\n", "\n", "Let's dive in and uncover the potential of ``ContinuousCarver`` in transforming the California Housing Prices Dataset for optimal predictive modeling.\n", "\n", "\n", "## Installation" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "# %pip install AutoCarver[jupyter]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Califorinia Housing Prices Data\n", "\n", "In this example notebook, we will use the California Housing Prices dataset.\n", "\n", "The California Housing Prices dataset is a well-known dataset in the field of machine learning and statistics. It provides information about housing districts in California and is frequently used for regression analysis and predictive modeling tasks.\n", "\n", "Comprising housing-related metrics for various districts in California, such as median house value, median income, housing median age, average rooms, average bedrooms, population, households, and more, the California Housing Prices dataset is a valuable resource for exploring the relationships between different features and predicting the median house values (continuous regression)." ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncHouseAgeAveRoomsAveBedrmsPopulationAveOccupLatitudeLongitudeMedHouseVal
08.325241.06.9841271.023810322.02.55555637.88-122.234.526
18.301421.06.2381370.9718802401.02.10984237.86-122.223.585
27.257452.08.2881361.073446496.02.80226037.85-122.243.521
35.643152.05.8173521.073059558.02.54794537.85-122.253.413
43.846252.06.2818531.081081565.02.18146737.85-122.253.422
\n", "
" ], "text/plain": [ " MedInc HouseAge AveRooms AveBedrms Population AveOccup Latitude \\\n", "0 8.3252 41.0 6.984127 1.023810 322.0 2.555556 37.88 \n", "1 8.3014 21.0 6.238137 0.971880 2401.0 2.109842 37.86 \n", "2 7.2574 52.0 8.288136 1.073446 496.0 2.802260 37.85 \n", "3 5.6431 52.0 5.817352 1.073059 558.0 2.547945 37.85 \n", "4 3.8462 52.0 6.281853 1.081081 565.0 2.181467 37.85 \n", "\n", " Longitude MedHouseVal \n", "0 -122.23 4.526 \n", "1 -122.22 3.585 \n", "2 -122.24 3.521 \n", "3 -122.25 3.413 \n", "4 -122.25 3.422 " ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn import datasets\n", "\n", "# Load dataset directly from sklearn\n", "housing = datasets.fetch_california_housing(as_frame=True)\n", "\n", "# conversion to pandas\n", "housing_data = housing[\"data\"]\n", "housing_data[housing[\"target_names\"][0]] = housing[\"target\"]\n", "\n", "# Display the first few rows of the dataset\n", "housing_data.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Target type and Carver selection" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "count 20640.000000\n", "mean 2.068558\n", "std 1.153956\n", "min 0.149990\n", "25% 1.196000\n", "50% 1.797000\n", "75% 2.647250\n", "max 5.000010\n", "Name: MedHouseVal, dtype: float64" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "target = \"MedHouseVal\"\n", "\n", "housing_data[target].describe()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The target ``\"MedHouseVal\"`` is a continuous target of type ``float64`` used in a regression task. Hence we will use ``AutoCarver.ContinuousCarver`` and ``AutoCarver.selectors.RegressionSelector`` in following code blocks." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Data Sampling" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "(np.float64(2.0666362048018514), np.float64(2.072459655020552))" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.model_selection import train_test_split\n", "\n", "# stratified sampling by target\n", "train_set, dev_set = train_test_split(housing_data, test_size=0.33, random_state=42)\n", "\n", "# checking target rate per dataset\n", "train_set[target].mean(), dev_set[target].mean()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Picking up columns to Carve" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncHouseAgeAveRoomsAveBedrmsPopulationAveOccupLatitudeLongitudeMedHouseVal
50880.980919.03.1877261.129964726.02.62093933.98-118.281.214
170964.223233.06.1896961.0866511015.02.37704937.46-122.233.637
56173.548842.04.8215771.0954361044.04.33195033.79-118.262.056
200601.646924.04.2741941.0483871686.04.53225835.87-119.260.476
8953.990914.04.6083031.0893502738.02.47111937.54-121.962.360
\n", "
" ], "text/plain": [ " MedInc HouseAge AveRooms AveBedrms Population AveOccup Latitude \\\n", "5088 0.9809 19.0 3.187726 1.129964 726.0 2.620939 33.98 \n", "17096 4.2232 33.0 6.189696 1.086651 1015.0 2.377049 37.46 \n", "5617 3.5488 42.0 4.821577 1.095436 1044.0 4.331950 33.79 \n", "20060 1.6469 24.0 4.274194 1.048387 1686.0 4.532258 35.87 \n", "895 3.9909 14.0 4.608303 1.089350 2738.0 2.471119 37.54 \n", "\n", " Longitude MedHouseVal \n", "5088 -118.28 1.214 \n", "17096 -122.23 3.637 \n", "5617 -118.26 2.056 \n", "20060 -119.26 0.476 \n", "895 -121.96 2.360 " ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_set.head()" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "MedInc float64\n", "HouseAge float64\n", "AveRooms float64\n", "AveBedrms float64\n", "Population float64\n", "AveOccup float64\n", "Latitude float64\n", "Longitude float64\n", "MedHouseVal float64\n", "dtype: object" ] }, "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# column data types\n", "train_set.dtypes" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "All features are quantitative continuous features at the exception of ``Latitude`` and ``Longitude`` which are geographical featues (not supported by ``AutoCarver`` as is). All other features will be added to the list of ``quantitative_features``." ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "from AutoCarver import Features\n", "\n", "# lists of features per data type\n", "features = Features(numericals=[\"MedInc\", \"HouseAge\", \"AveRooms\", \"AveBedrms\", \"Population\", \"AveOccup\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Using AutoCarver" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## AutoCarver settings\n", "\n", "### Representativness of modalities\n", "\n", "The attribute ``min_freq`` allows one to choose the minimum frequency per basic modalities. It is used:\n", "\n", "- For quantitative features, to define the number of quantiles to initialy discretize the features with.\n", "\n", "- For qualitative features, to define the threshold under which a modality is grouped to either a default value or its closest modality." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "min_freq = 0.1" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Tip:** should be set between ``0.01`` (slower, preciser, less robust) and ``0.2`` (faster, more robust)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Desired number of modalities\n", "\n", "The attribute ``max_n_mod`` allows one to choose the maximum number of modalities per carved feature. It is used by **Carvers** has the upper limit of number of modalities per consecutive combination of modalities." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [], "source": [ "max_n_mod = 4" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Tip:** should be set between ``3`` (faster, more robust) and ``7`` (slower, preciser, less robust)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Grouping NaNs\n", "\n", "The attribute ``dropna`` allows one to choose whether or not ``nan`` should be grouped with another modality. If set to ``True``, **Carvers** will first find the most suitable combination of non-``nan`` values, and then test out all possible combinations with ``nan``." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "dropna = False # anyway, there are no nan in this dataset" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Type of output carved features\n", "\n", "The attribute ``ordinal_encoding`` allows one to choose the output type:\n", "\n", "* Use ``True`` for integer output of ranked modalities (default)\n", "* Use ``False`` for string output of modalities" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "ordinal_encoding = True" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fitting AutoCarver\n", "\n", "* First, all quantitative features are discretized:\n", " 1. Using ``ContinuousDiscretizer`` for quantile discretization that keeps track of over-represented values (more frequent than ``min_freq``)\n", " 2. Using ``OrdinalDiscretizer`` for any remaining under-represented values (less frequent than ``min_freq/2``) to be grouped with its closest modality\n", "\n", "* Second, all features are carved following this recipe, for all classes of ``train_set[target]`` (except one):\n", " 1. The raw distribution is printed out on provided ``train_set`` and ``dev_set``. It's the output of the discretization step\n", " 2. Grouping modalities: all consecutive combinations of modalities are applied to ``train_set``\n", " 3. Computing associations: the association metric (Krsuskal-Wallis' statistic, by default) is computed with the provided ``train_set[target]``\n", " 4. Combinations are sorted in descending order by association value\n", " 5. Testing robustness: finds the first combination that checks the following:\n", " - Representativness of modalities on ``train_set`` and ``dev_set`` (all should be more frequent than ``min_freq/2``)\n", " - Distinct target rates per consecutive modalities on ``train_set`` and ``dev_set`` \n", " - No inversion of target rates between ``train_set`` and ``dev_set`` (same ordering of modalities by target rate)\n", " 6. (Optional) If requested via ``dropna=True``, and if any, all combinations of modalities with ``nan`` are applied to ``train_set`` and steps 3. and 4. are run\n", " 7. The carved distribution is printed out on provided ``train_set`` and ``dev_set``. It's the output of the carving step" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "------\n", "--- [QuantitativeDiscretizer] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])\n", " - [ContinuousDiscretizer] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])\n", " - [OrdinalDiscretizer] Fit Features(['HouseAge'])\n", "------\n", "\n", "---------\n", "------ [ContinuousCarver] Fit Features(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup'])\n", "--- [ContinuousCarver] Fit Quantitative('MedInc') (1/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 1.60e+001.11020.0500692
1.60e+00 < x <= 1.91e+001.12850.0500691
1.91e+00 < x <= 2.15e+001.21980.0500692
2.15e+00 < x <= 2.35e+001.31710.0500691
2.35e+00 < x <= 2.57e+001.38170.0500691
2.57e+00 < x <= 2.74e+001.54090.0500692
2.74e+00 < x <= 2.98e+001.61590.0500692
2.98e+00 < x <= 3.14e+001.69060.0499690
3.14e+00 < x <= 3.32e+001.82320.0500692
3.32e+00 < x <= 3.54e+001.90590.0500691
3.54e+00 < x <= 3.73e+002.00760.0502694
3.73e+00 < x <= 3.97e+002.02710.0498689
3.97e+00 < x <= 4.18e+002.14560.0500691
4.18e+00 < x <= 4.46e+002.24330.0500691
4.46e+00 < x <= 4.76e+002.36210.0501693
4.76e+00 < x <= 5.12e+002.39860.0499690
5.12e+00 < x <= 5.54e+002.64380.0500691
5.54e+00 < x <= 6.16e+002.93240.0500692
6.16e+00 < x <= 7.32e+003.45920.0500691
7.32e+00 < x4.37840.0500692
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
1.10170.0509347
1.04100.0502342
1.24070.0501341
1.29190.0506345
1.46760.0536365
1.56050.0417284
1.62800.0584398
1.75190.0471321
1.84430.0504343
1.85000.0498339
2.00400.0533363
2.08900.0502342
2.16410.0505344
2.27000.0540368
2.37680.0439299
2.50870.0479326
2.68140.0483329
2.98050.0479326
3.37480.0530361
4.37480.0483329
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 2.57e+001.23140.25003457
2.57e+00 < x <= 3.97e+001.80160.35004840
3.97e+00 < x <= 5.54e+002.35870.24993456
5.54e+00 < x3.59000.15012075
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
1.23150.25541740
1.82220.35092390
2.39530.24461666
3.57210.14911016
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('HouseAge') (2/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 8.00e+002.11580.0537742
8.00e+00 < x <= 1.20e+011.82200.0477659
1.20e+01 < x <= 1.50e+011.85900.0613847
1.50e+01 < x <= 1.80e+011.95470.09891367
1.80e+01 < x <= 2.20e+011.97390.08711205
2.20e+01 < x <= 2.50e+012.10550.0705975
2.50e+01 < x <= 2.80e+012.05120.07751072
2.80e+01 < x <= 3.10e+012.04390.0682943
3.10e+01 < x <= 3.30e+012.02750.0575795
3.30e+01 < x <= 3.50e+012.06510.0722999
3.50e+01 < x <= 3.70e+012.07500.0687950
3.70e+01 < x <= 4.20e+012.01020.08111121
4.20e+01 < x <= 4.50e+012.13010.0485670
4.50e+01 < x2.47850.10721483
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
2.02050.0526358
1.78270.0443302
1.87800.0556379
1.93910.0986672
2.02850.0891607
2.11790.0759517
2.16340.0743506
1.95460.0664452
2.05120.0565385
2.18620.0755514
2.07470.0659449
2.01740.0895610
2.00150.0489333
2.46510.1069728
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 2.20e+011.94940.34864820
2.20e+01 < x <= 3.70e+012.06230.41475734
3.70e+01 < x <= 4.50e+012.05500.12951791
4.50e+01 < x2.47850.10721483
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
1.94470.34032318
2.09640.41442823
2.01180.1384943
2.46510.1069728
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('AveRooms') (3/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 3.44e+001.91260.0500692
3.44e+00 < x <= 3.79e+001.82860.0500691
3.79e+00 < x <= 4.06e+001.81690.0500692
4.06e+00 < x <= 4.28e+001.84180.0500691
4.28e+00 < x <= 4.46e+001.75290.0500691
4.46e+00 < x <= 4.62e+001.79150.0500692
4.62e+00 < x <= 4.79e+001.82140.0500691
4.79e+00 < x <= 4.94e+001.76850.0500691
4.94e+00 < x <= 5.09e+001.74660.0500692
5.09e+00 < x <= 5.23e+001.77170.0500691
5.23e+00 < x <= 5.38e+001.86640.0500691
5.38e+00 < x <= 5.53e+001.84720.0500692
5.53e+00 < x <= 5.69e+001.91990.0500691
5.69e+00 < x <= 5.86e+001.99100.0500691
5.86e+00 < x <= 6.06e+002.08700.0500692
6.06e+00 < x <= 6.27e+002.19080.0500691
6.27e+00 < x <= 6.54e+002.40500.0500691
6.54e+00 < x <= 6.95e+002.68740.0500692
6.95e+00 < x <= 7.65e+003.11290.0500691
7.65e+00 < x3.17180.0500692
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
1.86590.0518353
1.87280.0505344
1.76270.0524357
1.80200.0543370
1.72230.0552376
1.68020.0452308
1.77070.0530361
1.80300.0443302
1.82090.0523356
1.83260.0437298
1.79230.0550375
1.93880.0514350
1.94650.0501341
2.02480.0468319
2.10490.0483329
2.22390.0490334
2.43390.0467318
2.76670.0468319
3.10010.0548373
3.24290.0483329
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 5.69e+001.82200.65008988
5.69e+00 < x <= 6.27e+002.08960.15002074
6.27e+00 < x <= 6.95e+002.54630.10001383
6.95e+00 < x3.14240.10001383
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
1.81620.65934491
2.11940.1442982
2.60060.0935637
3.16700.1031702
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('AveBedrms') (4/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 9.4000e-012.06840.0500692
9.4000e-01 < x <= 9.6724e-012.07350.0500691
9.6724e-01 < x <= 9.8319e-012.21670.0501693
9.8319e-01 < x <= 9.9576e-012.17060.0499690
9.9576e-01 < x <= 1.0066e+002.13100.0500692
1.0066e+00 < x <= 1.0154e+002.23580.0500691
1.0154e+00 < x <= 1.0247e+002.16680.0500691
1.0247e+00 < x <= 1.0331e+002.21020.0500692
1.0331e+00 < x <= 1.0412e+002.12950.0500691
1.0412e+00 < x <= 1.0495e+002.15480.0500691
1.0495e+00 < x <= 1.0576e+002.12380.0500692
1.0576e+00 < x <= 1.0665e+002.10250.0500691
1.0665e+00 < x <= 1.0768e+002.07040.0500691
1.0768e+00 < x <= 1.0878e+002.06640.0501693
1.0878e+00 < x <= 1.1003e+002.11180.0499690
1.1003e+00 < x <= 1.1161e+001.99370.0500691
1.1161e+00 < x <= 1.1382e+001.94050.0500691
1.1382e+00 < x <= 1.1738e+001.79900.0500692
1.1738e+00 < x <= 1.2732e+001.91620.0500691
1.2732e+00 < x1.65150.0500692
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
2.04160.0539367
2.20430.0527359
2.09970.0482328
2.18350.0487332
2.26280.0552376
2.16190.0480327
2.22950.0567386
2.16900.0493336
2.15810.0528360
2.12020.0476324
2.10390.0452308
2.15950.0509347
2.10370.0521355
2.06620.0484330
2.04870.0489333
1.95430.0467318
1.88710.0484330
1.86800.0499340
1.83710.0465317
1.71820.0498339
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 1.058e+002.15280.55007606
1.058e+00 < x <= 1.100e+002.08780.20002765
1.100e+00 < x <= 1.138e+001.96710.09991382
1.138e+00 < x1.78880.15012075
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
2.15970.55833803
2.09540.20041365
1.92010.0951648
1.80720.1462996
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('Population') (5/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 3.53e+021.98590.0501693
3.53e+02 < x <= 5.14e+022.16160.0501693
5.14e+02 < x <= 6.27e+022.11170.0501693
6.27e+02 < x <= 7.15e+022.28190.0497687
7.15e+02 < x <= 7.93e+022.03350.0509704
7.93e+02 < x <= 8.64e+022.21130.0492681
8.64e+02 < x <= 9.38e+022.07720.0498689
9.38e+02 < x <= 1.02e+032.13860.0500692
1.02e+03 < x <= 1.09e+032.04300.0503696
1.09e+03 < x <= 1.17e+032.05060.0496686
1.17e+03 < x <= 1.26e+032.08700.0505698
1.26e+03 < x <= 1.35e+032.01950.0497687
1.35e+03 < x <= 1.46e+032.00040.0502694
1.46e+03 < x <= 1.58e+032.11020.0498688
1.58e+03 < x <= 1.73e+032.03460.0500691
1.73e+03 < x <= 1.91e+031.91390.0499690
1.91e+03 < x <= 2.15e+032.00060.0500691
2.15e+03 < x <= 2.56e+032.07070.0500692
2.56e+03 < x <= 3.30e+031.96140.0500691
3.30e+03 < x2.04280.0500692
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
1.90120.0530361
2.19150.0520354
2.17060.0523356
2.10620.0514350
2.20190.0531362
2.17650.0490334
2.20250.0506345
2.13290.0553377
2.17440.0437298
2.13190.0480327
1.99390.0534364
2.00960.0465317
1.95690.0465317
1.97560.0504343
2.08150.0496338
2.02720.0461314
1.97890.0487332
1.93550.0496338
2.07140.0518353
2.01570.0487332
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 6.27e+022.08640.15032079
6.27e+02 < x <= 8.64e+022.17430.14982072
8.64e+02 < x <= 2.15e+032.04330.54987602
2.15e+03 < x2.02500.15012075
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
2.08670.15721071
2.16180.15361046
2.06070.53903672
2.00840.15021023
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "--- [ContinuousCarver] Fit Quantitative('AveOccup') (6/6)\n", " [ContinuousCarver] Raw distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 1.870e+002.71220.0500692
1.870e+00 < x <= 2.067e+002.66330.0500691
2.067e+00 < x <= 2.225e+002.33730.0500692
2.225e+00 < x <= 2.338e+002.30800.0500691
2.338e+00 < x <= 2.432e+002.19760.0500691
2.432e+00 < x <= 2.513e+002.20640.0500692
2.513e+00 < x <= 2.595e+002.17360.0500691
2.595e+00 < x <= 2.668e+002.18620.0500691
2.668e+00 < x <= 2.743e+002.13780.0500692
2.743e+00 < x <= 2.820e+002.19020.0500691
2.820e+00 < x <= 2.898e+002.18240.0500691
2.898e+00 < x <= 2.984e+002.07410.0500692
2.984e+00 < x <= 3.073e+002.02550.0501693
3.073e+00 < x <= 3.171e+001.99140.0498689
3.171e+00 < x <= 3.282e+001.89920.0500692
3.282e+00 < x <= 3.425e+001.89260.0500691
3.425e+00 < x <= 3.607e+001.70850.0500691
3.607e+00 < x <= 3.877e+001.56660.0500692
3.877e+00 < x <= 4.325e+001.45050.0500691
4.325e+00 < x1.42940.0500692
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
2.76840.0484330
2.53340.0435296
2.39890.0542369
2.36410.0533363
2.22720.0546372
2.29690.0489333
2.31790.0508346
2.07930.0467318
2.18470.0521355
2.17520.0504343
2.07620.0533363
2.05350.0501341
2.05350.0528360
1.94770.0458312
1.83970.0449306
1.88610.0514350
1.73010.0448305
1.62000.0499340
1.44230.0527359
1.45960.0515351
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ " [ContinuousCarver] Carved distribution\n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X distribution
 target_meanfrequencycount
x <= 2.22e+002.57090.15012075
2.22e+00 < x <= 3.07e+002.16810.50016915
3.07e+00 < x <= 3.61e+001.87290.19982763
3.61e+00 < x1.48220.15012075
\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
X_dev distribution
target_meanfrequencycount
2.56150.1461995
2.18360.51293494
1.85270.18691273
1.50560.15411050
\n" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from AutoCarver import ContinuousCarver\n", "from AutoCarver.discretizers.utils.base_discretizer import ProcessingConfig\n", "\n", "# intiating AutoCarver\n", "auto_carver = ContinuousCarver(\n", " features=features,\n", " min_freq=min_freq,\n", " max_n_mod=max_n_mod,\n", " config=ProcessingConfig(dropna=dropna, ordinal_encoding=ordinal_encoding, verbose=True, copy=True),\n", ")\n", "\n", "# fitting on training sample, a dev sample can be specified to evaluate carving robustness\n", "train_set_processed = auto_carver.fit_transform(train_set, train_set[target], X_dev=dev_set, y_dev=dev_set[target])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## AutoCarver analysis\n", "\n", "### Carving Summary" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
contenttarget_meanfrequencydroppeddropped_reason
featurecountkruskaln_modlabel
Quantitative('MedInc')3457.06037.18213540x <= 2.57e+001.2314210.250000FalseNone
4840.06037.182135412.57e+00 < x <= 3.97e+001.8015620.350014FalseNone
3456.06037.182135423.97e+00 < x <= 5.54e+002.3586600.249928FalseNone
2075.06037.182135435.54e+00 < x3.5900400.150058FalseNone
Quantitative('HouseAge')4820.0160.59961040x <= 2.20e+011.9493610.348568FalseNone
5734.0160.599610412.20e+01 < x <= 3.70e+012.0623060.414666FalseNone
1791.0160.599610423.70e+01 < x <= 4.50e+012.0550430.129520FalseNone
1483.0160.599610434.50e+01 < x2.4785420.107246FalseNone
Quantitative('AveRooms')8988.01401.05257240x <= 5.69e+001.8219990.649986FalseNone
2074.01401.052572415.69e+00 < x <= 6.27e+002.0895950.149986FalseNone
1383.01401.052572426.27e+00 < x <= 6.95e+002.5463150.100014FalseNone
36.95e+00 < x3.1424060.100014FalseNone
Quantitative('AveBedrms')7606.0320.78984540x <= 1.058e+002.1528320.550043FalseNone
2765.0320.789845411.058e+00 < x <= 1.100e+002.0877730.199957FalseNone
1382.0320.789845421.100e+00 < x <= 1.138e+001.9670660.099942FalseNone
2075.0320.789845431.138e+00 < x1.7888310.150058FalseNone
Quantitative('Population')2079.016.10970940x <= 6.27e+022.0863940.150347FalseNone
2072.016.109709416.27e+02 < x <= 8.64e+022.1742970.149841FalseNone
7602.016.109709428.64e+02 < x <= 2.15e+032.0432550.549754FalseNone
2075.016.109709432.15e+03 < x2.0249950.150058FalseNone
Quantitative('AveOccup')2075.0991.40830140x <= 2.22e+002.5708880.150058FalseNone
6915.0991.408301412.22e+00 < x <= 3.07e+002.1681260.500072FalseNone
2763.0991.408301423.07e+00 < x <= 3.61e+001.8728670.199812FalseNone
2075.0991.408301433.61e+00 < x1.4821830.150058FalseNone
\n", "
" ], "text/plain": [ " content \\\n", "feature count kruskal n_mod label \n", "Quantitative('MedInc') 3457.0 6037.182135 4 0 x <= 2.57e+00 \n", " 4840.0 6037.182135 4 1 2.57e+00 < x <= 3.97e+00 \n", " 3456.0 6037.182135 4 2 3.97e+00 < x <= 5.54e+00 \n", " 2075.0 6037.182135 4 3 5.54e+00 < x \n", "Quantitative('HouseAge') 4820.0 160.599610 4 0 x <= 2.20e+01 \n", " 5734.0 160.599610 4 1 2.20e+01 < x <= 3.70e+01 \n", " 1791.0 160.599610 4 2 3.70e+01 < x <= 4.50e+01 \n", " 1483.0 160.599610 4 3 4.50e+01 < x \n", "Quantitative('AveRooms') 8988.0 1401.052572 4 0 x <= 5.69e+00 \n", " 2074.0 1401.052572 4 1 5.69e+00 < x <= 6.27e+00 \n", " 1383.0 1401.052572 4 2 6.27e+00 < x <= 6.95e+00 \n", " 3 6.95e+00 < x \n", "Quantitative('AveBedrms') 7606.0 320.789845 4 0 x <= 1.058e+00 \n", " 2765.0 320.789845 4 1 1.058e+00 < x <= 1.100e+00 \n", " 1382.0 320.789845 4 2 1.100e+00 < x <= 1.138e+00 \n", " 2075.0 320.789845 4 3 1.138e+00 < x \n", "Quantitative('Population') 2079.0 16.109709 4 0 x <= 6.27e+02 \n", " 2072.0 16.109709 4 1 6.27e+02 < x <= 8.64e+02 \n", " 7602.0 16.109709 4 2 8.64e+02 < x <= 2.15e+03 \n", " 2075.0 16.109709 4 3 2.15e+03 < x \n", "Quantitative('AveOccup') 2075.0 991.408301 4 0 x <= 2.22e+00 \n", " 6915.0 991.408301 4 1 2.22e+00 < x <= 3.07e+00 \n", " 2763.0 991.408301 4 2 3.07e+00 < x <= 3.61e+00 \n", " 2075.0 991.408301 4 3 3.61e+00 < x \n", "\n", " target_mean \\\n", "feature count kruskal n_mod label \n", "Quantitative('MedInc') 3457.0 6037.182135 4 0 1.231421 \n", " 4840.0 6037.182135 4 1 1.801562 \n", " 3456.0 6037.182135 4 2 2.358660 \n", " 2075.0 6037.182135 4 3 3.590040 \n", "Quantitative('HouseAge') 4820.0 160.599610 4 0 1.949361 \n", " 5734.0 160.599610 4 1 2.062306 \n", " 1791.0 160.599610 4 2 2.055043 \n", " 1483.0 160.599610 4 3 2.478542 \n", "Quantitative('AveRooms') 8988.0 1401.052572 4 0 1.821999 \n", " 2074.0 1401.052572 4 1 2.089595 \n", " 1383.0 1401.052572 4 2 2.546315 \n", " 3 3.142406 \n", "Quantitative('AveBedrms') 7606.0 320.789845 4 0 2.152832 \n", " 2765.0 320.789845 4 1 2.087773 \n", " 1382.0 320.789845 4 2 1.967066 \n", " 2075.0 320.789845 4 3 1.788831 \n", "Quantitative('Population') 2079.0 16.109709 4 0 2.086394 \n", " 2072.0 16.109709 4 1 2.174297 \n", " 7602.0 16.109709 4 2 2.043255 \n", " 2075.0 16.109709 4 3 2.024995 \n", "Quantitative('AveOccup') 2075.0 991.408301 4 0 2.570888 \n", " 6915.0 991.408301 4 1 2.168126 \n", " 2763.0 991.408301 4 2 1.872867 \n", " 2075.0 991.408301 4 3 1.482183 \n", "\n", " frequency dropped \\\n", "feature count kruskal n_mod label \n", "Quantitative('MedInc') 3457.0 6037.182135 4 0 0.250000 False \n", " 4840.0 6037.182135 4 1 0.350014 False \n", " 3456.0 6037.182135 4 2 0.249928 False \n", " 2075.0 6037.182135 4 3 0.150058 False \n", "Quantitative('HouseAge') 4820.0 160.599610 4 0 0.348568 False \n", " 5734.0 160.599610 4 1 0.414666 False \n", " 1791.0 160.599610 4 2 0.129520 False \n", " 1483.0 160.599610 4 3 0.107246 False \n", "Quantitative('AveRooms') 8988.0 1401.052572 4 0 0.649986 False \n", " 2074.0 1401.052572 4 1 0.149986 False \n", " 1383.0 1401.052572 4 2 0.100014 False \n", " 3 0.100014 False \n", "Quantitative('AveBedrms') 7606.0 320.789845 4 0 0.550043 False \n", " 2765.0 320.789845 4 1 0.199957 False \n", " 1382.0 320.789845 4 2 0.099942 False \n", " 2075.0 320.789845 4 3 0.150058 False \n", "Quantitative('Population') 2079.0 16.109709 4 0 0.150347 False \n", " 2072.0 16.109709 4 1 0.149841 False \n", " 7602.0 16.109709 4 2 0.549754 False \n", " 2075.0 16.109709 4 3 0.150058 False \n", "Quantitative('AveOccup') 2075.0 991.408301 4 0 0.150058 False \n", " 6915.0 991.408301 4 1 0.500072 False \n", " 2763.0 991.408301 4 2 0.199812 False \n", " 2075.0 991.408301 4 3 0.150058 False \n", "\n", " dropped_reason \n", "feature count kruskal n_mod label \n", "Quantitative('MedInc') 3457.0 6037.182135 4 0 None \n", " 4840.0 6037.182135 4 1 None \n", " 3456.0 6037.182135 4 2 None \n", " 2075.0 6037.182135 4 3 None \n", "Quantitative('HouseAge') 4820.0 160.599610 4 0 None \n", " 5734.0 160.599610 4 1 None \n", " 1791.0 160.599610 4 2 None \n", " 1483.0 160.599610 4 3 None \n", "Quantitative('AveRooms') 8988.0 1401.052572 4 0 None \n", " 2074.0 1401.052572 4 1 None \n", " 1383.0 1401.052572 4 2 None \n", " 3 None \n", "Quantitative('AveBedrms') 7606.0 320.789845 4 0 None \n", " 2765.0 320.789845 4 1 None \n", " 1382.0 320.789845 4 2 None \n", " 2075.0 320.789845 4 3 None \n", "Quantitative('Population') 2079.0 16.109709 4 0 None \n", " 2072.0 16.109709 4 1 None \n", " 7602.0 16.109709 4 2 None \n", " 2075.0 16.109709 4 3 None \n", "Quantitative('AveOccup') 2075.0 991.408301 4 0 None \n", " 6915.0 991.408301 4 1 None \n", " 2763.0 991.408301 4 2 None \n", " 2075.0 991.408301 4 3 None " ] }, "execution_count": 20, "metadata": {}, "output_type": "execute_result" } ], "source": [ "auto_carver.summary" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* As requested with ``ordinal_encoding=True``, output labels are integers of modalities\n", "\n", "* For quantitative feature ``Population``, the selected combination of modalities groups populations as follows:\n", " * label ``0``: lower or equal to 630 people (``content=\"x <= 6.3e+02\"``)\n", " * label ``1``: greater than 630 people and lower or equal to 860 people (``content=\"6.3e+02 < x <= 8.6e+02\"``)\n", " * label ``2``: greater than 860 people and lower or equal to 2200 people (``content=\"8.6e+02 < x <= 2.2e+03\"``)\n", " * label ``3``: higher than 2200 people (``content=\"2.2e+03 < x\"``)\n", "\n", "### Detailed overview of tested combinations" ] }, { "cell_type": "code", "execution_count": 21, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
infokruskalcombinationn_moddropnatrainviabledev
0Raw distribution (n_mod=20>max_n_mod=4)1062.072498{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...20FalseNaNNaNNaN
1Not viable994.514410{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}False{'viable': False, 'info': 'Non-representative ...
2Not viable994.504665{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}False{'viable': False, 'info': 'Non-representative ...
3Not viable991.504255{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}False{'viable': False, 'info': 'Non-representative ...
4Best for kruskal and max_n_mod=4991.408301{'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0...4False{'viable': True, 'info': ''}True{'viable': True, 'info': ''}
\n", "
" ], "text/plain": [ " info kruskal \\\n", "0 Raw distribution (n_mod=20>max_n_mod=4) 1062.072498 \n", "1 Not viable 994.514410 \n", "2 Not viable 994.504665 \n", "3 Not viable 991.504255 \n", "4 Best for kruskal and max_n_mod=4 991.408301 \n", "\n", " combination n_mod dropna \\\n", "0 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 20 False \n", "1 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "2 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "3 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "4 {'x <= 1.870e+00': 'x <= 1.870e+00', '1.870e+0... 4 False \n", "\n", " train viable \\\n", "0 NaN NaN \n", "1 {'viable': True, 'info': ''} False \n", "2 {'viable': True, 'info': ''} False \n", "3 {'viable': True, 'info': ''} False \n", "4 {'viable': True, 'info': ''} True \n", "\n", " dev \n", "0 NaN \n", "1 {'viable': False, 'info': 'Non-representative ... \n", "2 {'viable': False, 'info': 'Non-representative ... \n", "3 {'viable': False, 'info': 'Non-representative ... \n", "4 {'viable': True, 'info': ''} " ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"AveOccup\"].history.head(7)" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'viable': False, 'info': 'Non-representative modality for min_freq=10.00%'}" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "features[\"AveOccup\"].history.dev[1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* The most associated combination of feature ``AveOccup`` (the first tested out, where ``info!=\"Raw distribution\"``) did not pass the viability tests. When looking in ``history.dev``:\n", " * ``\"Non-representative modality for min_freq=10.00%\"``: tells us that a modality is unstable between ``train_set`` and ``dev_set``\n", "\n", "* For feature feature ``AveOccup``, the 4th combination is the first to pass tests:\n", " - ``viabe=True``\n", " - ``info=\"Best for kruskal and max_n_mod=4\"``\n", " - Kruskal-Wallis' H with ``MedHouseVal`` is ``991.408301`` for this combination\n", " - Following combinations (less associated with the target) where not tested: ``info=\"Not checked\"``\n", "\n", "* For all combinations ``dropna=False`` means that it is not a combination in which ``nan``s are being grouped with other modalities (as requested with ``dropna=False``)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Saving and Loading AutoCarver" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Saving\n", "\n", "All **Carvers** can safely be stored as a .json file." ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [], "source": [ "auto_carver.save(\"continuous_carver.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Loading\n", "\n", "**Carvers** can safely be loaded from a .json file." ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "from AutoCarver import ContinuousCarver\n", "\n", "# loading json file\n", "auto_carver = ContinuousCarver.load('continuous_carver.json')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Applying AutoCarver" ] }, { "cell_type": "code", "execution_count": 25, "metadata": {}, "outputs": [], "source": [ "dev_set_processed = auto_carver.transform(dev_set)" ] }, { "cell_type": "code", "execution_count": 27, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncHouseAgeAveRoomsAveBedrmsPopulationAveOccup
0.00.2554320.3402820.6592780.5582800.1572230.146066
1.00.3508510.4144160.1441570.2003820.1535530.512918
2.00.2445680.1384320.0935110.0951260.5390490.186876
3.00.1491490.1068700.1030530.1462130.1501760.154140
\n", "
" ], "text/plain": [ " MedInc HouseAge AveRooms AveBedrms Population AveOccup\n", "0.0 0.255432 0.340282 0.659278 0.558280 0.157223 0.146066\n", "1.0 0.350851 0.414416 0.144157 0.200382 0.153553 0.512918\n", "2.0 0.244568 0.138432 0.093511 0.095126 0.539049 0.186876\n", "3.0 0.149149 0.106870 0.103053 0.146213 0.150176 0.154140" ] }, "execution_count": 27, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dev_set_processed[auto_carver.features].apply(lambda u: u.value_counts(dropna=False, normalize=True))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Feature Selection\n", "## Selectors settings\n", "### Features to select from\n", "\n", "Here all features have been carved using ``ContinuousCarver``, hence all features are qualitative.\n", "\n", "### Number of features to select\n", "\n", "The attribute ``n_best_per_type`` allows one to choose the number of features to be selected per data type (quantitative and qualitative)." ] }, { "cell_type": "code", "execution_count": 28, "metadata": {}, "outputs": [], "source": [ "n_best_per_type = 6 " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using Selectors" ] }, { "cell_type": "code", "execution_count": 29, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " [RegressionSelector] Selected Qualitative Features \n" ] }, { "data": { "text/html": [ "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
 featureNanModeKruskalEtaSquaredMeasureKruskalEtaSquaredRankTschuprowtFilterTschuprowtWith
0Quantitative('MedInc')0.00000.35000.43650.00000.0000itself
2Quantitative('AveRooms')0.00000.65000.10111.00000.3854MedInc
5Quantitative('AveOccup')0.00000.50010.07152.00000.1620AveRooms
3Quantitative('AveBedrms')0.00000.55000.02303.00000.1395MedInc
1Quantitative('HouseAge')0.00000.41470.01144.00000.1345AveRooms
4Quantitative('Population')0.00000.54980.00095.00000.1464HouseAge
\n" ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/plain": [ "Features(['MedInc', 'AveRooms', 'AveOccup', 'AveBedrms', 'HouseAge', 'Population'])" ] }, "execution_count": 29, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from AutoCarver.selectors import RegressionSelector\n", "from AutoCarver.discretizers.utils.base_discretizer import ProcessingConfig\n", "\n", "# select the most target associated qualitative features\n", "feature_selector = RegressionSelector(\n", " features=features,\n", " n_best_per_type=n_best_per_type,\n", " config=ProcessingConfig(verbose=True), # displays statistics\n", ")\n", "best_features = feature_selector.fit(train_set_processed, train_set_processed[target]).selected_features\n", "best_features" ] }, { "cell_type": "code", "execution_count": 30, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
MedIncAveRoomsAveOccupAveBedrmsHouseAgePopulation
50880.00.01.02.00.01.0
170962.01.01.01.01.02.0
56171.00.03.01.02.02.0
200600.00.03.00.01.02.0
8952.00.01.01.00.03.0
\n", "
" ], "text/plain": [ " MedInc AveRooms AveOccup AveBedrms HouseAge Population\n", "5088 0.0 0.0 1.0 2.0 0.0 1.0\n", "17096 2.0 1.0 1.0 1.0 1.0 2.0\n", "5617 1.0 0.0 3.0 1.0 2.0 2.0\n", "20060 0.0 0.0 3.0 0.0 1.0 2.0\n", "895 2.0 0.0 1.0 1.0 0.0 3.0" ] }, "execution_count": 30, "metadata": {}, "output_type": "execute_result" } ], "source": [ "train_set_processed[best_features].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "* Feature ``MedInc`` is the most associated with the target ``MedHouseVal``:\n", " - Kruskal-Wallis' H value is ``KruskalMeasure=6037.1821``\n", " - It has 0 % of NaNs (``NanMeasure=0.0000``) \n", " - Its mode represents 35 % of observed data (``ModeMeasure=0.3500``)\n", "\n", "* Feature ``AveRooms`` is strongly associated to feature ``MedInc``:\n", " - Tschuprow's T value is ``TschuprowtFilter=0.4015`` for ``TschuprowtWith=MedInc``\n", "\n", "* Here, no feature where filtered out for there inter-feature association or over-represented values (no thresholds were set)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Modeling\n", "Fitting model on train data" ] }, { "cell_type": "code", "execution_count": 31, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
XGBRegressor(base_score=None, booster=None, callbacks=None,\n",
       "             colsample_bylevel=None, colsample_bynode=None,\n",
       "             colsample_bytree=None, device=None, early_stopping_rounds=None,\n",
       "             enable_categorical=False, eval_metric=None, feature_types=None,\n",
       "             feature_weights=None, gamma=None, grow_policy=None,\n",
       "             importance_type=None, interaction_constraints=None,\n",
       "             learning_rate=None, max_bin=None, max_cat_threshold=None,\n",
       "             max_cat_to_onehot=None, max_delta_step=None, max_depth=None,\n",
       "             max_leaves=None, min_child_weight=None, missing=nan,\n",
       "             monotone_constraints=None, multi_strategy=None, n_estimators=None,\n",
       "             n_jobs=None, num_parallel_tree=None, ...)
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ], "text/plain": [ "XGBRegressor(base_score=None, booster=None, callbacks=None,\n", " colsample_bylevel=None, colsample_bynode=None,\n", " colsample_bytree=None, device=None, early_stopping_rounds=None,\n", " enable_categorical=False, eval_metric=None, feature_types=None,\n", " feature_weights=None, gamma=None, grow_policy=None,\n", " importance_type=None, interaction_constraints=None,\n", " learning_rate=None, max_bin=None, max_cat_threshold=None,\n", " max_cat_to_onehot=None, max_delta_step=None, max_depth=None,\n", " max_leaves=None, min_child_weight=None, missing=nan,\n", " monotone_constraints=None, multi_strategy=None, n_estimators=None,\n", " n_jobs=None, num_parallel_tree=None, ...)" ] }, "execution_count": 31, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from xgboost import XGBRegressor\n", "\n", "model = XGBRegressor()\n", "model.fit(train_set_processed[best_features], train_set_processed[target])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Saving model" ] }, { "cell_type": "code", "execution_count": 24, "metadata": {}, "outputs": [], "source": [ "model.save_model(\"regression_xgboost.json\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Prediction on dev dataset and performance" ] }, { "cell_type": "code", "execution_count": 32, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "0.7807707240408291" ] }, "execution_count": 32, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from sklearn.metrics import root_mean_squared_error\n", "\n", "dev_pred = model.predict(dev_set_processed[best_features])\n", "root_mean_squared_error(dev_set_processed[target], dev_pred)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What's next?\n", "\n", "* Thanks to **Carvers** all of your features are now optimally processed for your regression task!\n", "* As a final step towards your model, **Selectors** can prove to be handy tools to operate target optimal Data Pre-Selection, so make sure to check out [Selectors Examples](https://autocarver.readthedocs.io/en/latest/selectors_examples.html)!\n", "\n", "## Well done!\n", "\n", "Your commitment to achieving optimal results in continuous regression tasks shines through in your meticulous use of **AutoCarver**'s ``ContinuousCarver`` for data preprocessing. By fine-tuning and optimizing your dataset, you have set the stage for robust and accurate machine learning models.\n", "\n", "The ``ContinuousCarver`` has proven to be a valuable ally in your pursuit of excellence, carving out a path toward enhanced feature representation and model interpretability. Your dedication to refining the data preprocessing steps reflects a commitment to extracting the maximum value from your datasets.\n", "\n", "We extend our sincere appreciation for choosing **AutoCarver** as your companion in the data preprocessing journey. Your use of **AutoCarver** demonstrates a dedication to leveraging cutting-edge tools for achieving excellence in continuous regression tasks.\n", "\n", "As you transition to the modeling phase, may the carefully crafted features and preprocessing steps contribute to the success of your predictive models. We're excited to see the impact of your work and are grateful for the opportunity to be part of your data science endeavors.\n", "\n", "Thank you for trusting **AutoCarver**, and we wish you continued success in your data-driven ventures." ] } ], "metadata": { "kernelspec": { "display_name": "AutoCarver", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.15" } }, "nbformat": 4, "nbformat_minor": 2 }