diff --git a/docs/_freeze/posts/ibisml/index/execute-results/html.json b/docs/_freeze/posts/ibisml/index/execute-results/html.json new file mode 100644 index 000000000000..950118eafda8 --- /dev/null +++ b/docs/_freeze/posts/ibisml/index/execute-results/html.json @@ -0,0 +1,19 @@ +{ + "hash": "7961865bb3c84357060e112e10cd8afb", + "result": { + "engine": "jupyter", + "markdown": "---\ntitle: \"Using IbisML and DuckDB for a Kaggle competition: credit risk model stability\"\nauthor: \"Jiting Xu\"\ndate: \"2024-08-22\"\ncategories:\n - blog\n - duckdb\n - machine learning\n - feature engineering\n---\n\n## Introduction\nIn this post, we'll demonstrate how to use Ibis and [IbisML](https://github.com/ibis-project/ibis-ml)\nend-to-end for the\n[credit risk model stability Kaggle competition](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability).\n\n1. Load data and perform feature engineering on DuckDB backend using IbisML\n2. Perform last-mile ML data preprocessing on DuckDB backend using IbisML\n3. Train two models using different frameworks:\n * An XGBoost model within a scikit-learn pipeline.\n * A neural network with PyTorch and PyTorch Lightning.\n\nThe aim of this competition is to predict which clients are more likely to default on their\nloans by using both internal and external data sources.\n\nTo get started with Ibis and IbisML, please refer to the websites:\n\n* [Ibis](https://ibis-project.org/): An open-source dataframe library that works with any data system.\n* [IbisML](https://ibis-project.github.io/ibis-ml/): A library for building scalable ML pipelines.\n\n\n## Prerequisites\nTo run this example, you'll need to download the data from Kaggle website with a Kaggle user account and install Ibis, IbisML, and the necessary modeling library.\n\n### Download data\nYou need a Kaggle account to download the data. If you do not have one,\nfeel free to register one.\n\n1. Option 1: Manual download\n * Log into your Kaggle account and download all data from this\n [link](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data),\n unzip the files, and save them to your local disk.\n2. Option 2: Kaggle API\n * Go to your `Kaggle Account Settings`.\n * Under the `API` section, click on `Create New API Token`. This will download the `kaggle.json`\n file to your computer.\n * Place the `kaggle.json` file in the correct directory, normally it is under your home directory\n `~/.kaggle`:\n\n ```bash\n mkdir ~/.kaggle\n mv ~/Downloads/kaggle.json ~/.kaggle\n ```\n * Install Kaggle CLI and download the data:\n\n ```bash\n pip install kaggle\n kaggle competitions download -c home-credit-credit-risk-model-stability\n unzip home-credit-credit-risk-model-stability.zip\n ```\n\n### Install libraries\nTo use Ibis and IbisML with the DuckDB backend for building models, you'll need to install the\nnecessary packages. Depending on your preferred machine learning framework, you can choose\none of the following installation commands:\n\nFor PyTorch-based models:\n\n```{.bash}\npip install 'ibis-framework[duckdb]' ibis-ml torch pytorch-lightning\n```\n\nFor XGBoost and scikit-learn-based models:\n\n```{.bash}\npip install 'ibis-framework[duckdb]' ibis-ml xgboost[scikit-learn]\n```\n\nImport libraries:\n\n::: {#dba3e08c .cell execution_count=2}\n``` {.python .cell-code}\nimport ibis\nimport ibis.expr.datatypes as dt\nfrom ibis import _\nimport ibis_ml as ml\nfrom pathlib import Path\nfrom glob import glob\n\n# enable interactive mode for ibis\nibis.options.interactive = True\n```\n:::\n\n\nSet the backend for computing:\n\n::: {#d4b26ac9 .cell execution_count=3}\n``` {.python .cell-code}\ncon = ibis.duckdb.connect()\n# remove the black bars from duckdb's progress bar\ncon.raw_sql(\"set enable_progress_bar = false\")\n# DuckDB is the default backend for Ibis\nibis.set_backend(con)\n```\n:::\n\n\nSet data path:\n\n::: {#9930c4ad .cell execution_count=4}\n``` {.python .cell-code}\n# change the root path to yours\nROOT = Path(\"/Users/claypot/Downloads/home-credit-credit-risk-model-stability\")\nTRAIN_DIR = ROOT / \"parquet_files\" / \"train\"\nTEST_DIR = ROOT / \"parquet_files\" / \"test\"\n```\n:::\n\n\n## Data loading and processing\nWe'll use Ibis to read the Parquet files and perform the necessary processing for the next step.\n\n### Directory structure and tables\nSince there are many data files, let's start by examining the directory structure and\ntables within the train directory:\n\n```bash\n# change this to your directory\ntree -L 2 ~/Downloads/home-credit-credit-risk-model-stability/parquet_files/train\n```\n\n:::{.callout-note title=\"Click to show data files\" collapse=\"true\"}\n\n```bash\n~/Downloads/home-credit-credit-risk-model-stability/parquet_files/train\n├── train_applprev_1_0.parquet\n├── train_applprev_1_1.parquet\n├── train_applprev_2.parquet\n├── train_base.parquet\n├── train_credit_bureau_a_1_0.parquet\n├── train_credit_bureau_a_1_1.parquet\n├── train_credit_bureau_a_1_3.parquet\n├── train_credit_bureau_a_2_0.parquet\n├── train_credit_bureau_a_2_1.parquet\n├── train_credit_bureau_a_2_10.parquet\n├── train_credit_bureau_a_2_2.parquet\n├── train_credit_bureau_a_2_3.parquet\n├── train_credit_bureau_a_2_4.parquet\n├── train_credit_bureau_a_2_5.parquet\n├── train_credit_bureau_a_2_6.parquet\n├── train_credit_bureau_a_2_7.parquet\n├── train_credit_bureau_a_2_8.parquet\n├── train_credit_bureau_a_2_9.parquet\n├── train_credit_bureau_b_1.parquet\n├── train_credit_bureau_b_2.parquet\n├── train_debitcard_1.parquet\n├── train_deposit_1.parquet\n├── train_other_1.parquet\n├── train_person_1.parquet\n├── train_person_2.parquet\n├── train_static_0_0.parquet\n├── train_static_0_1.parquet\n├── train_static_cb_0.parquet\n├── train_tax_registry_a_1.parquet\n├── train_tax_registry_b_1.parquet\n└── train_tax_registry_c_1.parquet\n```\n\n:::\n\nThe `train_base.parquet` file is the base table, while the others are feature tables.\nLet's take a quick look at these tables.\n\n#### Base table\nThe base table (`train_base.parquet`) contains the unique ID, a binary target flag\nand other information for the training samples. This unique ID will serve as the\nlinking key for joining with other feature tables.\n\n* `case_id` - This is the unique ID for each loan. You'll need this ID to\n join feature tables to the base table. There are about 1.5m unique loans.\n* `date_decision` - This refers to the date when a decision was made regarding the\n approval of the loan.\n* `WEEK_NUM` - This is the week number used for aggregation. In the test sample,\n `WEEK_NUM` continues sequentially from the last training value of `WEEK_NUM`.\n* `MONTH` - This column represents the month when the approval decision was made.\n* `target` - This is the binary target flag, determined after a certain period based on\n whether or not the client defaulted on the specific loan.\n\nHere is several examples from the base table:\n\n::: {#54d40a5b .cell execution_count=5}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to get the top 5 rows of base table\"}\nibis.read_parquet(TRAIN_DIR / \"train_base.parquet\").head(5)\n```\n\n::: {.cell-output .cell-output-display execution_count=36}\n```{=html}\n
┏━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┓\n┃ case_id ┃ date_decision ┃ MONTH  ┃ WEEK_NUM ┃ target ┃\n┡━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━┩\n│ int64   │ string        │ int64  │ int64    │ int64  │\n├─────────┼───────────────┼────────┼──────────┼────────┤\n│       0 │ 2019-01-03    │ 201901 │        0 │      0 │\n│       1 │ 2019-01-03    │ 201901 │        0 │      0 │\n│       2 │ 2019-01-04    │ 201901 │        0 │      0 │\n│       3 │ 2019-01-03    │ 201901 │        0 │      0 │\n│       4 │ 2019-01-04    │ 201901 │        0 │      1 │\n└─────────┴───────────────┴────────┴──────────┴────────┘\n
\n```\n:::\n:::\n\n\n#### Feature tables\nThe remaining files contain features, consisting of approximately 370 features from\nprevious loan applications and external data sources. Their definitions can be found in the feature\ndefinition [file](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data)\nfrom the competition website.\n\nThere are several things we want to mention for the feature tables:\n\n* **Union datasets**: One dataset could be saved into multiple parquet files, such as\n`train_applprev_1_0.parquet` and `train_applprev_1_1.parquet`, We need to union this data.\n* **Dataset levels**: Datasets may have different levels, which we will explain as\nfollows:\n * **Depth = 0**: Each row in the table is identified by a unique `case_id`.\n In this case, you can directly join the features with the base table and use them as\n features for further analysis or processing.\n * **Depth > 0**: You will group the data based on the `case_id` and perform calculations\n or aggregations within each group.\n\nHere are two examples of tables with different levels.\n\nExample of table with depth = 0, `case_id` is the row identifier, features can be directly joined\n with the base table.\n\n::: {#fdaf873c .cell execution_count=6}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to get the top 5 rows of user static data\"}\nibis.read_parquet(TRAIN_DIR / \"train_static_cb_0.parquet\").head(5)\n```\n\n::: {.cell-output .cell-output-display execution_count=37}\n```{=html}\n
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┓\n┃ case_id ┃ assignmentdate_238D ┃ assignmentdate_4527235D ┃ assignmentdate_4955616D ┃ birthdate_574D ┃ contractssum_5085716L ┃ dateofbirth_337D ┃ dateofbirth_342D ┃ days120_123L ┃ days180_256L ┃ days30_165L ┃ days360_512L ┃ days90_310L ┃ description_5085714M ┃ education_1103M ┃ education_88M ┃ firstquarter_103L ┃ for3years_128L ┃ for3years_504L ┃ for3years_584L ┃ formonth_118L ┃ formonth_206L ┃ formonth_535L ┃ forquarter_1017L ┃ forquarter_462L ┃ forquarter_634L ┃ fortoday_1092L ┃ forweek_1077L ┃ forweek_528L ┃ forweek_601L ┃ foryear_618L ┃ foryear_818L ┃ foryear_850L ┃ fourthquarter_440L ┃ maritalst_385M ┃ maritalst_893M ┃ numberofqueries_373L ┃ pmtaverage_3A ┃ pmtaverage_4527227A ┃ pmtaverage_4955615A ┃ pmtcount_4527229L ┃ pmtcount_4955617L ┃ pmtcount_693L ┃ pmtscount_423L ┃ pmtssum_45A ┃ requesttype_4525192L ┃ responsedate_1012D ┃ responsedate_4527233D ┃ responsedate_4917613D ┃ riskassesment_302T ┃ riskassesment_940T ┃ secondquarter_766L ┃ thirdquarter_1082L ┃\n┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━┩\n│ int64   │ string              │ string                  │ string                  │ string         │ float64               │ string           │ string           │ float64      │ float64      │ float64     │ float64      │ float64     │ string               │ string          │ string        │ float64           │ float64        │ float64        │ float64        │ float64       │ float64       │ float64       │ float64          │ float64         │ float64         │ float64        │ float64       │ float64      │ float64      │ float64      │ float64      │ float64      │ float64            │ string         │ string         │ float64              │ float64       │ float64             │ float64             │ float64           │ float64           │ float64       │ float64        │ float64     │ string               │ string             │ string                │ string                │ string             │ float64            │ float64            │ float64            │\n├─────────┼─────────────────────┼─────────────────────────┼─────────────────────────┼────────────────┼───────────────────────┼──────────────────┼──────────────────┼──────────────┼──────────────┼─────────────┼──────────────┼─────────────┼──────────────────────┼─────────────────┼───────────────┼───────────────────┼────────────────┼────────────────┼────────────────┼───────────────┼───────────────┼───────────────┼──────────────────┼─────────────────┼─────────────────┼────────────────┼───────────────┼──────────────┼──────────────┼──────────────┼──────────────┼──────────────┼────────────────────┼────────────────┼────────────────┼──────────────────────┼───────────────┼─────────────────────┼─────────────────────┼───────────────────┼───────────────────┼───────────────┼────────────────┼─────────────┼──────────────────────┼────────────────────┼───────────────────────┼───────────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────────┤\n│     357 │ NULL                │ NULL                    │ NULL                    │ 1988-04-01     │                  NULL │ NULL             │ NULL             │         NULL │         NULL │        NULL │         NULL │        NULL │ a55475b1             │ a55475b1        │ a55475b1      │              NULL │           NULL │           NULL │           NULL │          NULL │          NULL │          NULL │             NULL │            NULL │            NULL │           NULL │          NULL │         NULL │         NULL │         NULL │         NULL │         NULL │               NULL │ a55475b1       │ a55475b1       │                 NULL │          NULL │                NULL │                NULL │              NULL │              NULL │          NULL │            6.0 │   6301.4000 │ NULL                 │ 2019-01-25         │ NULL                  │ NULL                  │ NULL               │               NULL │               NULL │               NULL │\n│     381 │ NULL                │ NULL                    │ NULL                    │ 1973-11-01     │                  NULL │ NULL             │ NULL             │         NULL │         NULL │        NULL │         NULL │        NULL │ a55475b1             │ a55475b1        │ a55475b1      │              NULL │           NULL │           NULL │           NULL │          NULL │          NULL │          NULL │             NULL │            NULL │            NULL │           NULL │          NULL │         NULL │         NULL │         NULL │         NULL │         NULL │               NULL │ a55475b1       │ a55475b1       │                 NULL │          NULL │                NULL │                NULL │              NULL │              NULL │          NULL │            6.0 │   4019.6000 │ NULL                 │ 2019-01-25         │ NULL                  │ NULL                  │ NULL               │               NULL │               NULL │               NULL │\n│     388 │ NULL                │ NULL                    │ NULL                    │ 1989-04-01     │                  NULL │ 1989-04-01       │ NULL             │          6.0 │          8.0 │         2.0 │         10.0 │         4.0 │ a55475b1             │ a55475b1        │ a55475b1      │               2.0 │           NULL │           NULL │           NULL │          NULL │          NULL │          NULL │             NULL │            NULL │            NULL │           NULL │          NULL │         NULL │         NULL │         NULL │         NULL │         NULL │                6.0 │ a55475b1       │ a55475b1       │                 10.0 │          NULL │                NULL │                NULL │              NULL │              NULL │          NULL │            6.0 │  14548.0000 │ NULL                 │ 2019-01-28         │ NULL                  │ NULL                  │ NULL               │               NULL │                3.0 │                5.0 │\n│     405 │ NULL                │ NULL                    │ NULL                    │ 1974-03-01     │                  NULL │ 1974-03-01       │ NULL             │          0.0 │          0.0 │         0.0 │          1.0 │         0.0 │ a55475b1             │ a55475b1        │ a55475b1      │               0.0 │           NULL │           NULL │           NULL │          NULL │          NULL │          NULL │             NULL │            NULL │            NULL │           NULL │          NULL │         NULL │         NULL │         NULL │         NULL │         NULL │                4.0 │ a55475b1       │ a55475b1       │                  1.0 │          NULL │                NULL │                NULL │              NULL │              NULL │          NULL │            6.0 │  10498.2400 │ NULL                 │ 2019-01-21         │ NULL                  │ NULL                  │ NULL               │               NULL │                2.0 │                0.0 │\n│     409 │ NULL                │ NULL                    │ NULL                    │ 1993-06-01     │                  NULL │ 1993-06-01       │ NULL             │          2.0 │          3.0 │         0.0 │          3.0 │         1.0 │ a55475b1             │ 717ddd49        │ a55475b1      │               4.0 │           NULL │           NULL │           NULL │          NULL │          NULL │          NULL │             NULL │            NULL │            NULL │           NULL │          NULL │         NULL │         NULL │         NULL │         NULL │         NULL │                1.0 │ a7fcb6e5       │ a55475b1       │                  3.0 │          NULL │                NULL │                NULL │              NULL │              NULL │          NULL │            7.0 │   6344.8804 │ NULL                 │ 2019-01-21         │ NULL                  │ NULL                  │ NULL               │               NULL │                0.0 │                4.0 │\n└─────────┴─────────────────────┴─────────────────────────┴─────────────────────────┴────────────────┴───────────────────────┴──────────────────┴──────────────────┴──────────────┴──────────────┴─────────────┴──────────────┴─────────────┴──────────────────────┴─────────────────┴───────────────┴───────────────────┴────────────────┴────────────────┴────────────────┴───────────────┴───────────────┴───────────────┴──────────────────┴─────────────────┴─────────────────┴────────────────┴───────────────┴──────────────┴──────────────┴──────────────┴──────────────┴──────────────┴────────────────────┴────────────────┴────────────────┴──────────────────────┴───────────────┴─────────────────────┴─────────────────────┴───────────────────┴───────────────────┴───────────────┴────────────────┴─────────────┴──────────────────────┴────────────────────┴───────────────────────┴───────────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────────┘\n
\n```\n:::\n:::\n\n\nExample of a table with depth = 1, we need to aggregate the features and collect statistics\nbased on `case_id` then join with the base table.\n\n::: {#4765269b .cell execution_count=7}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to get the top 5 rows of credit bureau data\"}\nibis.read_parquet(TRAIN_DIR / \"train_credit_bureau_b_1.parquet\").relocate(\n \"num_group1\"\n).order_by([\"case_id\", \"num_group1\"]).head(5)\n```\n\n::: {.cell-output .cell-output-display execution_count=38}\n```{=html}\n
┏━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓\n┃ num_group1 ┃ case_id ┃ amount_1115A ┃ classificationofcontr_1114M ┃ contractdate_551D ┃ contractmaturitydate_151D ┃ contractst_516M ┃ contracttype_653M ┃ credlmt_1052A ┃ credlmt_228A ┃ credlmt_3940954A ┃ credor_3940957M ┃ credquantity_1099L ┃ credquantity_984L ┃ debtpastduevalue_732A ┃ debtvalue_227A ┃ dpd_550P ┃ dpd_733P ┃ dpdmax_851P ┃ dpdmaxdatemonth_804T ┃ dpdmaxdateyear_742T ┃ installmentamount_644A ┃ installmentamount_833A ┃ instlamount_892A ┃ interesteffectiverate_369L ┃ interestrateyearly_538L ┃ lastupdate_260D ┃ maxdebtpduevalodued_3940955A ┃ numberofinstls_810L ┃ overdueamountmax_950A ┃ overdueamountmaxdatemonth_494T ┃ overdueamountmaxdateyear_432T ┃ periodicityofpmts_997L ┃ periodicityofpmts_997M ┃ pmtdaysoverdue_1135P ┃ pmtmethod_731M ┃ pmtnumpending_403L ┃ purposeofcred_722M ┃ residualamount_1093A ┃ residualamount_127A ┃ residualamount_3940956A ┃ subjectrole_326M ┃ subjectrole_43M ┃ totalamount_503A ┃ totalamount_881A ┃\n┡━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩\n│ int64      │ int64   │ float64      │ string                      │ string            │ string                    │ string          │ string            │ float64       │ float64      │ float64          │ string          │ float64            │ float64           │ float64               │ float64        │ float64  │ float64  │ float64     │ float64              │ float64             │ float64                │ float64                │ float64          │ float64                    │ float64                 │ string          │ float64                      │ float64             │ float64               │ float64                        │ float64                       │ string                 │ string                 │ float64              │ string         │ float64            │ string             │ float64              │ float64             │ float64                 │ string           │ string          │ float64          │ float64          │\n├────────────┼─────────┼──────────────┼─────────────────────────────┼───────────────────┼───────────────────────────┼─────────────────┼───────────────────┼───────────────┼──────────────┼──────────────────┼─────────────────┼────────────────────┼───────────────────┼───────────────────────┼────────────────┼──────────┼──────────┼─────────────┼──────────────────────┼─────────────────────┼────────────────────────┼────────────────────────┼──────────────────┼────────────────────────────┼─────────────────────────┼─────────────────┼──────────────────────────────┼─────────────────────┼───────────────────────┼────────────────────────────────┼───────────────────────────────┼────────────────────────┼────────────────────────┼──────────────────────┼────────────────┼────────────────────┼────────────────────┼──────────────────────┼─────────────────────┼─────────────────────────┼──────────────────┼─────────────────┼──────────────────┼──────────────────┤\n│          0 │     467 │         NULL │ ea6782cc                    │ 2011-06-15        │ 2031-06-13                │ 7241344e        │ 724be82a          │  3.000000e+06 │      10000.0 │     3.000000e+06 │ P164_34_168     │                2.0 │               1.0 │                  NULL │           NULL │      0.0 │      0.0 │        NULL │                 NULL │                NULL │                    0.0 │                  0.000 │             NULL │                       NULL │                    NULL │ 2019-01-20      │                         NULL │                NULL │                  NULL │                           NULL │                          NULL │ NULL                   │ a55475b1               │                 NULL │ a55475b1       │               NULL │ 96a8fdfe           │                  0.0 │                 0.0 │                    NULL │ fa4f56f1         │ ab3c25cf        │     3.000000e+06 │          10000.0 │\n│          1 │     467 │         NULL │ ea6782cc                    │ 2019-01-04        │ 2021-08-04                │ 7241344e        │ 724be82a          │          NULL │         NULL │     1.303650e+05 │ P164_34_168     │                1.0 │               2.0 │                  NULL │           NULL │      0.0 │      0.0 │        NULL │                 NULL │                NULL │                    0.0 │              26571.969 │             NULL │                       NULL │                    NULL │ 2019-01-20      │                         NULL │                NULL │                  NULL │                           NULL │                          NULL │ NULL                   │ a55475b1               │                 NULL │ a55475b1       │               NULL │ 96a8fdfe           │                 NULL │                NULL │                    NULL │ ab3c25cf         │ ab3c25cf        │     7.800000e+04 │         960000.0 │\n│          2 │     467 │      78000.0 │ ea6782cc                    │ 2016-10-25        │ 2019-10-25                │ 7241344e        │ 4257cbed          │          NULL │         NULL │             NULL │ c5a72b57        │               NULL │              NULL │                   0.0 │      26571.969 │     NULL │     NULL │         0.0 │                 11.0 │              2016.0 │                   NULL │                   NULL │          2898.76 │                       NULL │                    NULL │ 2019-01-10      │                          0.0 │                36.0 │                   0.0 │                           11.0 │                        2016.0 │ NULL                   │ a0b598e4               │                  0.0 │ e914c86c       │               10.0 │ 96a8fdfe           │                 NULL │                NULL │                    NULL │ a55475b1         │ a55475b1        │             NULL │             NULL │\n│          0 │    1445 │         NULL │ ea6782cc                    │ 2015-01-30        │ 2021-01-30                │ 7241344e        │ 1c9c5356          │  4.000000e+05 │     100000.0 │     7.400000e+04 │ b619fa46        │                2.0 │               5.0 │                   0.0 │           NULL │      0.0 │      0.0 │    200418.0 │                  1.0 │              2018.0 │                    0.0 │                  0.000 │             NULL │                       NULL │                    NULL │ 2019-01-19      │                          0.4 │                NULL │                   1.4 │                            2.0 │                        2018.0 │ NULL                   │ a55475b1               │                  0.0 │ a55475b1       │               NULL │ 60c73645           │                  0.0 │                 0.0 │                73044.18 │ daf49a8a         │ ab3c25cf        │     4.000000e+05 │         100000.0 │\n│          1 │    1445 │         NULL │ 01f63ac8                    │ 2014-09-12        │ 2021-09-12                │ 7241344e        │ 724be82a          │          NULL │         NULL │     4.000000e+05 │ 74bd67a8        │                3.0 │              17.0 │                  NULL │           NULL │      0.0 │      0.0 │        NULL │                 NULL │                NULL │                    0.0 │             209617.770 │             NULL │                       NULL │                    NULL │ 2019-01-13      │                         NULL │                NULL │                  NULL │                           NULL │                          NULL │ NULL                   │ a55475b1               │                 NULL │ a55475b1       │               NULL │ 96a8fdfe           │                 NULL │                NULL │                    NULL │ ab3c25cf         │ ab3c25cf        │     3.968006e+05 │         184587.8 │\n└────────────┴─────────┴──────────────┴─────────────────────────────┴───────────────────┴───────────────────────────┴─────────────────┴───────────────────┴───────────────┴──────────────┴──────────────────┴─────────────────┴────────────────────┴───────────────────┴───────────────────────┴────────────────┴──────────┴──────────┴─────────────┴──────────────────────┴─────────────────────┴────────────────────────┴────────────────────────┴──────────────────┴────────────────────────────┴─────────────────────────┴─────────────────┴──────────────────────────────┴─────────────────────┴───────────────────────┴────────────────────────────────┴───────────────────────────────┴────────────────────────┴────────────────────────┴──────────────────────┴────────────────┴────────────────────┴────────────────────┴──────────────────────┴─────────────────────┴─────────────────────────┴──────────────────┴─────────────────┴──────────────────┴──────────────────┘\n
\n```\n:::\n:::\n\n\nFor more details on features and its exploratory data analysis (EDA), you can refer to\nfeature definition and these Kaggle notebooks:\n\n* [Feature\n definition](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data#:~:text=calendar_view_week-,feature_definitions,-.csv)\n* [Home credit risk prediction\n EDA](https://www.kaggle.com/code/loki97/home-credit-risk-prediction-eda)\n* [Home credit CRMS 2024\n EDA](https://www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission)\n\n### Data loading and processing\nWe will perform the following data processing steps using Ibis and IbisML:\n\n* **Convert data types**: Ensure consistency by converting data types, as the same column\n in different sub-files may have different types.\n* **Aggregate features**: For tables with depth greater than 0, aggregate features based\n on `case_id`, including statistics calculation. You can collect statistics such as mean,\n median, mode, minimum, standard deviation, and others.\n* **Union and join datasets**: Combine multiple sub-files of the same dataset into one\n table, as some datasets are split into multiple sub-files with a common prefix. Afterward,\n join these tables with the base table.\n\n#### Convert data types\nWe'll use IbisML to create a chain of `Cast` steps, forming a recipe for data type\nconversion across the dataset. This conversion is based on the provided information\nextracted from column names. Columns that have similar transformations are indicated by a\ncapital letter at the end of their names:\n\n* P - Transform DPD (Days past due)\n* M - Masking categories\n* A - Transform amount\n* D - Transform date\n* T - Unspecified Transform\n* L - Unspecified Transform\n\nFor example, we'll define a IbisML transformation step to convert columns ends with `P`\nto floating number:\n\n::: {#92f70762 .cell execution_count=8}\n``` {.python .cell-code}\n# convert columns ends with P to floating number\nstep_cast_P_to_float = ml.Cast(ml.endswith(\"P\"), dt.float64)\n```\n:::\n\n\nNext, let's define additional type conversion transformations based on the postfix of column names:\n\n::: {#3e3be631 .cell execution_count=9}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to define more steps\"}\n# convert columns ends with A to floating number\nstep_cast_A_to_float = ml.Cast(ml.endswith(\"A\"), dt.float64)\n# convert columns ends with D to date\nstep_cast_D_to_date = ml.Cast(ml.endswith(\"D\"), dt.date)\n# convert columns ends with M to str\nstep_cast_M_to_str = ml.Cast(ml.endswith(\"M\"), dt.str)\n```\n:::\n\n\nWe'll construct the\n[IbisML Recipe](https://ibis-project.github.io/ibis-ml/reference/core.html#ibis_ml.Recipe)\nwhich chains together all the transformation steps.\n\n::: {#a275dbfd .cell execution_count=10}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to construct the recipe\"}\ndata_type_recipes = ml.Recipe(\n step_cast_P_to_float,\n step_cast_D_to_date,\n step_cast_M_to_str,\n step_cast_A_to_float,\n # cast some special columns\n ml.Cast([\"date_decision\"], \"date\"),\n ml.Cast([\"case_id\", \"WEEK_NUM\", \"num_group1\", \"num_group2\"], dt.int64),\n ml.Cast(\n [\n \"cardtype_51L\",\n \"credacc_status_367L\",\n \"requesttype_4525192L\",\n \"riskassesment_302T\",\n \"max_periodicityofpmts_997L\",\n ],\n dt.str,\n ),\n ml.Cast(\n [\n \"isbidproductrequest_292L\",\n \"isdebitcard_527L\",\n \"equalityempfrom_62L\",\n ],\n dt.int64,\n ),\n)\nprint(f\"Data format conversion recipe:\\n{data_type_recipes}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nData format conversion recipe:\nRecipe(Cast(endswith('P'), 'float64'),\n Cast(endswith('D'), 'date'),\n Cast(endswith('M'), 'string'),\n Cast(endswith('A'), 'float64'),\n Cast(cols(('date_decision',)), 'date'),\n Cast(cols(('case_id', 'WEEK_NUM', 'num_group1', 'num_group2')), 'int64'),\n Cast(cols(('cardtype_51L', 'credacc_status_367L', 'requesttype_4525192L', 'riskassesment_302T', 'max_periodicityofpmts_997L')),\n 'string'),\n Cast(cols(('isbidproductrequest_292L', 'isdebitcard_527L', 'equalityempfrom_62L')),\n 'int64'))\n```\n:::\n:::\n\n\n::: {.callout-tip}\nIbisML offers a powerful set of column selectors, allowing you to select columns based\non names, types, and patterns. For more information, you can refer to the IbisML column\nselectors [documentation](https://ibis-project.github.io/ibis-ml/reference/selectors.html).\n:::\n\n#### Aggregate features\nFor tables with a depth greater than 0 that can't be directly joined with the base table,\nwe need to aggregate the features by the `case_id`. You could compute the different statistics for numeric columns and\nnon-numeric columns.\n\nHere, we use the `maximum` as an example.\n\n::: {#4f274968 .cell execution_count=11}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to aggregate features by case_id using max\"}\ndef agg_by_id(table):\n return table.group_by(\"case_id\").agg(\n [\n table[col_name].max().name(f\"max_{col_name}\")\n for col_name in table.columns\n if col_name[-1] in (\"T\", \"L\", \"P\", \"A\", \"D\", \"M\")\n ]\n )\n```\n:::\n\n\n::: {.callout-tip}\nFor better predicting power, you need to collect different statistics based on the meaning of features. For simplicity,\nwe'll only collect the maximum value of the features here.\n:::\n\n#### Put them together\nWe'll put them together in a function reads parquet files, optionally handles regex patterns for\nmultiple sub-files, applies data type transformations defined by `data_type_recipes`, and\nperforms aggregation based on `case_id` if specified by the depth parameter.\n\n::: {#4116845d .cell execution_count=12}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to read and process data files\"}\ndef read_and_process_files(file_path, depth=None, is_regex=False):\n \"\"\"\n Read and process Parquet files.\n\n Args:\n file_path (str): Path to the file or regex pattern to match files.\n depth (int, optional): Depth of processing. If 1 or 2, additional aggregation is performed.\n is_regex (bool, optional): Whether the file_path is a regex pattern.\n\n Returns:\n ibis.Table: The processed Ibis table.\n \"\"\"\n if is_regex:\n # read and union multiple files\n chunks = []\n for path in glob(str(file_path)):\n chunk = ibis.read_parquet(path)\n # transform table using IbisML Recipe\n chunk = data_type_recipes.fit(chunk).to_ibis(chunk)\n chunks.append(chunk)\n table = ibis.union(*chunks)\n else:\n # read a single file\n table = ibis.read_parquet(file_path)\n # transform table using IbisML\n table = data_type_recipes.fit(table).to_ibis(table)\n\n # perform aggregation if depth is 1 or 2\n if depth in [1, 2]:\n table = agg_by_id(table)\n\n return table\n```\n:::\n\n\nLet's define two dictionaries, `train_data_store` and `test_data_store`, that organize and\nstore processed datasets for training and testing datasets.\n\n::: {#513c202e .cell execution_count=13}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to load all data into a dict\"}\ntrain_data_store = {\n \"df_base\": read_and_process_files(TRAIN_DIR / \"train_base.parquet\"),\n \"depth_0\": [\n read_and_process_files(TRAIN_DIR / \"train_static_cb_0.parquet\"),\n read_and_process_files(TRAIN_DIR / \"train_static_0_*.parquet\", is_regex=True),\n ],\n \"depth_1\": [\n read_and_process_files(\n TRAIN_DIR / \"train_applprev_1_*.parquet\", 1, is_regex=True\n ),\n read_and_process_files(TRAIN_DIR / \"train_tax_registry_a_1.parquet\", 1),\n read_and_process_files(TRAIN_DIR / \"train_tax_registry_b_1.parquet\", 1),\n read_and_process_files(TRAIN_DIR / \"train_tax_registry_c_1.parquet\", 1),\n read_and_process_files(TRAIN_DIR / \"train_credit_bureau_b_1.parquet\", 1),\n read_and_process_files(TRAIN_DIR / \"train_other_1.parquet\", 1),\n read_and_process_files(TRAIN_DIR / \"train_person_1.parquet\", 1),\n read_and_process_files(TRAIN_DIR / \"train_deposit_1.parquet\", 1),\n read_and_process_files(TRAIN_DIR / \"train_debitcard_1.parquet\", 1),\n ],\n \"depth_2\": [\n read_and_process_files(TRAIN_DIR / \"train_credit_bureau_b_2.parquet\", 2),\n ],\n}\n# we won't be submitting the predictions, so let's comment out the test data.\n# test_data_store = {\n# \"df_base\": read_and_process_files(TEST_DIR / \"test_base.parquet\"),\n# \"depth_0\": [\n# read_and_process_files(TEST_DIR / \"test_static_cb_0.parquet\"),\n# read_and_process_files(TEST_DIR / \"test_static_0_*.parquet\", is_regex=True),\n# ],\n# \"depth_1\": [\n# read_and_process_files(TEST_DIR / \"test_applprev_1_*.parquet\", 1, is_regex=True),\n# read_and_process_files(TEST_DIR / \"test_tax_registry_a_1.parquet\", 1),\n# read_and_process_files(TEST_DIR / \"test_tax_registry_b_1.parquet\", 1),\n# read_and_process_files(TEST_DIR / \"test_tax_registry_c_1.parquet\", 1),\n# read_and_process_files(TEST_DIR / \"test_credit_bureau_b_1.parquet\", 1),\n# read_and_process_files(TEST_DIR / \"test_other_1.parquet\", 1),\n# read_and_process_files(TEST_DIR / \"test_person_1.parquet\", 1),\n# read_and_process_files(TEST_DIR / \"test_deposit_1.parquet\", 1),\n# read_and_process_files(TEST_DIR / \"test_debitcard_1.parquet\", 1),\n# ],\n# \"depth_2\": [\n# read_and_process_files(TEST_DIR / \"test_credit_bureau_b_2.parquet\", 2),\n# ]\n# }\n```\n:::\n\n\nJoin all features data to base table:\n\n::: {#93211f82 .cell execution_count=14}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Define function to join feature tables to base table\"}\ndef join_data(df_base, depth_0, depth_1, depth_2):\n for i, df in enumerate(depth_0 + depth_1 + depth_2):\n df_base = df_base.join(\n df, \"case_id\", how=\"left\", rname=\"{name}_right\" + f\"_{i}\"\n )\n return df_base\n```\n:::\n\n\nGenerate train and test datasets:\n\n::: {#3b26eaf7 .cell execution_count=15}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to generate train and test datasets\"}\ndf_train = join_data(**train_data_store)\n# df_test = join_data(**test_data_store)\ntotal_rows = df_train.count().execute()\nprint(f\"There is {total_rows} rows and {len(df_train.columns)} columns\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nThere is 1526659 rows and 377 columns\n```\n:::\n:::\n\n\n### Select features\nGiven the large number of features (~370), we'll focus on selecting just a few of the most\ninformative ones by name for demonstration purposes in this post:\n\n::: {#36e0ec8e .cell execution_count=16}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to select important features for the train dataset\"}\ndf_train = df_train.select(\n \"case_id\",\n \"date_decision\",\n \"target\",\n # number of credit bureau queries for the last X days.\n \"days30_165L\",\n \"days360_512L\",\n \"days90_310L\",\n # number of tax deduction payments\n \"pmtscount_423L\",\n # sum of tax deductions for the client\n \"pmtssum_45A\",\n \"dateofbirth_337D\",\n \"education_1103M\",\n \"firstquarter_103L\",\n \"secondquarter_766L\",\n \"thirdquarter_1082L\",\n \"fourthquarter_440L\",\n \"maritalst_893M\",\n \"numberofqueries_373L\",\n \"requesttype_4525192L\",\n \"responsedate_4527233D\",\n \"actualdpdtolerance_344P\",\n \"amtinstpaidbefduel24m_4187115A\",\n \"annuity_780A\",\n \"annuitynextmonth_57A\",\n \"applicationcnt_361L\",\n \"applications30d_658L\",\n \"applicationscnt_1086L\",\n # average days past or before due of payment during the last 24 months.\n \"avgdbddpdlast24m_3658932P\",\n # average days past or before due of payment during the last 3 months.\n \"avgdbddpdlast3m_4187120P\",\n # end date of active contract.\n \"max_contractmaturitydate_151D\",\n # credit limit of an active loan.\n \"max_credlmt_1052A\",\n # number of credits in credit bureau\n \"max_credquantity_1099L\",\n \"max_dpdmaxdatemonth_804T\",\n \"max_dpdmaxdateyear_742T\",\n \"max_maxdebtpduevalodued_3940955A\",\n \"max_overdueamountmax_950A\",\n \"max_purposeofcred_722M\",\n \"max_residualamount_3940956A\",\n \"max_totalamount_503A\",\n \"max_cancelreason_3545846M\",\n \"max_childnum_21L\",\n \"max_currdebt_94A\",\n \"max_employedfrom_700D\",\n # client's main income amount in their previous application\n \"max_mainoccupationinc_437A\",\n \"max_profession_152M\",\n \"max_rejectreason_755M\",\n \"max_status_219L\",\n # credit amount of the active contract provided by the credit bureau\n \"max_amount_1115A\",\n # amount of unpaid debt for existing contracts\n \"max_debtpastduevalue_732A\",\n \"max_debtvalue_227A\",\n \"max_installmentamount_833A\",\n \"max_instlamount_892A\",\n \"max_numberofinstls_810L\",\n \"max_pmtnumpending_403L\",\n \"max_last180dayaveragebalance_704A\",\n \"max_last30dayturnover_651A\",\n \"max_openingdate_857D\",\n \"max_amount_416A\",\n \"max_amtdebitincoming_4809443A\",\n \"max_amtdebitoutgoing_4809440A\",\n \"max_amtdepositbalance_4809441A\",\n \"max_amtdepositincoming_4809444A\",\n \"max_amtdepositoutgoing_4809442A\",\n \"max_empl_industry_691L\",\n \"max_gender_992L\",\n \"max_housingtype_772L\",\n \"max_mainoccupationinc_384A\",\n \"max_incometype_1044T\",\n)\n\ndf_train.head()\n```\n\n::: {.cell-output .cell-output-display execution_count=47}\n```{=html}\n
┏━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┓\n┃ case_id ┃ date_decision ┃ target ┃ days30_165L ┃ days360_512L ┃ days90_310L ┃ pmtscount_423L ┃ pmtssum_45A ┃ dateofbirth_337D ┃ education_1103M ┃ firstquarter_103L ┃ secondquarter_766L ┃ thirdquarter_1082L ┃ fourthquarter_440L ┃ maritalst_893M ┃ numberofqueries_373L ┃ requesttype_4525192L ┃ responsedate_4527233D ┃ actualdpdtolerance_344P ┃ amtinstpaidbefduel24m_4187115A ┃ annuity_780A ┃ annuitynextmonth_57A ┃ applicationcnt_361L ┃ applications30d_658L ┃ applicationscnt_1086L ┃ avgdbddpdlast24m_3658932P ┃ avgdbddpdlast3m_4187120P ┃ max_contractmaturitydate_151D ┃ max_credlmt_1052A ┃ max_credquantity_1099L ┃ max_dpdmaxdatemonth_804T ┃ max_dpdmaxdateyear_742T ┃ max_maxdebtpduevalodued_3940955A ┃ max_overdueamountmax_950A ┃ max_purposeofcred_722M ┃ max_residualamount_3940956A ┃ max_totalamount_503A ┃ max_cancelreason_3545846M ┃ max_childnum_21L ┃ max_currdebt_94A ┃ max_employedfrom_700D ┃ max_mainoccupationinc_437A ┃ max_profession_152M ┃ max_rejectreason_755M ┃ max_status_219L ┃ max_amount_1115A ┃ max_debtpastduevalue_732A ┃ max_debtvalue_227A ┃ max_installmentamount_833A ┃ max_instlamount_892A ┃ max_numberofinstls_810L ┃ max_pmtnumpending_403L ┃ max_last180dayaveragebalance_704A ┃ max_last30dayturnover_651A ┃ max_openingdate_857D ┃ max_amount_416A ┃ max_amtdebitincoming_4809443A ┃ max_amtdebitoutgoing_4809440A ┃ max_amtdepositbalance_4809441A ┃ max_amtdepositincoming_4809444A ┃ max_amtdepositoutgoing_4809442A ┃ max_empl_industry_691L ┃ max_gender_992L ┃ max_housingtype_772L ┃ max_mainoccupationinc_384A ┃ max_incometype_1044T    ┃\n┡━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━┩\n│ int64   │ date          │ int64  │ float64     │ float64      │ float64     │ float64        │ float64     │ date             │ string          │ float64           │ float64            │ float64            │ float64            │ string         │ float64              │ string               │ date                  │ float64                 │ float64                        │ float64      │ float64              │ float64             │ float64              │ float64               │ float64                   │ float64                  │ date                          │ float64           │ float64                │ float64                  │ float64                 │ float64                          │ float64                   │ string                 │ float64                     │ float64              │ string                    │ float64          │ float64          │ date                  │ float64                    │ string              │ string                │ string          │ float64          │ float64                   │ float64            │ float64                    │ float64              │ float64                 │ float64                │ float64                           │ float64                    │ date                 │ float64         │ float64                       │ float64                       │ float64                        │ float64                         │ float64                         │ string                 │ string          │ string               │ float64                    │ string                  │\n├─────────┼───────────────┼────────┼─────────────┼──────────────┼─────────────┼────────────────┼─────────────┼──────────────────┼─────────────────┼───────────────────┼────────────────────┼────────────────────┼────────────────────┼────────────────┼──────────────────────┼──────────────────────┼───────────────────────┼─────────────────────────┼────────────────────────────────┼──────────────┼──────────────────────┼─────────────────────┼──────────────────────┼───────────────────────┼───────────────────────────┼──────────────────────────┼───────────────────────────────┼───────────────────┼────────────────────────┼──────────────────────────┼─────────────────────────┼──────────────────────────────────┼───────────────────────────┼────────────────────────┼─────────────────────────────┼──────────────────────┼───────────────────────────┼──────────────────┼──────────────────┼───────────────────────┼────────────────────────────┼─────────────────────┼───────────────────────┼─────────────────┼──────────────────┼───────────────────────────┼────────────────────┼────────────────────────────┼──────────────────────┼─────────────────────────┼────────────────────────┼───────────────────────────────────┼────────────────────────────┼──────────────────────┼─────────────────┼───────────────────────────────┼───────────────────────────────┼────────────────────────────────┼─────────────────────────────────┼─────────────────────────────────┼────────────────────────┼─────────────────┼──────────────────────┼────────────────────────────┼─────────────────────────┤\n│ 1915559 │ 2020-09-02    │      0 │         0.0 │          5.0 │         1.0 │           NULL │        NULL │ 1963-12-01       │ 717ddd49        │               4.0 │                6.0 │                2.0 │                4.0 │ a55475b1       │                  5.0 │ NULL                 │ NULL                  │                     0.0 │                       9490.187 │    1366.6000 │                  0.0 │                 0.0 │                  1.0 │                   0.0 │                      -4.0 │                     NULL │ NULL                          │              NULL │                   NULL │                     NULL │                    NULL │                             NULL │                      NULL │ NULL                   │                        NULL │                 NULL │ a55475b1                  │              1.0 │            0.000 │ 2012-11-15            │                    72000.0 │ a55475b1            │ a55475b1              │ T               │             NULL │                      NULL │               NULL │                       NULL │                 NULL │                    NULL │                   NULL │                              NULL │                       NULL │ NULL                 │            NULL │                          NULL │                          NULL │                           NULL │                            NULL │                            NULL │ NULL                   │ NULL            │ NULL                 │                    50000.0 │ PRIVATE_SECTOR_EMPLOYEE │\n│ 1915592 │ 2020-09-02    │      0 │         0.0 │          6.0 │         1.0 │           NULL │        NULL │ 1983-01-01       │ 6b2ae0fa        │               6.0 │               11.0 │                2.0 │                2.0 │ a55475b1       │                  6.0 │ NULL                 │ NULL                  │                     0.0 │                      61296.600 │    1268.4000 │                  0.0 │                 0.0 │                  0.0 │                   0.0 │                      -6.0 │                     NULL │ NULL                          │              NULL │                   NULL │                     NULL │                    NULL │                             NULL │                      NULL │ NULL                   │                        NULL │                 NULL │ a55475b1                  │              0.0 │            0.000 │ 2013-09-15            │                   199600.0 │ a55475b1            │ a55475b1              │ K               │             NULL │                      NULL │               NULL │                       NULL │                 NULL │                    NULL │                   NULL │                              NULL │                       NULL │ NULL                 │            NULL │                          NULL │                          NULL │                           NULL │                            NULL │                            NULL │ NULL                   │ NULL            │ NULL                 │                    70000.0 │ SALARIED_GOVT           │\n│ 1915605 │ 2020-09-02    │      0 │         0.0 │          1.0 │         0.0 │           NULL │        NULL │ 1977-01-01       │ a55475b1        │               0.0 │                1.0 │                2.0 │                1.0 │ a55475b1       │                  1.0 │ NULL                 │ NULL                  │                     0.0 │                     359920.470 │    8483.2000 │               6434.4 │                 0.0 │                  0.0 │                   0.0 │                     -15.0 │                     -6.0 │ NULL                          │              NULL │                   NULL │                     NULL │                    NULL │                             NULL │                      NULL │ NULL                   │                        NULL │                 NULL │ a55475b1                  │             NULL │        43596.227 │ 2014-01-15            │                   199600.0 │ a55475b1            │ a55475b1              │ T               │             NULL │                      NULL │               NULL │                       NULL │                 NULL │                    NULL │                   NULL │                              NULL │                       NULL │ NULL                 │            NULL │                          NULL │                          NULL │                           NULL │                            NULL │                            NULL │ NULL                   │ NULL            │ NULL                 │                   200000.0 │ SELFEMPLOYED            │\n│ 1915620 │ 2020-09-02    │      0 │         0.0 │          1.0 │         0.0 │           NULL │        NULL │ 1993-04-01       │ a55475b1        │               0.0 │                0.0 │                0.0 │                1.0 │ a55475b1       │                  1.0 │ NULL                 │ NULL                  │                     0.0 │                     129430.370 │    2368.2000 │                  0.0 │                 0.0 │                  0.0 │                   0.0 │                     -24.0 │                     NULL │ NULL                          │              NULL │                   NULL │                     NULL │                    NULL │                             NULL │                      NULL │ NULL                   │                        NULL │                 NULL │ a55475b1                  │             NULL │            0.000 │ 2018-06-15            │                    30000.0 │ a55475b1            │ a55475b1              │ K               │             NULL │                      NULL │               NULL │                       NULL │                 NULL │                    NULL │                   NULL │                              NULL │                       NULL │ NULL                 │            NULL │                          NULL │                          NULL │                           NULL │                            NULL │                            NULL │ NULL                   │ NULL            │ NULL                 │                    24000.0 │ SALARIED_GOVT           │\n│ 1915695 │ 2020-09-02    │      0 │         0.0 │          1.0 │         0.0 │           NULL │        NULL │ 1981-07-01       │ a55475b1        │               0.0 │                0.0 │                0.0 │                1.0 │ a55475b1       │                  1.0 │ NULL                 │ NULL                  │                     0.0 │                      15998.000 │    6839.8003 │                  0.0 │                 0.0 │                  0.0 │                   0.0 │                     -10.0 │                     NULL │ NULL                          │              NULL │                   NULL │                     NULL │                    NULL │                             NULL │                      NULL │ NULL                   │                        NULL │                 NULL │ a55475b1                  │             NULL │            0.000 │ 2016-01-15            │                    40000.0 │ a55475b1            │ a55475b1              │ K               │             NULL │                      NULL │               NULL │                       NULL │                 NULL │                    NULL │                   NULL │                              NULL │                       NULL │ NULL                 │            NULL │                          NULL │                          NULL │                           NULL │                            NULL │                            NULL │ NULL                   │ NULL            │ NULL                 │                    30000.0 │ SALARIED_GOVT           │\n└─────────┴───────────────┴────────┴─────────────┴──────────────┴─────────────┴────────────────┴─────────────┴──────────────────┴─────────────────┴───────────────────┴────────────────────┴────────────────────┴────────────────────┴────────────────┴──────────────────────┴──────────────────────┴───────────────────────┴─────────────────────────┴────────────────────────────────┴──────────────┴──────────────────────┴─────────────────────┴──────────────────────┴───────────────────────┴───────────────────────────┴──────────────────────────┴───────────────────────────────┴───────────────────┴────────────────────────┴──────────────────────────┴─────────────────────────┴──────────────────────────────────┴───────────────────────────┴────────────────────────┴─────────────────────────────┴──────────────────────┴───────────────────────────┴──────────────────┴──────────────────┴───────────────────────┴────────────────────────────┴─────────────────────┴───────────────────────┴─────────────────┴──────────────────┴───────────────────────────┴────────────────────┴────────────────────────────┴──────────────────────┴─────────────────────────┴────────────────────────┴───────────────────────────────────┴────────────────────────────┴──────────────────────┴─────────────────┴───────────────────────────────┴───────────────────────────────┴────────────────────────────────┴─────────────────────────────────┴─────────────────────────────────┴────────────────────────┴─────────────────┴──────────────────────┴────────────────────────────┴─────────────────────────┘\n
\n```\n:::\n:::\n\n\nUnivariate analysis:\n\n::: {#2b1a9226 .cell execution_count=17}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to describe the train dataset\"}\n# take the first 10 columns\ndf_train[df_train.columns[:10]].describe()\n```\n\n::: {.cell-output .cell-output-display execution_count=48}\n```{=html}\n
┏━━━━━━━━━━━━━━━━━┳━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓\n┃ name            ┃ pos   ┃ type    ┃ count   ┃ nulls  ┃ unique  ┃ mode     ┃ mean         ┃ std           ┃ min     ┃ p25         ┃ p50          ┃ p75          ┃ max          ┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩\n│ string          │ int16 │ string  │ int64   │ int64  │ int64   │ string   │ float64      │ float64       │ float64 │ float64     │ float64      │ float64      │ float64      │\n├─────────────────┼───────┼─────────┼─────────┼────────┼─────────┼──────────┼──────────────┼───────────────┼─────────┼─────────────┼──────────────┼──────────────┼──────────────┤\n│ case_id         │     0 │ int64   │ 1526659 │      0 │ 1526659 │ NULL     │ 1.286077e+06 │ 718946.592285 │     0.0 │ 766197.5000 │ 1.357358e+06 │ 1.739022e+06 │ 2.703454e+06 │\n│ target          │     2 │ int64   │ 1526659 │      0 │       2 │ NULL     │ 3.143728e-02 │      0.174496 │     0.0 │      0.0000 │ 0.000000e+00 │ 0.000000e+00 │ 1.000000e+00 │\n│ days30_165L     │     3 │ float64 │ 1526659 │ 140968 │      22 │ NULL     │ 5.177078e-01 │      0.899238 │     0.0 │      0.0000 │ 0.000000e+00 │ 1.000000e+00 │ 2.200000e+01 │\n│ days360_512L    │     4 │ float64 │ 1526659 │ 140968 │      92 │ NULL     │ 4.777066e+00 │      5.168856 │     0.0 │      1.0000 │ 3.000000e+00 │ 6.500000e+00 │ 1.150000e+02 │\n│ days90_310L     │     5 │ float64 │ 1526659 │ 140968 │      37 │ NULL     │ 1.211420e+00 │      1.655931 │     0.0 │      0.0000 │ 1.000000e+00 │ 2.000000e+00 │ 4.100000e+01 │\n│ pmtscount_423L  │     6 │ float64 │ 1526659 │ 954021 │      66 │ NULL     │ 5.839291e+00 │      4.148264 │     0.0 │      3.0000 │ 6.000000e+00 │ 7.000000e+00 │ 1.210000e+02 │\n│ pmtssum_45A     │     7 │ float64 │ 1526659 │ 954021 │  265229 │ NULL     │ 1.319994e+04 │  18117.218312 │     0.0 │   3156.4001 │ 8.391900e+03 │ 1.699200e+04 │ 4.768434e+05 │\n│ education_1103M │     9 │ string  │ 1526659 │  26183 │       5 │ a55475b1 │         NULL │          NULL │    NULL │        NULL │         NULL │         NULL │         NULL │\n└─────────────────┴───────┴─────────┴─────────┴────────┴─────────┴──────────┴──────────────┴───────────────┴─────────┴─────────────┴──────────────┴──────────────┴──────────────┘\n
\n```\n:::\n:::\n\n\n## Last-mile data preprocessing\nWe will perform the following transformation before feeding the data to models:\n\n* Missing value imputation\n* Encoding categorical variables\n* Handling date variables\n* Handling outliers\n* Scaling and normalization\n\n::: {.callout-note}\nIbisML provides a set of transformations. You can find the\n[roadmap](https://github.com/ibis-project/ibis-ml/issues/32).\nThe [IbisML website](https://ibis-project.github.io/ibis-ml/) also includes tutorials and API documentation.\n:::\n\n### Impute features\nImpute all numeric columns using the median. In real-life scenarios, it's important to\nunderstand the meaning of each feature and apply the appropriate imputation method for\ndifferent features. For more imputations, please refer to this\n[documentation](https://ibis-project.github.io/ibis-ml/reference/steps-imputation.html).\n\n::: {#99303c5a .cell execution_count=18}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to impute all numeric columns with median\"}\nstep_impute_median = ml.ImputeMedian(ml.numeric())\n```\n:::\n\n\n### Encode categorical features\nEncode all categorical features using one-hot-encode. For more encoding steps,\nplease refer to this\n[doc](https://ibis-project.github.io/ibis-ml/reference/steps-encoding.html).\n\n::: {#3e1df3b1 .cell execution_count=19}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to one-hot encode selected columns\"}\nohe_step = ml.OneHotEncode(\n [\n \"maritalst_893M\",\n \"requesttype_4525192L\",\n \"max_profession_152M\",\n \"max_gender_992L\",\n \"max_empl_industry_691L\",\n \"max_housingtype_772L\",\n \"max_incometype_1044T\",\n \"max_cancelreason_3545846M\",\n \"max_rejectreason_755M\",\n \"education_1103M\",\n \"max_status_219L\",\n ]\n)\n```\n:::\n\n\n### Handle date variables\nCalculate all the days difference between any date columns and the column `date_decision`:\n\n::: {#cb197a80 .cell execution_count=20}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to calculate days difference between date columns and date_decision\"}\ndate_cols = [col_name for col_name in df_train.columns if col_name[-1] == \"D\"]\ndays_to_decision_expr = {\n # difference in days\n f\"{col}_date_decision_diff\": (\n _.date_decision.epoch_seconds() - getattr(_, col).epoch_seconds()\n )\n / (60 * 60 * 24)\n for col in date_cols\n}\ndays_to_decision_step = ml.Mutate(days_to_decision_expr)\n```\n:::\n\n\nExtract information from the date columns:\n\n::: {#63129062 .cell execution_count=21}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to extract day and week info from date columns\"}\n# dow and month is set to catagoery\nexpand_date_step = ml.ExpandDate(ml.date(), [\"week\", \"day\"])\n```\n:::\n\n\n### Handle outliers\nCapping outliers using `z-score` method:\n\n::: {#fe15771d .cell execution_count=22}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to cap outliers for selected columns\"}\nstep_handle_outliers = ml.HandleUnivariateOutliers(\n [\"max_amount_1115A\", \"max_overdueamountmax_950A\"],\n method=\"z-score\",\n treatment=\"capping\",\n deviation_factor=3,\n)\n```\n:::\n\n\n### Construct recipe\nWe'll construct the last mile preprocessing [recipe](https://ibis-project.github.io/ibis-ml/reference/core.html#ibis_ml.Recipe)\nby chaining all transformation steps, which will be fitted to the training dataset and later applied test datasets.\n\n::: {#fd8a067a .cell execution_count=23}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to construct the recipe\"}\nlast_mile_preprocessing = ml.Recipe(\n expand_date_step,\n ml.Drop(ml.date()),\n # handle string columns\n ohe_step,\n ml.Drop(ml.string()),\n # handle numeric cols\n # capping outliers\n step_handle_outliers,\n step_impute_median,\n ml.ScaleMinMax(ml.numeric()),\n # fill missing value\n ml.FillNA(ml.numeric(), 0),\n ml.Cast(ml.numeric(), \"float32\"),\n)\nprint(f\"Last-mile preprocessing recipe: \\n{last_mile_preprocessing}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nLast-mile preprocessing recipe: \nRecipe(ExpandDate(date(), components=['week', 'day']),\n Drop(date()),\n OneHotEncode(cols(('maritalst_893M', 'requesttype_4525192L', 'max_profession_152M', 'max_gender_992L', 'max_empl_industry_691L', 'max_housingtype_772L', 'max_incometype_1044T', 'max_cancelreason_3545846M', 'max_rejectreason_755M', 'education_1103M', 'max_status_219L'))),\n Drop(string()),\n HandleUnivariateOutliers(cols(('max_amount_1115A', 'max_overdueamountmax_950A')),\n method='z-score',\n deviation_factor=3,\n treatment='capping'),\n ImputeMedian(numeric()),\n ScaleMinMax(numeric()),\n FillNA(numeric(), 0),\n Cast(numeric(), 'float32'))\n```\n:::\n:::\n\n\n## Modeling\nAfter completing data preprocessing with Ibis and IbisML, we proceed to the modeling\nphase. Here are two approaches:\n\n* Use IbisML as a independent data preprocessing component and hand off the data to downstream modeling\nframeworks with various output formats:\n - pandas Dataframe\n - NumPy Array\n - Polars Dataframe\n - Dask Dataframe\n - xgboost.DMatrix\n - Pyarrow Table\n* Use IbisML recipes as components within an sklearn Pipeline and\ntrain models similarly to how you would do with sklearn pipeline.\n\nWe will build an XGBoost model within a scikit-learn pipeline, and a neural network classifier using the\noutput transformed by IbisML recipes.\n\n### Train and test data splitting\nWe'll use hashing on the unique key to consistently split rows to different groups.\nHashing is robust to underlying changes in the data, such as adding, deleting, or\nreordering rows. This deterministic process ensures that each data point is always\nassigned to the same split, thereby enhancing reproducibility.\n\n::: {#07531183 .cell execution_count=24}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to split data into train and test\"}\nimport random\n\n# this enables the analysis to be reproducible when random numbers are used\nrandom.seed(222)\nrandom_key = str(random.getrandbits(256))\n\n# put 3/4 of the data into the training set\ndf_train = df_train.mutate(\n train_flag=(df_train.case_id.cast(dt.str) + random_key).hash().abs() % 4 < 3\n)\n# split the dataset by train_flag\n# todo: use ml.train_test_split() after next release\ntrain_data = df_train[df_train.train_flag].drop(\"train_flag\")\ntest_data = df_train[~df_train.train_flag].drop(\"train_flag\")\n\nX_train = train_data.drop(\"target\")\ny_train = train_data.target.cast(dt.float32).name(\"target\")\n\nX_test = test_data.drop(\"target\")\ny_test = test_data.target.cast(dt.float32).name(\"target\")\n\ntrain_cnt = X_train.count().execute()\ntest_cnt = X_test.count().execute()\nprint(f\"train dataset size = {train_cnt} \\ntest data size = {test_cnt}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\ntrain dataset size = 1144339 \ntest data size = 382320\n```\n:::\n:::\n\n\n::: {.callout-warning}\nHashing provides a consistent but pseudo-random distribution of data, which\nmay not precisely align with the specified train/test ratio. While hash codes\nensure reproducibility, they don't guarantee an exact split. Due to statistical variance,\nyou might find a slight imbalance in the distribution, resulting in marginally more or\nfewer samples in either the training or test dataset than the target percentage. This\nminor deviation from the intended ratio is a normal consequence of hash-based\npartitioning.\n:::\n\n### XGBoost\nIn this section, we integrate XGBoost into a scikit-learn pipeline to create a\nstreamlined workflow for training and evaluating our model.\n\nWe'll set up a pipeline that includes two components:\n\n* **Preprocessing**: This step applies the `last_mile_preprocessing` for final data preprocessing.\n* **Modeling**: This step applies the `xgb.XGBClassifier()` to train the XGBoost model.\n\n::: {#7a5ff114 .cell execution_count=25}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to built and fit the pipeline\"}\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.metrics import roc_auc_score\nimport xgboost as xgb\n\nmodel = xgb.XGBClassifier(\n n_estimators=100,\n max_depth=5,\n learning_rate=0.05,\n subsample=0.8,\n colsample_bytree=0.8,\n random_state=42,\n)\n# create the pipeline with the last mile ML recipes and the model\npipe = Pipeline([(\"last_mile_recipes\", last_mile_preprocessing), (\"model\", model)])\n# fit the pipeline on the training data\npipe.fit(X_train, y_train)\n```\n\n::: {.cell-output .cell-output-display execution_count=56}\n```{=html}\n
Pipeline(steps=[('last_mile_recipes',\n                 Recipe(ExpandDate(date(), components=['week', 'day']),\n                        Drop(date()),\n                        OneHotEncode(cols(('maritalst_893M', 'requesttype_4525192L', 'max_profession_152M', 'max_gender_992L', 'max_empl_industry_691L', 'max_housingtype_772L', 'max_incometype_1044T', 'max_cancelreason_3545846M', 'max_rejectreason_755M', 'education_1103M', 'max_sta...\n                               feature_types=None, gamma=None, grow_policy=None,\n                               importance_type=None,\n                               interaction_constraints=None, learning_rate=0.05,\n                               max_bin=None, max_cat_threshold=None,\n                               max_cat_to_onehot=None, max_delta_step=None,\n                               max_depth=5, max_leaves=None,\n                               min_child_weight=None, missing=nan,\n                               monotone_constraints=None, multi_strategy=None,\n                               n_estimators=100, n_jobs=None,\n                               num_parallel_tree=None, random_state=42, ...))])
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
\n```\n:::\n:::\n\n\nLet's evaluate the model on the test data using Gini index:\n\n::: {#123379e8 .cell execution_count=26}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to calculate the Gini score for the test dataset\"}\ny_pred_proba = pipe.predict_proba(X_test)[:, 1]\n# calculate the AUC score\nauc = roc_auc_score(y_test, y_pred_proba)\n\n# calculate the Gini score\ngini_score = 2 * auc - 1\nprint(f\"gini_score for test dataset: {gini_score:,}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\ngini_score for test dataset: 0.07954028892065734\n```\n:::\n:::\n\n\n::: {.callout-note}\nThe competition is evaluated using a Gini stability metric. For more information, see the\n[evaluation guidelines](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/overview/evaluation)\n:::\n\n### Neural network classifier\nBuild a neural network classifier using PyTorch and PyTorch Lightning.\n\n::: {.callout-warning}\nIt is not recommended to build a neural network classifier for this competition, we are building\nit solely for demonstration purposes.\n:::\n\nWe'll demonstrate how to build a model by directly passing the data to it. IbisML recipes can output\ndata in various formats, making it compatible with different modeling frameworks.\nLet's first train the recipe:\n\n::: {#6e7a02ad .cell execution_count=27}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to train the IbisML recipe\"}\n# train preprocessing recipe using training dataset\nlast_mile_preprocessing.fit(X_train, y_train)\n```\n\n::: {.cell-output .cell-output-display execution_count=58}\n```\nRecipe(ExpandDate(date(), components=['week', 'day']),\n Drop(date()),\n OneHotEncode(cols(('maritalst_893M', 'requesttype_4525192L', 'max_profession_152M', 'max_gender_992L', 'max_empl_industry_691L', 'max_housingtype_772L', 'max_incometype_1044T', 'max_cancelreason_3545846M', 'max_rejectreason_755M', 'education_1103M', 'max_status_219L'))),\n Drop(string()),\n HandleUnivariateOutliers(cols(('max_amount_1115A', 'max_overdueamountmax_950A')),\n method='z-score',\n deviation_factor=3,\n treatment='capping'),\n ImputeMedian(numeric()),\n ScaleMinMax(numeric()),\n FillNA(numeric(), 0),\n Cast(numeric(), 'float32'))\n```\n:::\n:::\n\n\nIn the previous cell, we trained the recipe using the training dataset. Now, we will\ntransform both the train and test datasets using the same recipe. The default output format is a `NumPy array`\n\n::: {#ac5aaa87 .cell execution_count=28}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to transform the datasets using fitted recipe\"}\n# transform train and test dataset using IbisML recipe\nX_train_transformed = last_mile_preprocessing.transform(X_train)\nX_test_transformed = last_mile_preprocessing.transform(X_test)\nprint(f\"train data shape = {X_train_transformed.shape}\")\nprint(f\"test data shape = {X_test_transformed.shape}\")\n```\n\n::: {.cell-output .cell-output-stdout}\n```\ntrain data shape = (1144339, 977)\ntest data shape = (382320, 977)\n```\n:::\n:::\n\n\nLet's define a neural network classifier using PyTorch and PyTorch Lighting:\n\n::: {#92200558 .cell execution_count=29}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to define a torch classifier\"}\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import DataLoader, TensorDataset\nimport pytorch_lightning as pl\nfrom pytorch_lightning import Trainer\n\n\nclass NeuralNetClassifier(pl.LightningModule):\n def __init__(self, input_dim, hidden_dim=8, output_dim=1):\n super().__init__()\n self.model = nn.Sequential(\n nn.Linear(input_dim, hidden_dim),\n nn.ReLU(),\n nn.Linear(hidden_dim, output_dim),\n )\n self.loss = nn.BCEWithLogitsLoss()\n self.sigmoid = nn.Sigmoid()\n\n def forward(self, x):\n return self.model(x)\n\n def training_step(self, batch, batch_idx):\n x, y = batch\n y_hat = self(x)\n loss = self.loss(y_hat.view(-1), y)\n self.log(\"train_loss\", loss)\n return loss\n\n def validation_step(self, batch, batch_idx):\n x, y = batch\n y_hat = self(x)\n loss = self.loss(y_hat.view(-1), y)\n self.log(\"val_loss\", loss)\n return loss\n\n def configure_optimizers(self):\n return optim.Adam(self.parameters(), lr=0.001)\n\n def predict_proba(self, x):\n self.eval()\n with torch.no_grad():\n x = x.to(self.device)\n return self.sigmoid(self(x))\n\n# initialize your Lightning Module\nnn_classifier = NeuralNetClassifier(input_dim=X_train_transformed.shape[1])\n```\n:::\n\n\nNow, we'll create the PyTorch DataLoader using the output from IbisML:\n\n::: {#76fda8db .cell execution_count=30}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to convert IbisML output to tensor\"}\ny_train_array = y_train.to_pandas().to_numpy().astype(np.float32)\nx_train_tensor = torch.from_numpy(X_train_transformed)\ny_train_tensor = torch.from_numpy(y_train_array)\ntrain_dataset = TensorDataset(x_train_tensor, y_train_tensor)\n\ny_test_array = y_test.to_pandas().to_numpy().astype(np.float32)\nX_test_tensor = torch.from_numpy(X_test_transformed)\ny_test_tensor = torch.from_numpy(y_test_array)\nval_dataset = TensorDataset(X_test_tensor, y_test_tensor)\n\ntrain_loader = DataLoader(train_dataset, batch_size=32, shuffle=False)\nval_loader = DataLoader(val_dataset, batch_size=32, shuffle=False)\n```\n:::\n\n\nInitialize the PyTorch Lightning Trainer:\n\n::: {#4331ae3c .cell execution_count=31}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to construct PyTorch Lightning Trainer\"}\n# initialize a Trainer\ntrainer = Trainer(max_epochs=2)\nprint(nn_classifier)\n```\n\n::: {.cell-output .cell-output-stdout}\n```\nNeuralNetClassifier(\n (model): Sequential(\n (0): Linear(in_features=977, out_features=8, bias=True)\n (1): ReLU()\n (2): Linear(in_features=8, out_features=1, bias=True)\n )\n (loss): BCEWithLogitsLoss()\n (sigmoid): Sigmoid()\n)\n```\n:::\n:::\n\n\nLet's train the classifier:\n\n::: {#fbfc82b9 .cell execution_count=32}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to train the pytorch classifier\"}\n# train the model\ntrainer.fit(nn_classifier, train_loader, val_loader)\n```\n\n::: {.cell-output .cell-output-display}\n```{=html}\n\n```\n:::\n\n::: {.cell-output .cell-output-display}\n```{=html}\n\n```\n:::\n\n::: {.cell-output .cell-output-display}\n```{=html}\n\n```\n:::\n\n::: {.cell-output .cell-output-display}\n```{=html}\n\n```\n:::\n:::\n\n\nLet's use the trained model to make a prediction:\n\n::: {#004ce1cf .cell execution_count=33}\n``` {.python .cell-code code-fold=\"true\" code-summary=\"Show code to predict using the trained pytorch classifier\"}\ny_pred = nn_classifier.predict_proba(X_test_tensor[:10])\ny_pred\n```\n\n::: {.cell-output .cell-output-display execution_count=64}\n```\ntensor([[0.0238],\n [0.0253],\n [0.0210],\n [0.0254],\n [0.0236],\n [0.0239],\n [0.0242],\n [0.0240],\n [0.0169],\n [0.0236]])\n```\n:::\n:::\n\n\n## Takeaways\nIbisML provides a powerful suite of last-mile preprocessing transformations, including an advanced column selector\nthat streamlines the selection and transformation of specific columns in your dataset.\n\nIt integrates seamlessly with scikit-learn pipelines, allowing you to incorporate preprocessing recipes directly into\nyour workflow. Additionally, IbisML supports a variety of data output formats such as Dask, NumPy, and Arrow, ensuring\ncompatibility with different machine learning frameworks.\n\nAnother key advantage of IbisML is its flexibility in performing data preprocessing across multiple backends, including\nDuckDB, Polars, Spark, BigQuery, and other Ibis backends. This enables you to preprocess your training data\nusing the backend that best suits your needs, whether for large or small datasets, on local machines or compute backends,\nand in both development and production environments. Stay tuned for a future post where we will explore this capability in\nmore detail.\n\n## Reference\n* [1st Place Solution](https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-risk-lightgbm)\n* [home-credit-2024-starter-notebook](https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook)\n* [EDA and Submission](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/508337)\n* [Home Credit Baseline](https://www.kaggle.com/code/greysky/home-credit-baseline)\n\n", + "supporting": [ + "index_files" + ], + "filters": [], + "includes": { + "include-in-header": [ + "\n\n\n\n" + ], + "include-after-body": [ + "\n" + ] + } + } +} \ No newline at end of file diff --git a/docs/posts/ibisml/index.qmd b/docs/posts/ibisml/index.qmd new file mode 100644 index 000000000000..9a22f5551750 --- /dev/null +++ b/docs/posts/ibisml/index.qmd @@ -0,0 +1,928 @@ +--- +title: "Using IbisML and DuckDB for a Kaggle competition: credit risk model stability" +author: "Jiting Xu" +date: "2024-08-22" +categories: + - blog + - duckdb + - machine learning + - feature engineering +--- + +## Introduction +In this post, we'll demonstrate how to use Ibis and [IbisML](https://github.com/ibis-project/ibis-ml) +end-to-end for the +[credit risk model stability Kaggle competition](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability). + +1. Load data and perform feature engineering on DuckDB backend using IbisML +2. Perform last-mile ML data preprocessing on DuckDB backend using IbisML +3. Train two models using different frameworks: + * An XGBoost model within a scikit-learn pipeline. + * A neural network with PyTorch and PyTorch Lightning. + +The aim of this competition is to predict which clients are more likely to default on their +loans by using both internal and external data sources. + +To get started with Ibis and IbisML, please refer to the websites: + +* [Ibis](https://ibis-project.org/): An open-source dataframe library that works with any data system. +* [IbisML](https://ibis-project.github.io/ibis-ml/): A library for building scalable ML pipelines. + + +## Prerequisites +To run this example, you'll need to download the data from Kaggle website with a Kaggle user account and install Ibis, IbisML, and the necessary modeling library. + +### Download data +You need a Kaggle account to download the data. If you do not have one, +feel free to register one. + +1. Option 1: Manual download + * Log into your Kaggle account and download all data from this + [link](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data), + unzip the files, and save them to your local disk. +2. Option 2: Kaggle API + * Go to your `Kaggle Account Settings`. + * Under the `API` section, click on `Create New API Token`. This will download the `kaggle.json` + file to your computer. + * Place the `kaggle.json` file in the correct directory, normally it is under your home directory + `~/.kaggle`: + + ```bash + mkdir ~/.kaggle + mv ~/Downloads/kaggle.json ~/.kaggle + ``` + * Install Kaggle CLI and download the data: + + ```bash + pip install kaggle + kaggle competitions download -c home-credit-credit-risk-model-stability + unzip home-credit-credit-risk-model-stability.zip + ``` + +### Install libraries +To use Ibis and IbisML with the DuckDB backend for building models, you'll need to install the +necessary packages. Depending on your preferred machine learning framework, you can choose +one of the following installation commands: + +For PyTorch-based models: + +```{.bash} +pip install 'ibis-framework[duckdb]' ibis-ml torch pytorch-lightning +``` + +For XGBoost and scikit-learn-based models: + +```{.bash} +pip install 'ibis-framework[duckdb]' ibis-ml xgboost[scikit-learn] +``` + +Import libraries: +```{python} +import ibis +import ibis.expr.datatypes as dt +from ibis import _ +import ibis_ml as ml +from pathlib import Path +from glob import glob + +# enable interactive mode for ibis +ibis.options.interactive = True +``` + +Set the backend for computing: +```{python} +con = ibis.duckdb.connect() +# remove the black bars from duckdb's progress bar +con.raw_sql("set enable_progress_bar = false") +# DuckDB is the default backend for Ibis +ibis.set_backend(con) +``` + +Set data path: +```{python} +# change the root path to yours +ROOT = Path("/Users/claypot/Downloads/home-credit-credit-risk-model-stability") +TRAIN_DIR = ROOT / "parquet_files" / "train" +TEST_DIR = ROOT / "parquet_files" / "test" +``` + +## Data loading and processing +We'll use Ibis to read the Parquet files and perform the necessary processing for the next step. + +### Directory structure and tables +Since there are many data files, let's start by examining the directory structure and +tables within the train directory: + +```bash +# change this to your directory +tree -L 2 ~/Downloads/home-credit-credit-risk-model-stability/parquet_files/train +``` + +:::{.callout-note title="Click to show data files" collapse="true"} + +```bash +~/Downloads/home-credit-credit-risk-model-stability/parquet_files/train +├── train_applprev_1_0.parquet +├── train_applprev_1_1.parquet +├── train_applprev_2.parquet +├── train_base.parquet +├── train_credit_bureau_a_1_0.parquet +├── train_credit_bureau_a_1_1.parquet +├── train_credit_bureau_a_1_3.parquet +├── train_credit_bureau_a_2_0.parquet +├── train_credit_bureau_a_2_1.parquet +├── train_credit_bureau_a_2_10.parquet +├── train_credit_bureau_a_2_2.parquet +├── train_credit_bureau_a_2_3.parquet +├── train_credit_bureau_a_2_4.parquet +├── train_credit_bureau_a_2_5.parquet +├── train_credit_bureau_a_2_6.parquet +├── train_credit_bureau_a_2_7.parquet +├── train_credit_bureau_a_2_8.parquet +├── train_credit_bureau_a_2_9.parquet +├── train_credit_bureau_b_1.parquet +├── train_credit_bureau_b_2.parquet +├── train_debitcard_1.parquet +├── train_deposit_1.parquet +├── train_other_1.parquet +├── train_person_1.parquet +├── train_person_2.parquet +├── train_static_0_0.parquet +├── train_static_0_1.parquet +├── train_static_cb_0.parquet +├── train_tax_registry_a_1.parquet +├── train_tax_registry_b_1.parquet +└── train_tax_registry_c_1.parquet +``` + +::: + +The `train_base.parquet` file is the base table, while the others are feature tables. +Let's take a quick look at these tables. + +#### Base table +The base table (`train_base.parquet`) contains the unique ID, a binary target flag +and other information for the training samples. This unique ID will serve as the +linking key for joining with other feature tables. + +* `case_id` - This is the unique ID for each loan. You'll need this ID to + join feature tables to the base table. There are about 1.5m unique loans. +* `date_decision` - This refers to the date when a decision was made regarding the + approval of the loan. +* `WEEK_NUM` - This is the week number used for aggregation. In the test sample, + `WEEK_NUM` continues sequentially from the last training value of `WEEK_NUM`. +* `MONTH` - This column represents the month when the approval decision was made. +* `target` - This is the binary target flag, determined after a certain period based on + whether or not the client defaulted on the specific loan. + +Here is several examples from the base table: + +```{python} +#| code-fold: true +#| code-summary: "Show code to get the top 5 rows of base table" +ibis.read_parquet(TRAIN_DIR / "train_base.parquet").head(5) +``` + +#### Feature tables +The remaining files contain features, consisting of approximately 370 features from +previous loan applications and external data sources. Their definitions can be found in the feature +definition [file](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data) +from the competition website. + +There are several things we want to mention for the feature tables: + +* **Union datasets**: One dataset could be saved into multiple parquet files, such as +`train_applprev_1_0.parquet` and `train_applprev_1_1.parquet`, We need to union this data. +* **Dataset levels**: Datasets may have different levels, which we will explain as +follows: + * **Depth = 0**: Each row in the table is identified by a unique `case_id`. + In this case, you can directly join the features with the base table and use them as + features for further analysis or processing. + * **Depth > 0**: You will group the data based on the `case_id` and perform calculations + or aggregations within each group. + +Here are two examples of tables with different levels. + +Example of table with depth = 0, `case_id` is the row identifier, features can be directly joined + with the base table. +```{python} +#| code-fold: true +#| code-summary: "Show code to get the top 5 rows of user static data" +ibis.read_parquet(TRAIN_DIR / "train_static_cb_0.parquet").head(5) +``` + +Example of a table with depth = 1, we need to aggregate the features and collect statistics +based on `case_id` then join with the base table. +```{python} +#| code-fold: true +#| code-summary: "Show code to get the top 5 rows of credit bureau data" +ibis.read_parquet(TRAIN_DIR / "train_credit_bureau_b_1.parquet").relocate( + "num_group1" +).order_by(["case_id", "num_group1"]).head(5) +``` + +For more details on features and its exploratory data analysis (EDA), you can refer to +feature definition and these Kaggle notebooks: + +* [Feature + definition](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/data#:~:text=calendar_view_week-,feature_definitions,-.csv) +* [Home credit risk prediction + EDA](https://www.kaggle.com/code/loki97/home-credit-risk-prediction-eda) +* [Home credit CRMS 2024 + EDA](https://www.kaggle.com/code/sergiosaharovskiy/home-credit-crms-2024-eda-and-submission) + +### Data loading and processing +We will perform the following data processing steps using Ibis and IbisML: + +* **Convert data types**: Ensure consistency by converting data types, as the same column + in different sub-files may have different types. +* **Aggregate features**: For tables with depth greater than 0, aggregate features based + on `case_id`, including statistics calculation. You can collect statistics such as mean, + median, mode, minimum, standard deviation, and others. +* **Union and join datasets**: Combine multiple sub-files of the same dataset into one + table, as some datasets are split into multiple sub-files with a common prefix. Afterward, + join these tables with the base table. + +#### Convert data types +We'll use IbisML to create a chain of `Cast` steps, forming a recipe for data type +conversion across the dataset. This conversion is based on the provided information +extracted from column names. Columns that have similar transformations are indicated by a +capital letter at the end of their names: + +* P - Transform DPD (Days past due) +* M - Masking categories +* A - Transform amount +* D - Transform date +* T - Unspecified Transform +* L - Unspecified Transform + +For example, we'll define a IbisML transformation step to convert columns ends with `P` +to floating number: +```{python} +# convert columns ends with P to floating number +step_cast_P_to_float = ml.Cast(ml.endswith("P"), dt.float64) +``` + +Next, let's define additional type conversion transformations based on the postfix of column names: +```{python} +#| code-fold: true +#| code-summary: "Show code to define more steps" +# convert columns ends with A to floating number +step_cast_A_to_float = ml.Cast(ml.endswith("A"), dt.float64) +# convert columns ends with D to date +step_cast_D_to_date = ml.Cast(ml.endswith("D"), dt.date) +# convert columns ends with M to str +step_cast_M_to_str = ml.Cast(ml.endswith("M"), dt.str) +``` + +We'll construct the +[IbisML Recipe](https://ibis-project.github.io/ibis-ml/reference/core.html#ibis_ml.Recipe) +which chains together all the transformation steps. + +```{python} +#| code-fold: true +#| code-summary: "Show code to construct the recipe" +data_type_recipes = ml.Recipe( + step_cast_P_to_float, + step_cast_D_to_date, + step_cast_M_to_str, + step_cast_A_to_float, + # cast some special columns + ml.Cast(["date_decision"], "date"), + ml.Cast(["case_id", "WEEK_NUM", "num_group1", "num_group2"], dt.int64), + ml.Cast( + [ + "cardtype_51L", + "credacc_status_367L", + "requesttype_4525192L", + "riskassesment_302T", + "max_periodicityofpmts_997L", + ], + dt.str, + ), + ml.Cast( + [ + "isbidproductrequest_292L", + "isdebitcard_527L", + "equalityempfrom_62L", + ], + dt.int64, + ), +) +print(f"Data format conversion recipe:\n{data_type_recipes}") +``` + +::: {.callout-tip} +IbisML offers a powerful set of column selectors, allowing you to select columns based +on names, types, and patterns. For more information, you can refer to the IbisML column +selectors [documentation](https://ibis-project.github.io/ibis-ml/reference/selectors.html). +::: + +#### Aggregate features +For tables with a depth greater than 0 that can't be directly joined with the base table, +we need to aggregate the features by the `case_id`. You could compute the different statistics for numeric columns and +non-numeric columns. + +Here, we use the `maximum` as an example. + +```{python} +#| code-fold: true +#| code-summary: "Show code to aggregate features by case_id using max" +def agg_by_id(table): + return table.group_by("case_id").agg( + [ + table[col_name].max().name(f"max_{col_name}") + for col_name in table.columns + if col_name[-1] in ("T", "L", "P", "A", "D", "M") + ] + ) +``` +::: {.callout-tip} +For better predicting power, you need to collect different statistics based on the meaning of features. For simplicity, +we'll only collect the maximum value of the features here. +::: + +#### Put them together +We'll put them together in a function reads parquet files, optionally handles regex patterns for +multiple sub-files, applies data type transformations defined by `data_type_recipes`, and +performs aggregation based on `case_id` if specified by the depth parameter. + +```{python} +#| code-fold: true +#| code-summary: "Show code to read and process data files" +def read_and_process_files(file_path, depth=None, is_regex=False): + """ + Read and process Parquet files. + + Args: + file_path (str): Path to the file or regex pattern to match files. + depth (int, optional): Depth of processing. If 1 or 2, additional aggregation is performed. + is_regex (bool, optional): Whether the file_path is a regex pattern. + + Returns: + ibis.Table: The processed Ibis table. + """ + if is_regex: + # read and union multiple files + chunks = [] + for path in glob(str(file_path)): + chunk = ibis.read_parquet(path) + # transform table using IbisML Recipe + chunk = data_type_recipes.fit(chunk).to_ibis(chunk) + chunks.append(chunk) + table = ibis.union(*chunks) + else: + # read a single file + table = ibis.read_parquet(file_path) + # transform table using IbisML + table = data_type_recipes.fit(table).to_ibis(table) + + # perform aggregation if depth is 1 or 2 + if depth in [1, 2]: + table = agg_by_id(table) + + return table +``` + +Let's define two dictionaries, `train_data_store` and `test_data_store`, that organize and +store processed datasets for training and testing datasets. + +```{python} +#| code-fold: true +#| code-summary: "Show code to load all data into a dict" +train_data_store = { + "df_base": read_and_process_files(TRAIN_DIR / "train_base.parquet"), + "depth_0": [ + read_and_process_files(TRAIN_DIR / "train_static_cb_0.parquet"), + read_and_process_files(TRAIN_DIR / "train_static_0_*.parquet", is_regex=True), + ], + "depth_1": [ + read_and_process_files( + TRAIN_DIR / "train_applprev_1_*.parquet", 1, is_regex=True + ), + read_and_process_files(TRAIN_DIR / "train_tax_registry_a_1.parquet", 1), + read_and_process_files(TRAIN_DIR / "train_tax_registry_b_1.parquet", 1), + read_and_process_files(TRAIN_DIR / "train_tax_registry_c_1.parquet", 1), + read_and_process_files(TRAIN_DIR / "train_credit_bureau_b_1.parquet", 1), + read_and_process_files(TRAIN_DIR / "train_other_1.parquet", 1), + read_and_process_files(TRAIN_DIR / "train_person_1.parquet", 1), + read_and_process_files(TRAIN_DIR / "train_deposit_1.parquet", 1), + read_and_process_files(TRAIN_DIR / "train_debitcard_1.parquet", 1), + ], + "depth_2": [ + read_and_process_files(TRAIN_DIR / "train_credit_bureau_b_2.parquet", 2), + ], +} +# we won't be submitting the predictions, so let's comment out the test data. +# test_data_store = { +# "df_base": read_and_process_files(TEST_DIR / "test_base.parquet"), +# "depth_0": [ +# read_and_process_files(TEST_DIR / "test_static_cb_0.parquet"), +# read_and_process_files(TEST_DIR / "test_static_0_*.parquet", is_regex=True), +# ], +# "depth_1": [ +# read_and_process_files(TEST_DIR / "test_applprev_1_*.parquet", 1, is_regex=True), +# read_and_process_files(TEST_DIR / "test_tax_registry_a_1.parquet", 1), +# read_and_process_files(TEST_DIR / "test_tax_registry_b_1.parquet", 1), +# read_and_process_files(TEST_DIR / "test_tax_registry_c_1.parquet", 1), +# read_and_process_files(TEST_DIR / "test_credit_bureau_b_1.parquet", 1), +# read_and_process_files(TEST_DIR / "test_other_1.parquet", 1), +# read_and_process_files(TEST_DIR / "test_person_1.parquet", 1), +# read_and_process_files(TEST_DIR / "test_deposit_1.parquet", 1), +# read_and_process_files(TEST_DIR / "test_debitcard_1.parquet", 1), +# ], +# "depth_2": [ +# read_and_process_files(TEST_DIR / "test_credit_bureau_b_2.parquet", 2), +# ] +# } +``` + +Join all features data to base table: + +```{python} +#| code-fold: true +#| code-summary: "Define function to join feature tables to base table" +def join_data(df_base, depth_0, depth_1, depth_2): + for i, df in enumerate(depth_0 + depth_1 + depth_2): + df_base = df_base.join( + df, "case_id", how="left", rname="{name}_right" + f"_{i}" + ) + return df_base +``` + +Generate train and test datasets: +```{python} +#| code-fold: true +#| code-summary: "Show code to generate train and test datasets" +df_train = join_data(**train_data_store) +# df_test = join_data(**test_data_store) +total_rows = df_train.count().execute() +print(f"There is {total_rows} rows and {len(df_train.columns)} columns") +``` +### Select features +Given the large number of features (~370), we'll focus on selecting just a few of the most +informative ones by name for demonstration purposes in this post: +```{python} +#| code-fold: true +#| code-summary: "Show code to select important features for the train dataset" +df_train = df_train.select( + "case_id", + "date_decision", + "target", + # number of credit bureau queries for the last X days. + "days30_165L", + "days360_512L", + "days90_310L", + # number of tax deduction payments + "pmtscount_423L", + # sum of tax deductions for the client + "pmtssum_45A", + "dateofbirth_337D", + "education_1103M", + "firstquarter_103L", + "secondquarter_766L", + "thirdquarter_1082L", + "fourthquarter_440L", + "maritalst_893M", + "numberofqueries_373L", + "requesttype_4525192L", + "responsedate_4527233D", + "actualdpdtolerance_344P", + "amtinstpaidbefduel24m_4187115A", + "annuity_780A", + "annuitynextmonth_57A", + "applicationcnt_361L", + "applications30d_658L", + "applicationscnt_1086L", + # average days past or before due of payment during the last 24 months. + "avgdbddpdlast24m_3658932P", + # average days past or before due of payment during the last 3 months. + "avgdbddpdlast3m_4187120P", + # end date of active contract. + "max_contractmaturitydate_151D", + # credit limit of an active loan. + "max_credlmt_1052A", + # number of credits in credit bureau + "max_credquantity_1099L", + "max_dpdmaxdatemonth_804T", + "max_dpdmaxdateyear_742T", + "max_maxdebtpduevalodued_3940955A", + "max_overdueamountmax_950A", + "max_purposeofcred_722M", + "max_residualamount_3940956A", + "max_totalamount_503A", + "max_cancelreason_3545846M", + "max_childnum_21L", + "max_currdebt_94A", + "max_employedfrom_700D", + # client's main income amount in their previous application + "max_mainoccupationinc_437A", + "max_profession_152M", + "max_rejectreason_755M", + "max_status_219L", + # credit amount of the active contract provided by the credit bureau + "max_amount_1115A", + # amount of unpaid debt for existing contracts + "max_debtpastduevalue_732A", + "max_debtvalue_227A", + "max_installmentamount_833A", + "max_instlamount_892A", + "max_numberofinstls_810L", + "max_pmtnumpending_403L", + "max_last180dayaveragebalance_704A", + "max_last30dayturnover_651A", + "max_openingdate_857D", + "max_amount_416A", + "max_amtdebitincoming_4809443A", + "max_amtdebitoutgoing_4809440A", + "max_amtdepositbalance_4809441A", + "max_amtdepositincoming_4809444A", + "max_amtdepositoutgoing_4809442A", + "max_empl_industry_691L", + "max_gender_992L", + "max_housingtype_772L", + "max_mainoccupationinc_384A", + "max_incometype_1044T", +) + +df_train.head() +``` +Univariate analysis: +```{python} +#| code-fold: true +#| code-summary: "Show code to describe the train dataset" +# take the first 10 columns +df_train[df_train.columns[:10]].describe() +``` + +## Last-mile data preprocessing +We will perform the following transformation before feeding the data to models: + +* Missing value imputation +* Encoding categorical variables +* Handling date variables +* Handling outliers +* Scaling and normalization + +::: {.callout-note} +IbisML provides a set of transformations. You can find the +[roadmap](https://github.com/ibis-project/ibis-ml/issues/32). +The [IbisML website](https://ibis-project.github.io/ibis-ml/) also includes tutorials and API documentation. +::: + +### Impute features +Impute all numeric columns using the median. In real-life scenarios, it's important to +understand the meaning of each feature and apply the appropriate imputation method for +different features. For more imputations, please refer to this +[documentation](https://ibis-project.github.io/ibis-ml/reference/steps-imputation.html). +```{python} +#| code-fold: true +#| code-summary: "Show code to impute all numeric columns with median" +step_impute_median = ml.ImputeMedian(ml.numeric()) +``` + +### Encode categorical features +Encode all categorical features using one-hot-encode. For more encoding steps, +please refer to this +[doc](https://ibis-project.github.io/ibis-ml/reference/steps-encoding.html). + +```{python} +#| code-fold: true +#| code-summary: "Show code to one-hot encode selected columns" +ohe_step = ml.OneHotEncode( + [ + "maritalst_893M", + "requesttype_4525192L", + "max_profession_152M", + "max_gender_992L", + "max_empl_industry_691L", + "max_housingtype_772L", + "max_incometype_1044T", + "max_cancelreason_3545846M", + "max_rejectreason_755M", + "education_1103M", + "max_status_219L", + ] +) +``` + +### Handle date variables +Calculate all the days difference between any date columns and the column `date_decision`: +```{python} +#| code-fold: true +#| code-summary: "Show code to calculate days difference between date columns and date_decision" +date_cols = [col_name for col_name in df_train.columns if col_name[-1] == "D"] +days_to_decision_expr = { + # difference in days + f"{col}_date_decision_diff": ( + _.date_decision.epoch_seconds() - getattr(_, col).epoch_seconds() + ) + / (60 * 60 * 24) + for col in date_cols +} +days_to_decision_step = ml.Mutate(days_to_decision_expr) +``` +Extract information from the date columns: +```{python} +#| code-fold: true +#| code-summary: "Show code to extract day and week info from date columns" +# dow and month is set to catagoery +expand_date_step = ml.ExpandDate(ml.date(), ["week", "day"]) +``` + +### Handle outliers +Capping outliers using `z-score` method: +```{python} +#| code-fold: true +#| code-summary: "Show code to cap outliers for selected columns" +step_handle_outliers = ml.HandleUnivariateOutliers( + ["max_amount_1115A", "max_overdueamountmax_950A"], + method="z-score", + treatment="capping", + deviation_factor=3, +) +``` + +### Construct recipe +We'll construct the last mile preprocessing [recipe](https://ibis-project.github.io/ibis-ml/reference/core.html#ibis_ml.Recipe) +by chaining all transformation steps, which will be fitted to the training dataset and later applied test datasets. + +```{python} +#| code-fold: true +#| code-summary: "Show code to construct the recipe" +last_mile_preprocessing = ml.Recipe( + expand_date_step, + ml.Drop(ml.date()), + # handle string columns + ohe_step, + ml.Drop(ml.string()), + # handle numeric cols + # capping outliers + step_handle_outliers, + step_impute_median, + ml.ScaleMinMax(ml.numeric()), + # fill missing value + ml.FillNA(ml.numeric(), 0), + ml.Cast(ml.numeric(), "float32"), +) +print(f"Last-mile preprocessing recipe: \n{last_mile_preprocessing}") +``` + +## Modeling +After completing data preprocessing with Ibis and IbisML, we proceed to the modeling +phase. Here are two approaches: + +* Use IbisML as a independent data preprocessing component and hand off the data to downstream modeling +frameworks with various output formats: + - pandas Dataframe + - NumPy Array + - Polars Dataframe + - Dask Dataframe + - xgboost.DMatrix + - Pyarrow Table +* Use IbisML recipes as components within an sklearn Pipeline and +train models similarly to how you would do with sklearn pipeline. + +We will build an XGBoost model within a scikit-learn pipeline, and a neural network classifier using the +output transformed by IbisML recipes. + +### Train and test data splitting +We'll use hashing on the unique key to consistently split rows to different groups. +Hashing is robust to underlying changes in the data, such as adding, deleting, or +reordering rows. This deterministic process ensures that each data point is always +assigned to the same split, thereby enhancing reproducibility. + +```{python} +#| code-fold: true +#| code-summary: "Show code to split data into train and test" +import random + +# this enables the analysis to be reproducible when random numbers are used +random.seed(222) +random_key = str(random.getrandbits(256)) + +# put 3/4 of the data into the training set +df_train = df_train.mutate( + train_flag=(df_train.case_id.cast(dt.str) + random_key).hash().abs() % 4 < 3 +) +# split the dataset by train_flag +# todo: use ml.train_test_split() after next release +train_data = df_train[df_train.train_flag].drop("train_flag") +test_data = df_train[~df_train.train_flag].drop("train_flag") + +X_train = train_data.drop("target") +y_train = train_data.target.cast(dt.float32).name("target") + +X_test = test_data.drop("target") +y_test = test_data.target.cast(dt.float32).name("target") + +train_cnt = X_train.count().execute() +test_cnt = X_test.count().execute() +print(f"train dataset size = {train_cnt} \ntest data size = {test_cnt}") +``` + +::: {.callout-warning} +Hashing provides a consistent but pseudo-random distribution of data, which +may not precisely align with the specified train/test ratio. While hash codes +ensure reproducibility, they don't guarantee an exact split. Due to statistical variance, +you might find a slight imbalance in the distribution, resulting in marginally more or +fewer samples in either the training or test dataset than the target percentage. This +minor deviation from the intended ratio is a normal consequence of hash-based +partitioning. +::: + +### XGBoost +In this section, we integrate XGBoost into a scikit-learn pipeline to create a +streamlined workflow for training and evaluating our model. + +We'll set up a pipeline that includes two components: + +* **Preprocessing**: This step applies the `last_mile_preprocessing` for final data preprocessing. +* **Modeling**: This step applies the `xgb.XGBClassifier()` to train the XGBoost model. + +```{python} +#| code-fold: true +#| code-summary: "Show code to built and fit the pipeline" +from sklearn.pipeline import Pipeline +from sklearn.metrics import roc_auc_score +import xgboost as xgb + +model = xgb.XGBClassifier( + n_estimators=100, + max_depth=5, + learning_rate=0.05, + subsample=0.8, + colsample_bytree=0.8, + random_state=42, +) +# create the pipeline with the last mile ML recipes and the model +pipe = Pipeline([("last_mile_recipes", last_mile_preprocessing), ("model", model)]) +# fit the pipeline on the training data +pipe.fit(X_train, y_train) +``` + +Let's evaluate the model on the test data using Gini index: +```{python} +#| code-fold: true +#| code-summary: "Show code to calculate the Gini score for the test dataset" +y_pred_proba = pipe.predict_proba(X_test)[:, 1] +# calculate the AUC score +auc = roc_auc_score(y_test, y_pred_proba) + +# calculate the Gini score +gini_score = 2 * auc - 1 +print(f"gini_score for test dataset: {gini_score:,}") +``` + +::: {.callout-note} +The competition is evaluated using a Gini stability metric. For more information, see the +[evaluation guidelines](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/overview/evaluation) +::: + +### Neural network classifier +Build a neural network classifier using PyTorch and PyTorch Lightning. + +::: {.callout-warning} +It is not recommended to build a neural network classifier for this competition, we are building +it solely for demonstration purposes. +::: + +We'll demonstrate how to build a model by directly passing the data to it. IbisML recipes can output +data in various formats, making it compatible with different modeling frameworks. +Let's first train the recipe: +```{python} +#| code-fold: true +#| code-summary: "Show code to train the IbisML recipe" +# train preprocessing recipe using training dataset +last_mile_preprocessing.fit(X_train, y_train) +``` + +In the previous cell, we trained the recipe using the training dataset. Now, we will +transform both the train and test datasets using the same recipe. The default output format is a `NumPy array` + +```{python} +#| code-fold: true +#| code-summary: "Show code to transform the datasets using fitted recipe" +# transform train and test dataset using IbisML recipe +X_train_transformed = last_mile_preprocessing.transform(X_train) +X_test_transformed = last_mile_preprocessing.transform(X_test) +print(f"train data shape = {X_train_transformed.shape}") +print(f"test data shape = {X_test_transformed.shape}") +``` + +Let's define a neural network classifier using PyTorch and PyTorch Lighting: +```{python} +#| code-fold: true +#| code-summary: "Show code to define a torch classifier" +import numpy as np +import torch +import torch.nn as nn +import torch.optim as optim +from torch.utils.data import DataLoader, TensorDataset +import pytorch_lightning as pl +from pytorch_lightning import Trainer + + +class NeuralNetClassifier(pl.LightningModule): + def __init__(self, input_dim, hidden_dim=8, output_dim=1): + super().__init__() + self.model = nn.Sequential( + nn.Linear(input_dim, hidden_dim), + nn.ReLU(), + nn.Linear(hidden_dim, output_dim), + ) + self.loss = nn.BCEWithLogitsLoss() + self.sigmoid = nn.Sigmoid() + + def forward(self, x): + return self.model(x) + + def training_step(self, batch, batch_idx): + x, y = batch + y_hat = self(x) + loss = self.loss(y_hat.view(-1), y) + self.log("train_loss", loss) + return loss + + def validation_step(self, batch, batch_idx): + x, y = batch + y_hat = self(x) + loss = self.loss(y_hat.view(-1), y) + self.log("val_loss", loss) + return loss + + def configure_optimizers(self): + return optim.Adam(self.parameters(), lr=0.001) + + def predict_proba(self, x): + self.eval() + with torch.no_grad(): + x = x.to(self.device) + return self.sigmoid(self(x)) + +# initialize your Lightning Module +nn_classifier = NeuralNetClassifier(input_dim=X_train_transformed.shape[1]) +``` + +Now, we'll create the PyTorch DataLoader using the output from IbisML: +```{python} +#| code-fold: true +#| code-summary: "Show code to convert IbisML output to tensor" +y_train_array = y_train.to_pandas().to_numpy().astype(np.float32) +x_train_tensor = torch.from_numpy(X_train_transformed) +y_train_tensor = torch.from_numpy(y_train_array) +train_dataset = TensorDataset(x_train_tensor, y_train_tensor) + +y_test_array = y_test.to_pandas().to_numpy().astype(np.float32) +X_test_tensor = torch.from_numpy(X_test_transformed) +y_test_tensor = torch.from_numpy(y_test_array) +val_dataset = TensorDataset(X_test_tensor, y_test_tensor) + +train_loader = DataLoader(train_dataset, batch_size=32, shuffle=False) +val_loader = DataLoader(val_dataset, batch_size=32, shuffle=False) +``` + +Initialize the PyTorch Lightning Trainer: +```{python} +#| code-fold: true +#| code-summary: "Show code to construct PyTorch Lightning Trainer" +# initialize a Trainer +trainer = Trainer(max_epochs=2) +print(nn_classifier) +``` + +Let's train the classifier: +```{python} +#| code-fold: true +#| code-summary: "Show code to train the pytorch classifier" +# train the model +trainer.fit(nn_classifier, train_loader, val_loader) +``` + +Let's use the trained model to make a prediction: +```{python} +#| code-fold: true +#| code-summary: "Show code to predict using the trained pytorch classifier" +y_pred = nn_classifier.predict_proba(X_test_tensor[:10]) +y_pred +``` + +## Takeaways +IbisML provides a powerful suite of last-mile preprocessing transformations, including an advanced column selector +that streamlines the selection and transformation of specific columns in your dataset. + +It integrates seamlessly with scikit-learn pipelines, allowing you to incorporate preprocessing recipes directly into +your workflow. Additionally, IbisML supports a variety of data output formats such as Dask, NumPy, and Arrow, ensuring +compatibility with different machine learning frameworks. + +Another key advantage of IbisML is its flexibility in performing data preprocessing across multiple backends, including +DuckDB, Polars, Spark, BigQuery, and other Ibis backends. This enables you to preprocess your training data +using the backend that best suits your needs, whether for large or small datasets, on local machines or compute backends, +and in both development and production environments. Stay tuned for a future post where we will explore this capability in +more detail. + +## Reference +* [1st Place Solution](https://www.kaggle.com/code/yuuniekiri/fork-of-home-credit-risk-lightgbm) +* [home-credit-2024-starter-notebook](https://www.kaggle.com/code/jetakow/home-credit-2024-starter-notebook) +* [EDA and Submission](https://www.kaggle.com/competitions/home-credit-credit-risk-model-stability/discussion/508337) +* [Home Credit Baseline](https://www.kaggle.com/code/greysky/home-credit-baseline)