deskit is a flexible, lightweight, and easy-to-use ensembling library that implements Dynamic Ensemble Selection (DES) algorithms for ensembling multiple ML models on a given dataset.
The library works entirely with data, taking as input a validation dataset along with precomputed predictions and outputting a dictionary of weights per model. This means that it can be used with any library or model without requiring any wrappers, including custom models, popular ML libraries, and APIs.
deskit includes several DES algorithms, and it works with both classification and regression.
See the full documentation here.
Ensemble learning in machine learning refers to when multiple models trained on a single dataset combine their predictions to create a single, more accurate prediction, usually through weighted voting or picking the best model.
DES refers to techniques where the models or their voting weights are selected dynamically for every test case. This selection bases on the idea of competence regions, which is the concept that there are regions of feature space where certain models perform particularly well, so every base model can be an expert in a different region. Only the most competent, or an ensemble of the most competent models is selected for the prediction.
Through empirical studies, DES has been shown to perform best on small-sized, imbalanced, or heterogeneous datasets, as well as non-stationary data (concept drift), models that haven't perfected a dataset, and when used on an ensemble of models with differing architectures and perspectives.
However, DES is not an automatic improvement. It tends to perform worse when datasets are homogeneous or have low diversity, when the validation set isn't a good representation of the test set, when using very high dimensional data or few training samples, or when a single model dominates a dataset.
pip install deskit
# The library runs with Nearest Neighbors from sklearn for exact KNN
pip install scikit-learn
# Alternatively, ANN can be used for faster runtimes at the cost of
# slightly lower accuracy. The following three are supported;
# Install the one you want to use.
pip install faiss-cpu # FAISS (good default for most datasets)
pip install annoy # Annoy (memory-efficient, simple)
pip install hnswlib # HNSW (best for high-dimensional data)Python (>= 3.9)
NumPy (>= 1.21)
For a more detailed understanding of how to use the library, consult the documentation.
from deskit.des.knorau import KNORAU
# 1. Train your models
models = {"rf": rf, "xgb": xgb, "mlp": mlp}
# 2. Get predictions on a held-out validation set
# Regression: scalar arrays
# Classification: probability arrays OR hard predictions
val_preds = {name: m.predict_proba(X_val) for name, m in models.items()}
# 3. Fit the router
router = KNORAU(task="classification", metric="accuracy", mode="max", k=20)
router.fit(X_val, y_val, val_preds)
# 4. Route test samples
test_preds = {name: m.predict_proba(X_test) for name, m in models.items()}
for i, x in enumerate(X_test):
weights = router.predict(x, temperature=0.1)
# weights example: {"rf": 0.7, "xgb": 0.2, "mlp": 0.1}
prediction = sum(weights[n] * test_preds[n][i] for n in weights)For classification with probability arrays, blend the output the same way to get a final probability distribution, then take the argmax.
Most DES libraries are tied to scikit-learn. deskit only ever sees a numpy feature matrix and a dict of prediction arrays, so the models themselves are never touched after training. This allows for more flexibility and a lighter library.
Furthermore, deskit works with both classification and regression, while the majority of DES libraries and literature is focused only on classification tasks.
# PyTorch example
with torch.no_grad():
val_preds = {name: m(X_val_t).cpu().numpy() for name, m in models.items()}
test_preds = {name: m(X_test_t).cpu().numpy() for name, m in models.items()}
router = KNORAU(task="classification", metric="accuracy", mode="max", k=20)
router.fit(X_val, y_val, val_preds)
weights = router.predict(X_test[i])Full explanation of the algorithms, syntax, and parameters is available in the documentation. If you're struggling to decide on which algorithm to use, see the algorithm selection guide.
| Method | Best for | Notes |
|---|---|---|
DEWS-U |
Regression | Softmax over neighborhood-averaged scores. Temperature controls sharpness. |
DEWS-I |
Regression | Like DEWS-U but scores are inverse-distance weighted. |
DEWS-T |
Both | Like DEWS-I but fits a weighted trend line over neighbor scores. |
DEWS-V |
Regression | Like DEWS-U but scores are variance-penalized. |
DEWS-IV |
Regression | Like DEWS-V but scores are also inverse-distance weighted. |
LWSE-U |
Both | Per-sample NNLS weight estimation over the local neighbourhood. |
LWSE-I |
Both | Like LWSE-U but rows are inverse-distance weighted. |
KNORA-U |
Classification | Each model earns one vote per neighbor it correctly classifies. |
KNORA-E |
Classification | Only models correct on all neighbors survive; falls back to smaller neighborhoods. |
KNORA-IU |
Classification | Like KNORA-U but votes are inverse-distance weighted. |
OLA |
Both | Hard selection: only the single best model in the neighborhood contributes. |
deskit supports three Approximate Nearest Neighbour backends plus exact search:
| Preset | Backend | Install | Notes |
|---|---|---|---|
exact |
sklearn KNN | scikit-learn |
Exact, no extra deps |
balanced |
FAISS IVF | faiss-cpu |
~98% recall, good default |
fast |
FAISS IVF | faiss-cpu |
~95% recall, faster queries |
turbo |
FAISS flat | faiss-cpu |
Exact via FAISS, GPU-friendly |
high_dim_balanced |
HNSW | hnswlib |
Best for >100 features, balanced |
high_dim_fast |
HNSW | hnswlib |
Best for >100 features, faster |
Annoy is also available as a custom backend — memory-efficient and simple, good for datasets that need to be persisted to disk.
# Exact search (no extra deps)
router = KNORAU(..., preset="exact")
# High-dimensional data
router = KNORAU(..., preset="high_dim_balanced")
# Custom FAISS config
router = KNORAU(..., preset="custom", finder="faiss",
index_type="ivf", n_probes=50)
# Annoy
router = KNORAU(..., preset="custom", finder="annoy",
n_trees=100, search_k=-1)Any callable (y_true, y_pred) -> float works:
def pinball(y_true, y_pred, alpha=0.9):
e = y_true - y_pred
return alpha * e if e >= 0 else (alpha - 1) * e
router = DEWSU(task="regression", metric=pinball, mode="min", k=20)Built-in metric strings: accuracy, mae, mse, rmse, log_loss, prob_correct.
deskit can be used with non-tabular data types like images, time series, and more. However, when used, the passed features either need to be run through a feature extractor beforehand, such as a CNN backbone for images.
20-seed benchmark (seeds 0–19) on standard sklearn and OpenML datasets. "Best Single" is the best individual model selected on the validation set. "Simple Average" is uniform equal-weight blending, included as a baseline.
It is important to consider that these experiments were run with the default hyperparameters, meaning that
they could vary greatly with different values, and results could improve with tuning.
For a more detailed benchmark breakdown, see the benchmark in the documentation.
To see the full results, see results.txt in the tests folder.
This pool was selected for having variability in architectures while avoiding a single dominant model.
deskit algorithms tested: OLA, DEWS-U, DEWS-I, DEWS-T, DEWS-V, DEWS-IV, LWSE-U, LWSE-I, KNORA-U, KNORA-E, KNORA-IU.
Pool: KNN, Decision Tree, SVR, Ridge, Bayesian Ridge.
% shown as delta vs Best Single. 20-seed mean.
| Dataset | Best Single | Simple Avg | deskit best |
|---|---|---|---|
| California Housing (sklearn) | 0.3956 | +7.99% | −2.54% (DEWS-I) |
| Bike Sharing (OpenML) | 51.678 | +47.77% | −6.86% (DEWS-I) |
| Abalone (OpenML) | 1.4981 | +1.14% | +1.47% (KNORA-U/KNORA-IU) |
| Diabetes (sklearn) | 44.504 | +3.18% | +0.86% (DEWS-IV) |
| Concrete Strength (OpenML) | 5.2686 | +23.66% | −5.41% (LWSE-I) |
deskit beats best single and simple averaging on 3/5 regression datasets. This shows how DES can provide a strong boost if used on the right dataset, but it might be counterproductive if used blindly.
KNORA variants are designed for classification, which explains the poor performance on regression datasets; However, some exceptions can occur in certain datasets when the target is discrete and classification-like (like in Abalone).
DEWS-I and LWSE-I show the largest improvements on their respective datasets.
Pool: KNN, Decision Tree, Gaussian NB, SVM-RBF, Logistic Regression.
% shown as delta vs Best Single. 20-seed mean.
| Dataset | Best Single | Simple Avg | deskit best |
|---|---|---|---|
| HAR (OpenML) | 98.24% | −0.33% | +0.16% (DEWS-T) |
| Yeast (OpenML) | 58.87% | +0.77% | +1.66% (KNORA-IU) |
| Image Segment (OpenML) | 93.70% | +1.40% | +2.25% (DEWS-T / DEWS-IV) |
| Waveform (OpenML) | 85.91% | −0.98% | −0.39% (DEWS-T) |
| Vowel (OpenML) | 89.95% | −2.05% | +2.95% (LWSE-I) |
deskit beats or matches best single and simple averaging on 4/5 classification datasets.
Consider that usually it is recommended to only use one algorithm at a time, this benchmark ran eleven of them at the
same time, so with a single one runtime is expected to be about 11x faster. For this benchmark, preset='balanced' was used,
so the backend was an ANN algorithm with FAISS IVF.
| Dataset | deskit (11 algorithms) |
|---|---|
| California Housing | 351.0 ms |
| Bike Sharing | 283.5 ms |
| Abalone | 72.9 ms |
| Diabetes | 14.0 ms |
| Concrete Strength | 22.5 ms |
| HAR | 693.1 ms |
| Yeast | 44.7 ms |
| Image Segment | 69.9 ms |
| Waveform | 124.5 ms |
| Vowel | 39.0 ms |
deskit caches all model predictions on the validation set at fit time and reads from that matrix at inference.
Issues and PRs welcome.
