This repository contains all of the necesary components for creating new models which are compatible with the VIEWS pipeline. The views-models repository also contains all of the already implemented VIEWS models (with the exception of HydraNet), at both PRIO-GRID-month and country-month levels of analysis, along with information about prediction targets, input data and model algorithms.
- Key Terms and Definitions
- Time Partitioning
- Model Naming Conventions
- Creating New Models
- Model scripts
- Model filesystems
- Running a single model
- Ensembles
- Creating a new ensemble
- Ensemble scripts
- Ensemble filesystem
- Running an ensemble
- Implemented Models
- Model Catalogs
In VIEWS terminology a model is defined as:
- A specific instantiation of a machine learning algorithm,
- Trained using a predetermined and unique set of hyperparameters,
- On a well-defined set of input features,
- The specific input features for every model are referred to as querysets.
- A model predicts specific outcome target or targets.
- In the case of stepshift models, a model is understood as all code and all artifacts necessary to generate a comprehensive 36 month forecast for the specified target.
- Note that, two models, identical in all other aspects, will be deemed distinct if varying post-processing techniques are applied to their generated predictions. For instance, if one model's predictions undergo calibration or normalization while the other's do not. Similarly, two models identical in all aspects are considered distinct if they utilize different input features (querysets).
VIEWS models all currently use the same time partitioning model to divide the time-axis of the input dataset up into three segments. The boundaries of these partitions are currently fixed in the views-pipeline-core package, but they will be made user-configurable in the future. Partitions are labelled by their VIEWS month_id, where month 001 is January 1980, month 121 is January 1990, and so on. The partitions are as follows:
- calibration: training interval: 121 - 396, test interval: 397 - 444
- validation: training interval: 121 - 444, test interval: 445 - 456
- forecasting: training interval: 121 - (current VIEWS month -2)
The models belonging to the VIEWS pipeline follow a predetermined naming standard. Models no longer carry descriptive titles (e.g., transform_log_clf_name_LGBMClassifier_reg_name_LGBMRegressor). Although such titles provided some information about the models, as models are developed over time, this type of naming could cause confusion and ultimately small differences could not be communicated properly through the model titles. Instead, we rely on the metadata of the model for model specifications and being able to substantively differentiate them between each other.
Additionally, the new naming convention for models in the pipeline takes the form of adjective_noun, adding more models alphabetically. For example, the first model to be added can be named amazing_apple, the second model bad_bunny, etc. This is a popular practice, and Weights & Biases implements this naming convention automatically.
The views-models repository contains the tools for creating new models, as well as creating new model ensembles. All of the necessary components are found in the build_model_scaffold.py and build_ensemble_scaffold.py files. The goal of this part of the VIEWS pipeline is the ability to simply create models which have the right structure and fit into the VIEWS directory structure. This makes the models uniform, consistent, and allows for easier replicability.
As with other parts of the VIEWS pipeline, we aim to make interactions with our pipeline as simple and straightforward as possible. In the context of the views-models, when creating a new model or ensemble, the user is closely guided through the steps which are needed, in an intuitive manner. This allows for the model creation processes to be consistent no matter how experienced the creator is. After providing a name for the model or ensemble, guided to be in the form adjective_noun, the user can specify the desired model algorithm and the model architecture package. Currently, only stepshift models are supported, however, we work on expanding the list of supported algorithms and model architectures. Then, the scaffold builders create all of the model files and model directories, uniformly structured. This instantly removes possibilities of error, increases efficiency and effectiveness as it decreases manual inputs of code. Finally, this allows all of our users, no matter their level of proficiency, to seamlessly interact with out pipeline in no time.
To run the model scaffold builder, execute
python build_model_scaffold.py
You will be asked to enter a name for your model in lowercase adjective_noun form. If the scaffolder is happy with your proposed model name, it will create a new directory with your chosen name. This directory in turn contains the scripts and folders needed to run your model and store intermediate data belonging to it. It is the responsibility of the model creator to make changes to the newly created scripts where appropriate - see below for further information on which scripts need to be updated. The scripts created are as follows (see further down for a description of the filesystem):
It is the responsibility of the model creator to write a README file for their model. This should give a concise, human-readable description of the model:
- what it forecasts
- what algorithm(s) it uses
- what hyperparameters it relies on and whether these have been or can be optimised (e.g. in a sweep)
- a brief description of what input data it requires
- how it is or should be evaluated
- (preferably) some notes on performance.
This shell script is the principal means by which a model should be run (e.g. by executing source run.sh arg1 arg2... at a terminal prompt - see 'Running a single model' below). You probably will not need to modify it, but it is important to understand what it is for.
The VIEWS platform is designed to support models of arbitrary form. A model may need to import many external libraries or modules and the set of modules required by one model are quite likely to be incompatible with those of another (a 'dependency conflict').
The VIEWS platform solves this problem by building a custom Python environment for every model. A Python environment is an isolated sandbox into which a particular set of modules can be installed, and it does not matter if the modules installed on one environment are incompatible with those installed in another. Code execution can be quickly switched between environments, so that models with dependency conflicts can be easily executed in series.
The run.sh script first builds the environment required to run a model, specified in the requirements.txt file - see below), and then executes the model inside that environment by passing its main.py file (see below) to the Python interpreter.
The purpose of this file is to specify which modules (probably including their versions or an acceptable range thereof) need to be installed in the model-specific environment built by run.sh.
It is the model creator's responsibility to ensure that this file is correctly populated. Only modules named in this file (and their dependencies) will be installed in the model env. If your model needs numpy and it is not installed by any other dependencies, it needs to be specified here.
It is strongly advised to specify a range of acceptable versions for each installed module using the standard notation, e.g. views-stepshifter>=1.0.0,<2.0.0.
Once the run.sh script has created the model's environment, it activates the environment and executes the main.py file inside it. The main.py has several tasks:
- it uses the
ModelPathManagerfromviews-pipeline-coreto establish where it is on the host machine's filesystem so that other scripts and modules can be found by the Python interpreter at runtime - it logs into
weights-and-biases- all runs executed in the VIEWS platform are automatically externally logged to the weights-and-biases web platform - URLs are printed to the terminal during model/ensemble execution, which will take users to webpages showing live logging and analytics - it parses command line arguments (forwarded by
run.sh) which specify whether the model is to be trained, whether a sweep over hyperparameters should be performed, etc. - it then calls the relevant
Managerfromviews-pipeline-corewhich superintends the execution of the model. Every class of models has its own custom manager (e.g.StepShifterManagerlooks after stepshifted regression models). If you are introducing a new class of model to VIEWS, you will need to create a new Manager class for it.
Make sure to import your model manager class and include it in the appropriate sections in the main.py script!
As well as understanding the function of the model scripts, users and developers need to have a grasp of the structure of the model filesystem. A description of each of the directories follows below:
An artifact is the result of training a model on a particular set of input data. For example, if a regression model is trained on a particular input, the set of regression coefficients calculated by the model constitute an artifact. The artifact can be stored and later used to make predictions from new data without needing to train the model again.
The VIEWS platform allows users to store model-specific artifacts locally. If you have never trained a particular model, this directory will be empty.
This directory contains Python scripts used to control model configuration. Model creators need to ensure that all settings needed to configure a model or a model sweep are contained in these scripts and correctly defined.
-
config_deployment.py: The VIEWS platform is designed to permit new models to be tested and developed in parallel with established (i.e. 'production') models which are used to generate our publicly-disseminated forecasts. A model'sdeployment_statusmust be specified in this script and must be one ofshadow,deployed,baseline, ordeprecatedto indicate its stage of development. An under-development model which should not be used in production should have statusshadow. Fully developed production models have statusdeployed. Simple models used as references or yardsticks arebaseline. If a production model is superseded, it can be retired from the production system by setting its status todeprecated. A model MUST NOT be givendeployedstatus without discussion with the modelling team. -
config_hyperparameters.py: Most models will rely on algorithms for which hyperparameters need to be specified (even if invisibly by default). This script contains dictionary specifying any required model-specific hyperparameters to be read at runtime. -
config_meta.py: This script specifies the most basic model parameters, e.g. the model's name, the name of its forecasting algorithm, the dependent variable it forecasts, the name of its input data queryset (see below), its creator. This dictionary must be populated correctly, since it controls important aspects of model execution further down the pipeline. -
config_queryset.py: Most VIEWS models are anticipated to need to fetch data from the central VIEWS database via theviewserclient. This is done by specifying aqueryset. A queryset is a representation of a data table. It consists of a name, a target level-of-analysis (into which all data is automatically transformed) and one or more Columns. A Column, in turn, has a name, a source level-of-analysis, the name of a raw feature from the VIEWS database and zero or more transforms from theviews-transformation-library. The queryset is passed via the viewser client to a server which executes the required database fetches and transformations and returns the dataset as a single dataframe (or, in the future, a tensor). Theconfig_queryset.pyspecifies the queryset, and it is the model creator's responsibility to ensure that the specification is correct. -
config_sweep.py: During model development, developers will often wish to perform sweeps over ranges of model hyperparameters for optimisation purposes (hyperparameter tuning). This script allows such sweeps to be configured, specifying which parameters ranges are to explored and what is to be optimised.
The VIEWS platform allows local storage of data for convenience, both raw data (i.e. input data from a queryset fetch) and generated data (e.g. forecasts), all of which is stored in this directory.
The platform produces detailed logs during execution which are printed to the terminal, exported to weights-and-biases and also saved locally in this directory.
While the use of Jupyter notebooks is generally discouraged on the grounds of stability and git interoperability, this directory is provided for those who wish to use them during model development.
Users should note, however, that Jupyter notebooks MUST NOT be used to run production models.
Convenience directory where figures, papers or slides relating to particular models can be stored.
Logs shipped to weights-and-biases are also stored locally here for convenience
A model is run by executing the run.sh script in its root directory, which checks to see if an appropriate environment for the model exists, creates one if not, activates the environment, and executes the model's main.py inside it. The model can be run by executing the main.py directly, but it is then up to the user to ensure that the model's environment is correctly built and activated. However, if the environment is setup once e.g. by executing the run.sh script, it can be activated at a later point in time and the model can be run by by executing the main.py directly.
The run.sh and main.py both require command line arguments to control their behaviour (command line arguments submitted to run.sh are simply passed on to main.py). A description of these arguments follows:
-
-ror--run_typefollowed by one of [calibration,validation,forecasting]: choose the run type -
-sor--sweep: perform a sweep run (run type must becalibration) -
-tor--train: flag indicating whether a new model artifact should be trained -
-eor--evaluate: flag indicating if model should be evaluated -
-for--forecast: flag indicating if forecasts are to be generated -
-aor--artifact_name: flag allowing the name of the artifact to be evaluated to be supplied -
-saor--saved: flag to indicate that saved data/artifacts should be used -
-oor--override_month: flag allowing one to specify a month other than the most recent month with data from which to forecast -
-ddor--drift_self_test: flag enabling drift-detection self_test at data-fetch -
-etor--eval_type: flag allowing type of evaluation to be performed to be specified
Consequently, in order to train and evaluate a model it is either possible to execute python main.py -run_type calibration -t -e or run.sh -run_type calibration -t -e. The first command runs the script directly, while the second one also handles environment setup before execution. Of course, these commands can be used to run already existing models (see the Catalogs for a list of already existing models). Consult the Glossary and the Model Documentation Series to learn more about different run types.
An ensemble is a combination of models which has greater predictive power than any of the models does singly. Ensemble forecasts can be simple averages over the forecasts of their constituent models, or a more sophisticated weighted average, where the weights are computed by optimising the ensemble's predictive performance over a specially reserved data partition, using, for example, an evolutionary algorithm. (The latter is not yet implemented in the VIEWS pipeline).
It is also possible to reconcile one ensemble with another (usually at a different spatial resolution) to, for example, force the forecasts to agree over well-defined spatial areas such as countries. The VIEWS pipeline allows point priogrid-level forecasts to be reconciled with country-level forecasts on a month-by-month basis (accounting for the fact that countries change size or appear/disappear altogether).
The procedure for creating a new ensemble is much the same as that for creating a new model. The build_ensemble_scaffold.py script is run and, once it is supplied with a legal lower case adjective_noun ensemble name, a filesystem very similar to that created for a new model is built. As in the case of creating new models, make sure to update the appropriate model scripts (indicated below).
It is the responsibility of the ensemble creator to write a README file for their ensemble. This should give a concise, human-readable description of the ensemble:
- what it forecasts
- which constituent models it ensembles over
- how the ensembling is done
- how it is or should be evaluated
- (preferably) some notes on performance.
This shell script is the principal means by which an ensemble should be run (e.g. by executing source run.sh arg1 arg2... at a terminal prompt - see 'Running an ensemble' below). You probably will not need to modify it, but it is important to understand what it is for.
The run.sh script first builds the environment required to run the ensemble, specified in the requirements.txt file - see below), and then executes the ensemble inside that environment by passing its main.py file (see below) to the Python interpreter.
The purpose of this file is to specify which modules (probably including their versions or an acceptable range thereof) need to be installed in the ensemble-specific environment built by run.sh.
It is the ensemble creator's responsibility to ensure that this file is correctly populated. Only modules named in this file (and their dependencies) will be installed in the ensemble env. If your ensemble needs numpy and it is not installed by any other dependencies, it needs to be specified here.
It is strongly advised to specify a range of acceptable versions for each installed module using the standard notation, e.g. views-stepshifter>=1.0.0,<2.0.0.
Once the run.sh script has created the ensemble's environment, it activates the environment and executes the main.py file inside it. The main.py has several tasks:
- it uses the
EnsemblePathManagerfromviews-pipeline-coreto establish where it is on the host machine's filesystem so that other scripts and modules can be found by the Python interpreter at runtime - it logs into
weights-and-biases- all runs executed in the VIEWS platform are automatically externally logged to the weights-and-biases web platform - URLs are printed to the terminal during model/ensemble execution, which will take users to webpages showing live logging and analytics - it parses command line arguments (forwarded by
run.sh) which specify whether the ensemble is to be trained, whether forecasts are to be generated, etc. - it then calls the
EnsembleManagerfromviews-pipeline-corewhich superintends the execution of the ensemble.
As well as understanding the function of the ensemble scripts, users and developers need to have a grasp of the structure of the ensemble filesystem. A description of each of the directories follows below:
Currently not used by ensembles.
This directory contains Python scripts used to control model configuration. Model creators need to ensure that all settings needed to configure a model or a model sweep are contained in these scripts and correctly defined.
-
config_deployment.py: An ensemble'sdeployment_statusmust be specified in this script and must be one ofshadow,deployed,baseline, ordeprecatedto indicate its stage of development. An under-development ensemble which should not be used in production should have statusshadow. Fully developed production ensembles have statusdeployed. Ensembles used as references or yardsticks arebaseline. If a production ensemble is superseded, it can be retired from the production system by setting its status todeprecated. An ensemble MUST NOT be givendeployedstatus without discussion with the modelling team. -
config_hyperparameters.py: This is currently only used to configure the number of timesteps forward the ensemble forecasts -
config_meta.py: This script specifies the most basic ensemble parameters, e.g. the ensemble's name, the models it ensembles over, the dependent variable it forecasts, the aggregation scheme used to perform the ensembling, which reconciliation algorithm it to be applied, which other ensemble it should be reconciled with, and its creator. This dictionary must be populated correctly, since it controls important aspects of ensemble execution further down the pipeline.
The VIEWS platform allows local storage of data for convenience, in this directory.
The platform produces detailed logs during execution which are printed to the terminal, exported to weights-and-biases and also saved locally in this directory.
Convenience directory where figures, papers or slides relating to particular models can be stored.
Logs shipped to weights-and-biases are also stored locally here for convenience
An ensemble is run by executing the run.sh script in its root directory, which checks to see if an appropriate environment for the ensemble exists, creates one if not, activates the environment, and executes the ensemble's main.py inside it. The ensemble can be run by executing the main.py directly, but it is then up to the user to ensure that the model's environment is correctly built and activated.
The run.sh and main.py both require command line arguments to control their behaviour (command line arguments submitted to run.sh are simply passed on to main.py). A description of these arguments follows:
-
-ror--run_typefollowed by one of [calibration,validation,forecasting]: choose the run type -
-tor--train: flag indicating whether new model artifacts should be trained -
-eor--evaluate: flag indicating if the ensemble should be evaluated -
-for--forecast: flag indicating if forecasts are to be generated -
-saor--saved: flag to indicate that saved data/artifacts should be used -
-oor--override_month: flag allowing one to specify a month other than the most recent month with data from which to forecast -
-ddor--drift_self_test: flag enabling drift-detection self_test at data-fetch -
-etor--eval_type: flag allowing type of evaluation to be performed to be specified
Consequently, in order to train a model and generate predictions, execute either python main.py -t --run_type forecasting -f or run.sh -t --run_type forecasting -f. Of course, these commands can be used to run already existing ensembles (see the Catalogs for a list of already existing ensembles). Consult the Glossary and the Model Documentation Series to learn more about different run types.
As of now, the only implemented model architecture is the stepshifter model. Experienced users have the possibility to develop their own model architecture including their own model class manager. Head over to views-pipeline-core for further information on the model class manager and on how to develop new model architectures.
In addition to the possibility of easily creating new models and ensembles, in order to maintain an organized and structured overview over all of the implemented models, the views-models repository also contains model catalogs containing all of the information about individual models. This information is collected from the metadata of each model and entails:
- Model name
- Model algorithm
- The prediction target
- Input features/Queryset
- The non-default hyperparameters
- Forecasting type
- Implementation status
- Implementation date
- The creator of the model
The catalogs are automatically filled out and updated, through a GitHub action, with every new model or ensemble which is created.
The catalogs for all of the existing VIEWS models can be found below. The models catalogs are separated based on the models' level of analysis - country-month models and PRIO-GRID-month models, with the ensamble catalog all the way at the bottom. All of the information about algorithms, input features, hyperparameters and other model specification are included.
| Model Name | Algorithm | Targets | Input Features | Non-default Hyperparameters | Forecasting Type | Implementation Status | Implementation Date | Author |
|---|---|---|---|---|---|---|---|---|
| bittersweet_symphony | XGBRegressor | ln_ged_sb_dep | - fatalities003_all_features | - hyperparameters bittersweet_symphony | None | shadow | NA | Marina |
| brown_cheese | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_baseline | - hyperparameters brown_cheese | None | shadow | NA | Borbála |
| car_radio | XGBRegressor | ln_ged_sb_dep | - fatalities003_topics | - hyperparameters car_radio | None | shadow | NA | Borbála |
| counting_stars | XGBRegressor | ln_ged_sb_dep | - fatalities003_conflict_history_long | - hyperparameters counting_stars | None | shadow | NA | Borbála |
| demon_days | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_faostat | - hyperparameters demon_days | None | shadow | NA | Marina |
| electric_relaxation | RandomForestRegressor | ged_sb_dep | - escwa001_cflong | - hyperparameters electric_relaxation | None | deprecated | NA | Sara |
| fast_car | HurdleModel | ln_ged_sb_dep | - fatalities003_vdem_short | - hyperparameters fast_car | None | shadow | NA | Borbála |
| fluorescent_adolescent | HurdleModel | ln_ged_sb_dep | - fatalities003_joint_narrow | - hyperparameters fluorescent_adolescent | None | shadow | NA | Marina |
| good_riddance | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_joint_narrow | - hyperparameters good_riddance | None | shadow | NA | Marina |
| green_squirrel | HurdleModel | ln_ged_sb_dep | - fatalities003_joint_broad | - hyperparameters green_squirrel | None | shadow | NA | Borbála |
| heavy_rotation | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_joint_broad | - hyperparameters heavy_rotation | None | shadow | NA | Borbála |
| high_hopes | HurdleModel | ln_ged_sb_dep | - fatalities003_conflict_history | - hyperparameters high_hopes | None | shadow | NA | Borbála |
| little_lies | HurdleModel | ln_ged_sb_dep | - fatalities003_joint_narrow | - hyperparameters little_lies | None | shadow | NA | Marina |
| national_anthem | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_wdi_short | - hyperparameters national_anthem | None | shadow | NA | Borbála |
| ominous_ox | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_conflict_history | - hyperparameters ominous_ox | None | shadow | NA | Borbála |
| plastic_beach | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_aquastat | - hyperparameters plastic_beach | None | shadow | NA | Marina |
| popular_monster | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_topics | - hyperparameters popular_monster | None | shadow | NA | Borbála |
| teen_spirit | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_faoprices | - hyperparameters teen_spirit | None | shadow | NA | Marina |
| twin_flame | HurdleModel | ln_ged_sb_dep | - fatalities003_topics | - hyperparameters twin_flame | None | shadow | NA | Borbála |
| yellow_submarine | XGBRFRegressor | ln_ged_sb_dep | - fatalities003_imfweo | - hyperparameters yellow_submarine | None | shadow | NA | Marina |
| Model Name | Algorithm | Targets | Input Features | Non-default Hyperparameters | Forecasting Type | Implementation Status | Implementation Date | Author |
|---|---|---|---|---|---|---|---|---|
| bad_blood | LGBMRegressor | ln_ged_sb_dep | - fatalities003_pgm_natsoc | - hyperparameters bad_blood | None | shadow | NA | Xiaolong |
| blank_space | HurdleModel | ln_ged_sb_dep | - fatalities003_pgm_natsoc | - hyperparameters blank_space | None | shadow | NA | Xiaolong |
| caring_fish | XGBRegressor | ln_ged_sb_dep | - fatalities003_pgm_conflict_history | - hyperparameters caring_fish | None | shadow | NA | Xiaolong |
| chunky_cat | LGBMRegressor | ln_ged_sb_dep | - fatalities003_pgm_conflictlong | - hyperparameters chunky_cat | None | shadow | NA | Xiaolong |
| dark_paradise | HurdleModel | ln_ged_sb_dep | - fatalities003_pgm_conflictlong | - hyperparameters dark_paradise | None | shadow | NA | Xiaolong |
| invisible_string | LGBMRegressor | ln_ged_sb_dep | - fatalities003_pgm_broad | - hyperparameters invisible_string | None | shadow | NA | Xiaolong |
| lavender_haze | HurdleModel | ln_ged_sb_dep | - fatalities003_pgm_broad | - hyperparameters lavender_haze | None | shadow | NA | Xiaolong |
| midnight_rain | LGBMRegressor | ln_ged_sb_dep | - fatalities003_pgm_escwa_drought | - hyperparameters midnight_rain | None | shadow | NA | Xiaolong |
| old_money | HurdleModel | ln_ged_sb_dep | - fatalities003_pgm_escwa_drought | - hyperparameters old_money | None | shadow | NA | Xiaolong |
| orange_pasta | LGBMRegressor | ln_ged_sb_dep | - fatalities003_pgm_baseline | - hyperparameters orange_pasta | None | shadow | NA | Xiaolong |
| purple_alien | HydraNet | ln_sb_best, ln_ns_best, ln_os_best, ln_sb_best_binarized, ln_ns_best_binarized, ln_os_best_binarized | - escwa001_cflong | - hyperparameters purple_alien | None | shadow | NA | Simon |
| wildest_dream | HurdleModel | ln_ged_sb_dep | - fatalities003_pgm_conflict_sptime_dist | - hyperparameters wildest_dream | None | shadow | NA | Xiaolong |
| yellow_pikachu | HurdleModel | ln_ged_sb_dep | - fatalities003_pgm_conflict_treelag | - hyperparameters yellow_pikachu | None | shadow | NA | Xiaolong |
| Model Name | Algorithm | Targets | Input Features | Non-default Hyperparameters | Forecasting Type | Implementation Status | Implementation Date | Author |
|---|---|---|---|---|---|---|---|---|
| cruel_summer | ln_ged_sb_dep | None | - hyperparameters cruel_summer | None | shadow | NA | Xiaolong | |
| pink_ponyclub | ln_ged_sb_dep | None | - hyperparameters pink_ponyclub | None | shadow | NA | Xiaolong | |
| skinny_love | ln_ged_sb_dep | None | - hyperparameters skinny_love | None | shadow | NA | Xiaolong | |
| white_mustang | ln_ged_sb_dep | None | - hyperparameters white_mustang | None | deployed | NA | Xiaolong |
Special thanks to the VIEWS MD&D Team for their collaboration and support.