Validation framework for model

[sgreenbury]

Aim: define metrics to be used at different parts of the modelling to validate model against data. E.g. flows from QUANT model. See section in wiki.

[sgreenbury]

Some measures to consider:

• Modelled travel times compared to reported travel times in NTS
• Modelled flows between MSOAs compared to observed flows used in QUANT
• See here for commuting flows census data. We can check the SPC people to matched workplace flows is consistent with this data.

[Hussein-Mahfouz]

Guidelines from dft on activity and agent based models: TAG unit M5-4

[Hussein-Mahfouz]

Putting this here for now but could be a separate issue on calibration later:

"We need data sets to be recognized as components on the same level as models. Such data components can then enter the integrated frameworks at various places, not only at the top, as input to drive the whole integrated model, and at the bottom, to compare with the output and to calibrate the model. Data components can be also used between components to test, adjust, and correct the data flows inside the integrated model. This can substantially increase the efficiency and accuracy of the integration process, and reduce the overall complexity of the calibration task for the whole integrated model." - ‘Integronsters’, integral and integrated modeling

A useful exercise could be to identify the datasets that could be used to calibrate at intermediate points of the pipeline

[sgreenbury]

Notes on tasks:

• Retrieve data on MSOA to MSOA flows by mode
• Function to measure difference in NTS (ground truth) and estimated travel time (from travel time matrices)
• Calculate the aggregate MSOA to MSOA flows for a SPC matched population

[sgreenbury]

We also need to determine a set of metrics for measuring quality of matching between the two datasets as part of task1.

[BZ-BowenZhang]

We also need to determine a set of metrics for measuring quality of matching between the two datasets as part of task1.

I completely agree with @sgreenbury, and I think calculating the metrics for comparison is easy. But after generating metrics, I have a question: How do we judge the 'goodness'? In other words, how do we set a threshold value as the acceptable standard? We do not have another candidate dataset to compare, but maybe we can compare it with other synthetic populations published in previous papers.

[sgreenbury]

Adding reference with validation methods from @stuartlynn

[BZ-BowenZhang]

• Add more markdown descriptions for the notebook Validation_SPC_with Cencus

[BZ-BowenZhang]

As discussed on 20th Sep:

• The Census (commuting OD) dataset will be used as the benchmark dataset to validate the performance of different synthetic datasets (SPC and AcBM). The Census and SPC validation is completed and ready to merge with the main branch PR 17 validation framework for model #50
• For the next step, more validation work will be done after the whole Leeds population with activity chain is available, still comparing with the census data and comparing with the SPC.
• In the further London case, we may have more option datasets to compare/validate such as mobile phone data from Tao's Group or travel flow from TfL etc.