Performance improvements

[slinlee]

This is the main github issue tracking performance #2848

The loading speed when toggling between experiments is ok. But in a model with +30 experiments when clicking on "+ New Experiment" or on "Train Policy" or toggling between new experiments it takes about 10 seconds to load. It increasingly takes longer as I start more experiments.

Sources of slow performance

1. Rendering the reward function syntax highlighting
2. Calculating the simulation metrics for each experiment
3. Passing all the data related to each experiment

Tasks to improve performance

• Lazy load project/model view experiment grid Project/Model view experiment grid lazy load data #2903 This should help with 1 and 3. @fionnachan
• Add simulation metrics to database - need cache or database for simulation metrics #2312 This will help with 2 and 3 (since there will be less data) @alexamakarov
• Use shallow copies of experiments when loading for experiment grid Use shallow copies of experiments when loading experiment grid #3048 This helps with 3. Potentially @alexamakarov . With the above two changes we may get enough performance improvements already.

cc @kepricon @SaharEs

[SaharEs]

The loading speed when toggling between experiments is ok. But in a model with +30 experiments when clicking on "+ New Experiment" or on "Train Policy" or toggling between new experiments it takes about 10 seconds to load. It increasingly takes longer as I start more experiments.

@slinlee This is not an accurate description of the situation anymore. In models with many reward variables, Toggling between experiments and new experiments works fine at start but as training progresses and values get populated, ALL navigation becomes slow. This includes: loading model dashboard, switching between models, loading experiments, and switching between experiments.