Automate gamma values3

[Femkemilene]

The gamma value automation script, which uses regulated simulated annealing to calculate gamma values. It runs all the selected models 5 years past their end year, and finds gamma values that minimise the difference in the 'rate of change' before and after the start of the simulation.

To avoid finding local minima, we run the code 3 times. It's relatively slow, so consider setting no_it to 4 when you run the code.

Is the code clear? I realise it's not the easiest algorithm, so hope my commenting is sufficient.

Testing:
a. Do the gamma values make sense? Check major technologies and major countries for all
b. Are the gamma values comparable to manually determined gamma values?
c. Does the code work with different selections of modules? Other settings?

I'm building from the power sector update (which should be merged first), so only check the changes to automate_gamma_values

[Copilot]

Pull Request Overview

This PR implements an automated gamma value calculation script for the FTT model using regulated simulated annealing. The script optimizes gamma values by running models 5 years past their end year and minimizing differences in rate of change before and after simulation start.

• Removes multiple empty MTFT (power sector technology mix) CSV files for various countries
• Updates the MJET.csv file to restructure power sector technology classifications
• Implements automated gamma value optimization to avoid local minima through multiple iterations

Reviewed Changes

Copilot reviewed 94 out of 161 changed files in this pull request and generated 1 comment.

File	Description
MTFT_*.csv (multiple countries)	Removes empty power sector technology matrix files
MJET.csv	Updates power technology classifications with new numbering scheme

Comments suppressed due to low confidence (1)

Inputs/S0/FTT-P/MJET.csv:1

• [nitpick] The change from 'BIGCC' (Biomass Integrated Gasification Combined Cycle) to generic 'Biomass' loses important technical specificity about the technology type. This could make it harder to distinguish between different biomass power generation methods.

,1 Hard coal,2 Other coal etc,3 Crude oil etc,4 Heavy fuel oil,5 Middle distillates,6 Other gas,7 Natural gas,8 Electricity,9 Heat,10 Combustible waste,11 Biofuels,12 Hydrogen

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[Copilot]

The numbering scheme shows inconsistency where row 10 contains '9 OCGT' and row 11 contains '10 OCGT + CCS'. This creates confusion between line numbers and technology IDs. Consider aligning the technology numbering with the row structure for better maintainability.

[ab-uoe]

Since MGAM has been removed in this pull request all previous regional MGAM CSVs should also be deleted since it causes problems with the automation script.

But doing away with a separate MGAM sheet means that convert_masterfile_to_csv script would also not generate and update these regional MGAM csvs which renders the front end manual gamma sliders unusable.

as of now the mechanism to make tweaks in to the gamma values that were generated with the automation script is to directly edit them in the masterfile and rerun the model, which is less convenient than the sliders.

Minor: classification_titles.xlsx should also be updated to the new 22 technologies or it causes problems in the front end when selecting techs.

The algorithm is working well, ran 10 rounds (for FTT:P), with step size at .2 * .99 (150 iterations). Churned out gamma with a min and max value of -1.01 and 1.03. Results were good.

[Femkemilene]

MGAMs are not stored on git, but on your local PC. To avoid deleting files you might want to keep, github retains these when switching branch. I'm not quite sure how to fix this: our solution to do away with the Excel files, move to big csv files (rather than chopped up ones) should lessen how annoying this is, but not fix it either. The solution here is likely to add a cleaner script, which is a separate issue, and should likely happen after we've converted the data inputs into big csv + parquet.

Do we want to preserve the gamma sliders? I would like to remove them altogether, as the algorithm should be good enough, and manual tweaking is not very scientific. I can try to remove that tab from the frontend; not sure if that's python or vue code.

Would you have time to synch this branch with main? Needs to happen before a merge anyway.

[ab-uoe]

Hi Femke, discussed w/ Ian some of his concerns with the automation algo.

Points from him below:

• The code runs successfully but the algorithm doesn’t work with many strong kinks in the shares projections; See Spain, Greece etc.
• Works more effectively on countries with few technologies
• Still requires manual adjustment, but rate of change ROC, much less sensitive to changes
• Longer run, more iterations and smaller adjustment
• Normally 2 main techs need changing, gas/solar/on-shore

[Femkemilene]

@ldxib2:
I'm working on a final weighting where bigger technologies are prioritised in the final decision which run is best for each region.

I don't fully understand your comments, but wanted to double check if you tested this with MWKA turned off? The algorithm does not deal well with MWKA values.

To clarify:
-> You want the algorithm to change to have longer runs, more iterations and smaller adjustments? My experimenting seems to indicate that we can do with fewer iterations instead and more runs instead.

[Femkemilene]

I've rewritten the code to be:

1. A bit easier to understand
2. To use an algorithm closer to classical simulated annealing: the acceptance of an individual gamma value depends on whether it improved its own technology (50%), but also whether the collection of gamma values improve the overall country score (50%), weighted by the sqrt of shares. That is: the algorithm prioritises larger technologies a bit.
3. I've changed the unit to be in log share space, as that is closer to the logic of FTT. It might be improved further if we do a different transformation (S/1-S), as that follows the logistic growth logic. Not that FTT is fully logistic of course.
4. Played with the parameters, so that the final values are usually very similar. In some countries, there's clearly multiple minima and the final values differ between the two runs.

I'm happy with the current settings. Adrian, are you?

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 11 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.