M3SA

Radu-Nicolae · 2024-09-10T11:55:09Z

Summary

M3SA - Multi-Meta-Model Simulation Analysis. Full functionality implemented and integrated into OpenDC.

Multi-Model

We design a novel simulation tool, able to leverage simulation data of multiple models into the same tool.
The Multi-Model has extensive customizability for computation and plotting.

Meta-Model

We create the Meta-Model, which embeds functionality from the Multi-Model.
The Meta-Model aggregates other models simulations, and is capable of performing meta-predictions.
In plain English, the Meta-Model predicts using other models' predictions, merged using various functions (i.e., meta-functions), such as mean or median.

We propose M3SA, a tool for ICT simulation analysis, using multi-model techniques. M3SA provides solutions to improve the performance of existing simulations of the Information and Communication Technology infrastructure.
We release M3SA as free and open-source software for the community to use. M3SA follows modern, state-of-the-art engineering processes in the industry and produces reproducible results.
M3SA integrated both the Multi-Model and the Meta-Model.
Input: JSON interface
Fully documented code
Fully documented input and JSON schema validator
M3SA can be integrated with any simulator. We provide an integration tutorial.
We integrate M3SA with OpenDC, and can be enabled with flag
-a
when running OpenDC, via
ScenarioCli.kt
.

Paper

M3SA is part of the research efforts of AtLarge Research Team, towards massivizing computer systems. We conduct scientific research and follow the AtLarge Design Vision. We design, analyze, and engineer the M3SA system, towards performance, accuracy, universality, and usability.

M3SA: Exploring the Performance and Climate Impact of Datacenters by Multi-Model Simulation and Analysis
Authors: Radu Nicolae (@Radu-Nicolae), Alexandru Iosup (@AlexandruIosup), Dante Niewenhuis (@DanteNiewenhuis)

External Dependencies 🍀

N/A

Breaking API Changes ⚠️

N/A

Simply specify none (N/A) if not applicable.

…flow

DanteNiewenhuis

There are several things that need to be updated for this PR to be accepted. I have divided this review into two parts. First, I will discuss things that need to be updated for me to accept this PR. Second, I will discuss some things I strongly suggest improving, but will not stop the PR from being accepted. I do however expect you to take a look at them, especially the easy to fix comments. I have also left many comments in the files.

Mandatory

Rename opendc-m3sa to opendc-experiment-m3sa so it adheres to the overall naming scheme of OpenDC.

[RN] Done

There are multiple files that are changed in this PR that should not have been touched at all. For instance, all the package.json and package-lock.json files should not be changed. This is also why the docker is not building at the moment.

[RN] Done

ExperimentCli, and Scenarios should not be changed for the functionalities m3sa currently brings. m3sa should have its own Cli that can be executed separately. m3sa should create Scenarios to run, and execute them using ScenarioRunner.

[RN] Done - not changed anymore. M3SACli and ExperimentCli are now, perhaps temporary, two separate entities.

It should be much clearer how m3sa currently runs. I have been going through the Python code, and it is not clear to me what the order of execution is, where the actual simulation is being executed etc.

[RN] Not clear to me what is the mandatory part regarding (4). I'll assume it's documentation, which is (already) done -- input guideline, readme of m3sa, and integration tutorial.

Add a requirements.txt

[RN] Done, you can find it in the root folder of the python code

Add a README to the m3sa experiment in which you detail how to run m3sa

[RN] (Already) Done

Suggestions

I will leave the Python files to you. However, I do have some suggestions:

You are executing a lot of functions in the init of your classes. For instance, MetaModel computes everything and creates all outputs. This is not common in Python, and makes it difficult to understand what is happening.
Not everything has to happen inside a class. This is not Java, you can make separate functions.
Please take a look at Pandas and NumPy functions. There are multiple functions you have written yourself that are already implemented by either Pandas or NumPy. These will always be faster than whatever we can write in Python.
Adding Typing could greatly improve your Python code. Currently, users either need a good understanding of the code to understand what is expected.

DanteNiewenhuis · 2024-09-11T12:48:31Z

opendc-experiments/opendc-experiments-base/build.gradle.kts

Why does experiment-base implement m3sa?

As far as I understand, m3sa uses base, not the other way around.

both m3sa and experiment-base use, in one way or another, each other.

In which way does experiment-base use m3sa?
Also this would be circular importing, which is not possible.

DanteNiewenhuis · 2024-09-11T12:49:41Z

.../opendc-experiments-base/src/main/kotlin/org/opendc/experiments/base/runner/ExperimentCli.kt

Again, Experiment.kt should not be used by m3sa, and thus not changed

let's see if we have a better alternative -- how would you suggest to make this change? what code to add in which file?

...experiments-base/src/main/kotlin/org/opendc/experiments/base/scenario/ExperimentFactories.kt

DanteNiewenhuis · 2024-09-11T12:54:50Z

settings.gradle.kts

Please rename the module to opendc-experiment-m3sa, like the other moduiles

opendc-experiment-m3sa is inaccurate and redundant. similarly to opendc-experiment-base, opendc-experiment-faas, opendc-experiment-tf20.

these should all be changed and remove experiment.
=> opendc-tf20, opendc-base, opendc-faas, opendc-m3sa.

For the sake of consistency, I will rename to opendc-experiments-m3sa. Consider that redundant and inaccurate namings diminish the codebase readability and overall quality.

I know that this might not be a somewhat older style of naming things (and thus redundant).
However, that is something to solve in the future (if we want).

let's do that in the next PR. I'll handle.

DanteNiewenhuis · 2024-09-11T12:56:03Z

...experiments-base/src/main/kotlin/org/opendc/experiments/base/scenario/ExperimentFactories.kt

@@ -79,6 +79,7 @@ public fun getExperiment(experimentSpec: ExperimentSpec): List<Scenario> {
                carbonTracePath = scenarioSpec.carbonTracePath,
                exportModelSpec = scenarioSpec.exportModel,
                outputFolder = outputFolder,
+                m3saSetup = experimentSpec.m3saSetup,


Is m3saSetup ever used when running a scenario?

yes, m3saSetup is used in scenario-post run. triggers and links the scenario with m3sa

DanteNiewenhuis · 2024-09-13T07:00:09Z

opendc-experiments/opendc-m3sa/src/main/python/models/MultiModel.py

+
+        self.max_model_len = min([len(model.raw_sim_data) for model in self.models])
+
+    def compute_windowed_aggregation(self):


Pandas or NumPy should have functions that do this for you.
There will always be more performant than what you can write in Python

oh, I see. How to adapt this?

DanteNiewenhuis · 2024-09-13T07:01:36Z

opendc-experiments/opendc-m3sa/src/main/python/models/MultiModel.py

+                numeric_values = model.raw_sim_data
+                model.processed_sim_data = self.mean_of_chunks(numeric_values, self.window_size)
+
+    def generate_plot(self):


Why would this function not be called with the plot type?
It is not clear to me where this.plot_type is set.

Also, I thought the idea was to automatically create multiple plots?

well, not really. this depends on the user -- the plotting process takes time as well, right? Why wasting time on plotting if the user may want to only get the data of the metamodel?

see more in the paper §4, the Meta-Model.

the plot_type is set by the user -- either time_series, cumulative, cumulative_time_series.

For more details, have a look at NFRs for §3 and §4 of the paper for better clarification. alternatively, have a look at the documentation.

DanteNiewenhuis · 2024-09-13T07:09:26Z

opendc-experiments/opendc-m3sa/src/main/python/models/Model.py

+        the 'parse_trackr' method can be called to load additional experimental details from a corresponding JSON file.
+    """
+
+    def __init__(self, raw_sim_data, id, path):


You can just turn this into a DataClass

Given that this class seems to be purely holding data

I see. How would we benefit from this, @DanteNiewenhuis?

[TODO#1] read about DataClasses

DanteNiewenhuis · 2024-09-13T07:12:05Z

opendc-experiments/opendc-m3sa/src/main/python/input_parser.py

+    return input_json
+
+
+def find_root_dir():


This is fairly dangerous code.
The chance of this failing is very high.
For instance, what if I have multiple README.md files in my project?
What if I run m3sa from another folder?

if you have multiple README.md files, in the same folder (i.e., the root folder), you break the law of operating systems :))

I agree this can be improved -- what better ways to ensure we are in the root folder do we have?

DanteNiewenhuis · 2024-09-13T07:13:15Z

opendc-experiments/opendc-m3sa/src/main/python/simulator_specifics.py

+✅ Good: "host", "simulation_data", "cats_predictions"
+❌ Wrong: "host.json", "opendc/folder_x/folder_y/data"
+"""
+SIMULATION_DATA_FILE = "host"  # opendc outputs in file host.parquet


This file should be removed.
Given that this is the integrated version of m3sa, it can by definition not be used by other simulators.

true. will remove, nice catch.

[TODO#2]

…rror the opendc master branch

…ulti CPUs are combined into one. This is for performance and explainability. (atlarge-research#255) (#37) Co-authored-by: Dante Niewenhuis <d.niewenhuis@hotmail.com>

…ulti CPUs are combined into one. This is for performance and explainability. (atlarge-research#255) (#38) Co-authored-by: Dante Niewenhuis <d.niewenhuis@hotmail.com>

…ulti CPUs are combined into one. This is for performance and explainability. (atlarge-research#255) (#39) Co-authored-by: Dante Niewenhuis <d.niewenhuis@hotmail.com>

Radu-Nicolae · 2024-09-18T08:54:41Z

...xperiments-base/src/main/kotlin/org/opendc/experiments/base/scenario/specs/ExperimentSpec.kt

m3saSetup in ExperimentSpec -- non mandatory to setup, but part of an experiment, used when parsing the JSON file. this is part of the JSON file parser.

DanteNiewenhuis

The PR is better than the previous version, but I still have some issues.

First, you are still changing things in the ExperimentSpecs. This should not be needed.

Second, As far as I can see, the Python files have not been changed. This means that I will not understand what is happening.

Finally, it seems like there is no pre-processing being done. As far as I understand, the multi/meta model creates multiple models from the given input. I cannot find any such pre processing in the code at the moment.

DanteNiewenhuis · 2024-09-18T09:05:01Z

...xperiments-base/src/main/kotlin/org/opendc/experiments/base/scenario/specs/ExperimentSpec.kt

 @Serializable
 public data class ExperimentSpec(
    var id: Int = -1,
    var name: String = "",
    val outputFolder: String = "output",
    val initialSeed: Int = 0,
    val runs: Int = 1,
+    val m3saSetup: String = "",


Why is this defined in the ExperimentSpec?

DanteNiewenhuis · 2024-09-23T07:08:03Z

opendc-web/opendc-web-ui/package-lock.json

Remove this change

DanteNiewenhuis · 2024-09-23T07:12:25Z

...nts/opendc-experiments-m3sa/src/main/kotlin/org/opendc/experiments/m3sa/runner/M3SARunner.kt

As far as I can see, this file is not needed. You can just import runExperiment from experiment-base

If you want to do something different here at some point, you can create it later.

DanteNiewenhuis · 2024-09-23T07:14:24Z

...pendc-experiments-m3sa/src/main/kotlin/org/opendc/experiments/m3sa/scenario/M3SAFactories.kt

Why does this have to be a different file than just the ExperimentFactory?

Radu-Nicolae added 7 commits September 10, 2024 13:44

(feat) demo files are now ignored

52f3738

integrating m3sa changes with opendc

be9011f

gitignore ignores demo

1122bb4

m3sa linked, tested, works 🎉🎆

42ff227

linting & checks fully pass

cf91362

m3sa documentation (re...)added

f26cce2

package.json added, a potentail solution for Build Docker Images work…

8b7de0b

…flow

DanteNiewenhuis requested changes Sep 13, 2024

View reviewed changes

Radu-Nicolae and others added 18 commits September 14, 2024 11:53

(fix) opendc-m3sa renamed to opendc-experiments-m3sa

0932a99

(feat) Model is now a dataclass

918cc5c

(fix) package and package-lock reverted as before the PR, now they mi…

3dd068a

…rror the opendc master branch

(fix) Experiments renamed to experiment

3ba2843

Merge branch 'master' into m3sa-integration-pr-branch

c93cf01

branch updated with changes from master branch

70f4952

trying to fix the build docker image failed workflow

33ab751

trying to fix the build docker image failed workflow

864c1c4

All simulation are now run with a single CPU and single MemoryUnit. m…

2b67e2e

…ulti CPUs are combined into one. This is for performance and explainability. (atlarge-research#255) (#37) Co-authored-by: Dante Niewenhuis <d.niewenhuis@hotmail.com>

All simulation are now run with a single CPU and single MemoryUnit. m…

904faea

…ulti CPUs are combined into one. This is for performance and explainability. (atlarge-research#255) (#38) Co-authored-by: Dante Niewenhuis <d.niewenhuis@hotmail.com>

All simulation are now run with a single CPU and single MemoryUnit. m…

36fbcc7

…ulti CPUs are combined into one. This is for performance and explainability. (atlarge-research#255) (#39) Co-authored-by: Dante Niewenhuis <d.niewenhuis@hotmail.com>

Merge branch 'atlarge-research:master' into m3sa-integration-pr-branch

1e401a9

[TEMP](feat) m3saCli decoupled from experimentCli

b9107b1

spotless and minor refactoring

8909923

(feat)[TEMP] decoupling m3sa from experiment

2ef3776

spotless applied

5be1008

documentation resolved

f129706

requirements.txt added

59fbd93

Radu-Nicolae requested a review from DanteNiewenhuis September 18, 2024 08:52

Radu-Nicolae commented Sep 18, 2024

View reviewed changes

DanteNiewenhuis requested changes Sep 23, 2024

View reviewed changes


		self.max_model_len = min([len(model.raw_sim_data) for model in self.models])

		def compute_windowed_aggregation(self):

M3SA - Multi-Meta-Model Simulation Analyzer #251

Are you sure you want to change the base?

M3SA - Multi-Meta-Model Simulation Analyzer #251

Conversation

Radu-Nicolae commented Sep 10, 2024

Summary

Multi-Model

Meta-Model

M3SA

Paper

External Dependencies 🍀

Breaking API Changes ⚠️

DanteNiewenhuis left a comment • edited by Radu-Nicolae Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Radu-Nicolae Sep 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanteNiewenhuis left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DanteNiewenhuis left a comment •

edited by Radu-Nicolae

Loading

Radu-Nicolae Sep 13, 2024 •

edited

Loading