From ff223521efbea90990eebbb71b5b1fe9444e9f34 Mon Sep 17 00:00:00 2001 From: Amr Akmal Moustafa Abouelmagd Date: Fri, 23 May 2025 13:08:09 -0700 Subject: [PATCH 1/4] Benchpark new developer docs --- .../benchpark-developer-docs.rst | 288 ++++++++++++++++++ docs/docs-requirements.txt | 4 + docs/index.rst | 6 + 3 files changed, 298 insertions(+) create mode 100644 docs/developer-docs/benchpark-developer-docs.rst create mode 100644 docs/docs-requirements.txt diff --git a/docs/developer-docs/benchpark-developer-docs.rst b/docs/developer-docs/benchpark-developer-docs.rst new file mode 100644 index 000000000..7e91ce64f --- /dev/null +++ b/docs/developer-docs/benchpark-developer-docs.rst @@ -0,0 +1,288 @@ +=============================================== +Benchpark Design Concepts and Command Workflow +=============================================== + +This document outlines the important concepts and patterns used consistently in Benchpark (BP) code design, along with detailed explanations of key commands and their workflow. + +Table of Contents +================= + +This document covers the following main sections: + +* `Core Concepts and Patterns`_ - Understanding Variants, Consistent Flow, and Modifiers +* `Benchpark Command Workflow`_ - Step-by-step execution of key Benchpark commands +* `Debugging Benchpark with VSCode`_ - Setting up debugging environment for development + +Core Concepts and Patterns +========================== + +1. Variants +----------- + +Variants are predefined keywords that execute specific logic and set configurations for benchmarks based on whether these keywords are found in user arguments. + +**How Variants Work:** + +* All possible variants are evaluated at the beginning of the ``concretize()`` function +* This function is called by classes that directly inherits the ``Spec`` class (e.g., ``SystemSpec``, ``ExperimentSpec``) +* We collect all possible variants (keywords that, when present, lead to specific configurations being set) +* These configurations affect the experiments to run, such as adding specific libraries or tools, and their effect is dumped to configuration files (mainly yaml files) so later Spack and Ramble can procide the needed dependencies/libraries requested. + +**Determining Acceptable Variants:** + +In both ``benchpark system`` and ``benchpark experiment`` commands, you'll find this pattern:: + + cls = ....get_obj_class(self.name) + +* ``self`` contains the extra arguments passed in the user command +* ``self.name`` contains the extra arguments passed in the user command +* We get the parent directory of the Python class path responsible for creating the ``self.name`` object +* For example, in ``benchpark system ... command``, this is set when creating the ``SystemSpec`` object that calls its parent class ``Spec``, right before executing ``concretize()`` +* We depend on Ramble for resolving this + +**Example:** + +For the command:: + + benchpark system init --dest=amr-ruby-system llnl-cluster cluster=ruby + +* ``self.name`` will be resolved to ``llnl-cluster`` +* This resolves to get the path of the ``LlnlCluster`` class that contains the variants it accepts + +**Variant Processing:** + +1. Ramble gets all variants (and dependent variants from any imports of the class) and registers them +2. We check user-provided variants against all acceptable variants to ensure they are as expected +3. Passed user variants are added to generated YAML files +4. Later, we import needed modifiers or load required libraries/tools to provide the requested functionality + +2. Consistent Flow for Benchpark Commands +----------------------------------------- + +**Flow Overview:** + +1. **Entry Point:** ``benchpark`` → ``lib/main.py`` + + * Gets needed imports, including ``cmd.experiment`` + * Executes ``benchpark.experiment`` (``lib/benchpark/experiment.py``) + * Checks if Ramble and Spack exist in home directory + * Clones and installs them if they don't exist + +2. **Command Processing:** + + * Lists all acceptable commands + * Parses user command to determine which action to execute (e.g., experiment, system) + * Executes the corresponding command function: + + * ``experiment.py`` → ``command()`` + * ``system.py`` → ``command()`` + * etc. + +3. Modifiers +------------ + +Modifiers provide extra functionality to: + +* Help gather more information (such as affinity data) +* Enable tools (such as Caliper) to produce ``.cali`` files for later analysis using Thicket + +Benchpark Command Workflow +========================== + +1. System Initialization +------------------------ + +**Command:** + +.. code-block:: bash + + benchpark system init --dest=amr-ruby-system llnl-cluster cluster=ruby + +**Generated Output:** + +* **YAML files:** Define software needed (to be installed by Spack later) +* **YAML files:** Define the LLNL cluster system to be used + +**Generated Directory Structure:** ``amr-ruby-system/`` + +* ``variables.yaml``: Contains configurations for job execution (#nodes, #cores, etc.) + +2. Experiment Initialization +---------------------------- + +**Command:** + +.. code-block:: bash + + benchpark experiment init --dest=amg2023-benchmark amg2023 +openmp + +**Generated Directory Structure:** ``amg2023-benchmark/`` + +* ``ramble.yaml``: + + * Defines problems and experiments to run + * Contains required modifiers and packages to be installed + +3. Setup Workspace +------------------ + +**Command:** + +.. code-block:: bash + + benchpark setup ./amr-amg2023-benchmark ./amr-ruby-system amr-workspace/ + +**Generated Directory Structure:** ``amr-workspace/`` + +**New Directory Created:** ``amr-amg2023/amr-ruby-system/workspace`` + +* Based on definitions in ``amr-ruby-system`` and ``amr-amg2023-benchmark`` +* Contains everything as defined before this step +* Creates necessary scripts for configuring Ramble and Spack installation + +4. Environment Setup +-------------------- + +**Command:** + +.. code-block:: bash + + . amr-workspace/setup.sh + +**Purpose:** + +* Runs two simple scripts to set up Ramble and Spack previously installed for next steps + +5. Workspace Configuration +-------------------------- + +**Location:** + +.. code-block:: bash + + cd ./amr-workspace/amr-amg2023-benchmark/amr-ruby-system/workspace/ + +**Command:** + +.. code-block:: bash + + ramble --workspace-dir . --disable-progress-bar workspace setup + +**What Happens:** + +* **Spack Role:** Installs all needed software +* **Ramble Role:** Sets up all experiments/problems defined previously for execution +* **Benchpark Role:** Complete at this point + +**Generated Content:** + +* New directory ``experiments`` is created +* Contains script ``execute_experiment`` for the next step +* This script is the main one that Ramble runs to fire jobs for experiments on the HPC cluster +* Script includes all needed steps to run experiments using: + + * Requested benchmark + * Configurations + * Packages + * Libraries + * etc. + +6. Experiment Execution +----------------------- + +**Command:** + +.. code-block:: bash + + ramble --disable-progress-bar --workspace-dir . on + +**Purpose:** + +* Runs the ``execute_experiment`` script generated from the previous step +* Execution occurs on the chosen LLNL cluster (e.g., Ruby or Dane) + +Debugging Benchpark with VSCode +=============================== + +This section provides a step-by-step guide for setting up VSCode debugging for Benchpark development. + +Setup Instructions +------------------ + +1. **Install Python Debugger Extension** + + Install the official Python debugger extension in VSCode. + +2. **Create VSCode Configuration Directory** + + In the ``benchpark/`` root directory: + + .. code-block:: bash + + mkdir .vscode + +3. **Create Launch Configuration File** + + .. code-block:: bash + + touch .vscode/launch.json + +4. **Configure Debug Settings** + + Paste the following configuration into ``launch.json``: + + .. code-block:: json + + { + "version": "0.2.0", + "configurations": [ + { + "name": "Debug benchpark init (ruby)", + "type": "debugpy", + "request": "launch", + "program": "${workspaceFolder}/bin/benchpark", + "args": [ + "system", + "init", + "--dest=amr-ruby-system", + "llnl-cluster", + "cluster=ruby" + ], + "console": "integratedTerminal" + }, + { + "name": "Debug benchpark init (quartz)", + "type": "debugpy", + "request": "launch", + "program": "${workspaceFolder}/bin/benchpark", + "args": [ + "system", + "init", + "--dest=amr-quartz-system", + "llnl-cluster", + "cluster=quartz" + ], + "console": "integratedTerminal" + }, + { + "name": "Debug benchpark list clusters", + "type": "debugpy", + "request": "launch", + "program": "${workspaceFolder}/bin/benchpark", + "args": [ + "cluster", + "list" + ], + "console": "integratedTerminal" + } + ] + } + +5. **Usage Notes** + + This configuration provides debugging examples for the commands: + + * ``benchpark system init ...`` + * ``benchpark experiment init ...`` + + Additional commands can be added by appending them to the ``configurations`` array and properly setting the attributes for each new debug configuration. \ No newline at end of file diff --git a/docs/docs-requirements.txt b/docs/docs-requirements.txt new file mode 100644 index 000000000..9c7626e23 --- /dev/null +++ b/docs/docs-requirements.txt @@ -0,0 +1,4 @@ +sphinx +sphinx_rtd_theme +sphinxcontrib-programoutput +pandas diff --git a/docs/index.rst b/docs/index.rst index fbbfff618..6b62725b0 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -54,6 +54,12 @@ scripts developer-scripts +.. toctree:: + :maxdepth: 1 + :caption: Developer Docs + + developer-docs/benchpark-developer-docs.rst + .. toctree:: :maxdepth: 1 :caption: Contributing From 6aa2f5fe77ccbcba024f1a85ef3899b794b4c980 Mon Sep 17 00:00:00 2001 From: Amr Akmal Moustafa Abouelmagd Date: Fri, 23 May 2025 14:54:49 -0700 Subject: [PATCH 2/4] fixes --- .../benchpark-developer-docs.rst | 69 ++++++++++--------- 1 file changed, 36 insertions(+), 33 deletions(-) diff --git a/docs/developer-docs/benchpark-developer-docs.rst b/docs/developer-docs/benchpark-developer-docs.rst index 7e91ce64f..d66dd2739 100644 --- a/docs/developer-docs/benchpark-developer-docs.rst +++ b/docs/developer-docs/benchpark-developer-docs.rst @@ -25,7 +25,7 @@ Variants are predefined keywords that execute specific logic and set configurati * All possible variants are evaluated at the beginning of the ``concretize()`` function * This function is called by classes that directly inherits the ``Spec`` class (e.g., ``SystemSpec``, ``ExperimentSpec``) -* We collect all possible variants (keywords that, when present, lead to specific configurations being set) +* We get all possible acceptable variants (args/keywords that, when present, lead to specific configurations being set) that are applicable for the required benchmark/experiment. * These configurations affect the experiments to run, such as adding specific libraries or tools, and their effect is dumped to configuration files (mainly yaml files) so later Spack and Ramble can procide the needed dependencies/libraries requested. **Determining Acceptable Variants:** @@ -34,17 +34,30 @@ In both ``benchpark system`` and ``benchpark experiment`` commands, you'll find cls = ....get_obj_class(self.name) -* ``self`` contains the extra arguments passed in the user command -* ``self.name`` contains the extra arguments passed in the user command +* ``self`` contains all the Specs, in other words the extra arguments passed in the user command, defined for the current context +* ``self.name`` contains the name of the core component spec relative to the current context, examples: + + * ``llnl-cluster`` in benchmark system commands running BP on LLNL systems + + * At executing benchpark system command (``benchpark system init --dest=.... llnl-cluster cluster=...``), ``self.name`` resolves to ``llnl-cluster`` since it is the core component, as it defines the system we will run our experiments on + + * ``saxpy`` when running benchmark experiment command + + * At executing benchpark experiment command (``benchpark experiment init --dest=... saxpy``), ``self.name`` resolves to ``saxpy`` as it is the core component that defines the benchmark that will be used +* A valid question could be, why not the ``cluster=ruby`` be used as ``self.name``? + + * Based on my experiments, order of args passed to benchpark does really matters. By default our parser expectes key-value pairs to be at the end, and prior is the Spec name. + * We get the parent directory of the Python class path responsible for creating the ``self.name`` object -* For example, in ``benchpark system ... command``, this is set when creating the ``SystemSpec`` object that calls its parent class ``Spec``, right before executing ``concretize()`` +* ``self.name`` is set when creating the corresponding ChildSpec object, (``SystemSpec``, ``ExperimentSpec``) where they inherit from ``Spec`` class (the parent class). This is done right before executing ``concretize()`` * We depend on Ramble for resolving this + **Example:** For the command:: - benchpark system init --dest=amr-ruby-system llnl-cluster cluster=ruby + benchpark system init --dest=test-ruby-system llnl-cluster cluster=ruby * ``self.name`` will be resolved to ``llnl-cluster`` * This resolves to get the path of the ``LlnlCluster`` class that contains the variants it accepts @@ -96,14 +109,14 @@ Benchpark Command Workflow .. code-block:: bash - benchpark system init --dest=amr-ruby-system llnl-cluster cluster=ruby + benchpark system init --dest=test-ruby-system llnl-cluster cluster=ruby **Generated Output:** * **YAML files:** Define software needed (to be installed by Spack later) * **YAML files:** Define the LLNL cluster system to be used -**Generated Directory Structure:** ``amr-ruby-system/`` +**Generated Directory Structure:** ``test-ruby-system/`` * ``variables.yaml``: Contains configurations for job execution (#nodes, #cores, etc.) @@ -114,9 +127,9 @@ Benchpark Command Workflow .. code-block:: bash - benchpark experiment init --dest=amg2023-benchmark amg2023 +openmp + benchpark experiment init --dest=test-amg2023-benchmark amg2023 +openmp -**Generated Directory Structure:** ``amg2023-benchmark/`` +**Generated Directory Structure:** ``test-amg2023-benchmark/`` * ``ramble.yaml``: @@ -130,13 +143,13 @@ Benchpark Command Workflow .. code-block:: bash - benchpark setup ./amr-amg2023-benchmark ./amr-ruby-system amr-workspace/ + benchpark setup ./test-amg2023-benchmark ./test-ruby-system test-workspace/ -**Generated Directory Structure:** ``amr-workspace/`` +**Generated Directory Structure:** ``test-workspace/`` -**New Directory Created:** ``amr-amg2023/amr-ruby-system/workspace`` +**New Directory Created:** ``test-amg2023-benchmark/test-ruby-system/workspace`` -* Based on definitions in ``amr-ruby-system`` and ``amr-amg2023-benchmark`` +* Based on definitions in ``test-ruby-system`` and ``test-amg2023-benchmark`` * Contains everything as defined before this step * Creates necessary scripts for configuring Ramble and Spack installation @@ -147,7 +160,7 @@ Benchpark Command Workflow .. code-block:: bash - . amr-workspace/setup.sh + . test-workspace/setup.sh **Purpose:** @@ -160,7 +173,7 @@ Benchpark Command Workflow .. code-block:: bash - cd ./amr-workspace/amr-amg2023-benchmark/amr-ruby-system/workspace/ + cd ./test-workspace/test-amg2023-benchmark/test-ruby-system/workspace/ **Command:** @@ -237,41 +250,31 @@ Setup Instructions "version": "0.2.0", "configurations": [ { - "name": "Debug benchpark init (ruby)", + "name": "Debug benchpark system init", "type": "debugpy", "request": "launch", "program": "${workspaceFolder}/bin/benchpark", "args": [ "system", "init", - "--dest=amr-ruby-system", + "--dest=test-ruby-system", "llnl-cluster", "cluster=ruby" ], "console": "integratedTerminal" }, { - "name": "Debug benchpark init (quartz)", + "name": "Debug benchpark experiment init", "type": "debugpy", "request": "launch", "program": "${workspaceFolder}/bin/benchpark", "args": [ - "system", + "experiment", "init", - "--dest=amr-quartz-system", - "llnl-cluster", - "cluster=quartz" - ], - "console": "integratedTerminal" - }, - { - "name": "Debug benchpark list clusters", - "type": "debugpy", - "request": "launch", - "program": "${workspaceFolder}/bin/benchpark", - "args": [ - "cluster", - "list" + "--dest=test-saxpy-benchmark", + "saxpy", + "+openmp", + "affinity=mpi" ], "console": "integratedTerminal" } From eb6aac3aa9e65b39079986f8462cb89a33a54e51 Mon Sep 17 00:00:00 2001 From: Amr Akmal <12779125+amroakmal@users.noreply.github.com> Date: Sun, 25 May 2025 19:34:24 -0700 Subject: [PATCH 3/4] Update benchpark-developer-docs.rst --- docs/developer-docs/benchpark-developer-docs.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/developer-docs/benchpark-developer-docs.rst b/docs/developer-docs/benchpark-developer-docs.rst index d66dd2739..7d14b1558 100644 --- a/docs/developer-docs/benchpark-developer-docs.rst +++ b/docs/developer-docs/benchpark-developer-docs.rst @@ -26,7 +26,7 @@ Variants are predefined keywords that execute specific logic and set configurati * All possible variants are evaluated at the beginning of the ``concretize()`` function * This function is called by classes that directly inherits the ``Spec`` class (e.g., ``SystemSpec``, ``ExperimentSpec``) * We get all possible acceptable variants (args/keywords that, when present, lead to specific configurations being set) that are applicable for the required benchmark/experiment. -* These configurations affect the experiments to run, such as adding specific libraries or tools, and their effect is dumped to configuration files (mainly yaml files) so later Spack and Ramble can procide the needed dependencies/libraries requested. +* These configurations affect the experiments to run, such as adding specific libraries or tools, and their effect is dumped to configuration files (mainly yaml files) so later Spack and Ramble can provide the needed dependencies/libraries requested. **Determining Acceptable Variants:** @@ -46,7 +46,7 @@ In both ``benchpark system`` and ``benchpark experiment`` commands, you'll find * At executing benchpark experiment command (``benchpark experiment init --dest=... saxpy``), ``self.name`` resolves to ``saxpy`` as it is the core component that defines the benchmark that will be used * A valid question could be, why not the ``cluster=ruby`` be used as ``self.name``? - * Based on my experiments, order of args passed to benchpark does really matters. By default our parser expectes key-value pairs to be at the end, and prior is the Spec name. + * Based on my experiments, order of args passed to benchpark does really matters. By default our parser expects key-value pairs to be at the end, and prior is the Spec name. * We get the parent directory of the Python class path responsible for creating the ``self.name`` object * ``self.name`` is set when creating the corresponding ChildSpec object, (``SystemSpec``, ``ExperimentSpec``) where they inherit from ``Spec`` class (the parent class). This is done right before executing ``concretize()`` @@ -288,4 +288,4 @@ Setup Instructions * ``benchpark system init ...`` * ``benchpark experiment init ...`` - Additional commands can be added by appending them to the ``configurations`` array and properly setting the attributes for each new debug configuration. \ No newline at end of file + Additional commands can be added by appending them to the ``configurations`` array and properly setting the attributes for each new debug configuration. From 84a6e8d4e42339dc19ca735657f649aac7c65ce9 Mon Sep 17 00:00:00 2001 From: Amr Akmal Moustafa Abouelmagd Date: Wed, 11 Jun 2025 13:37:36 -0700 Subject: [PATCH 4/4] Resolved comments --- .../benchpark-developer-docs.rst | 25 +++++++++---------- docs/docs-requirements.txt | 4 --- 2 files changed, 12 insertions(+), 17 deletions(-) delete mode 100644 docs/docs-requirements.txt diff --git a/docs/developer-docs/benchpark-developer-docs.rst b/docs/developer-docs/benchpark-developer-docs.rst index 7d14b1558..189f4d16b 100644 --- a/docs/developer-docs/benchpark-developer-docs.rst +++ b/docs/developer-docs/benchpark-developer-docs.rst @@ -2,7 +2,7 @@ Benchpark Design Concepts and Command Workflow =============================================== -This document outlines the important concepts and patterns used consistently in Benchpark (BP) code design, along with detailed explanations of key commands and their workflow. +This document outlines the important concepts and patterns used consistently in Benchpark code design, along with detailed explanations of key commands and their workflow. Table of Contents ================= @@ -19,7 +19,7 @@ Core Concepts and Patterns 1. Variants ----------- -Variants are predefined keywords that execute specific logic and set configurations for benchmarks based on whether these keywords are found in user arguments. +Variants are directives that execute specific logic and set configurations for experiments, systems, and packages. Variants are of the form `name=value`, where the `value` is selected from a set of possible values, defined by the given experiment, system, or package. **How Variants Work:** @@ -37,7 +37,7 @@ In both ``benchpark system`` and ``benchpark experiment`` commands, you'll find * ``self`` contains all the Specs, in other words the extra arguments passed in the user command, defined for the current context * ``self.name`` contains the name of the core component spec relative to the current context, examples: - * ``llnl-cluster`` in benchmark system commands running BP on LLNL systems + * ``llnl-cluster`` in benchmark system commands running Benchpark on LLNL systems * At executing benchpark system command (``benchpark system init --dest=.... llnl-cluster cluster=...``), ``self.name`` resolves to ``llnl-cluster`` since it is the core component, as it defines the system we will run our experiments on @@ -49,7 +49,7 @@ In both ``benchpark system`` and ``benchpark experiment`` commands, you'll find * Based on my experiments, order of args passed to benchpark does really matters. By default our parser expects key-value pairs to be at the end, and prior is the Spec name. * We get the parent directory of the Python class path responsible for creating the ``self.name`` object -* ``self.name`` is set when creating the corresponding ChildSpec object, (``SystemSpec``, ``ExperimentSpec``) where they inherit from ``Spec`` class (the parent class). This is done right before executing ``concretize()`` +* ``self.name`` is set when creating the corresponding child object, (``SystemSpec``, ``ExperimentSpec``) where they inherit from ``Spec`` class (the parent class). This is done right before executing ``concretize()`` * We depend on Ramble for resolving this @@ -79,7 +79,7 @@ For the command:: * Gets needed imports, including ``cmd.experiment`` * Executes ``benchpark.experiment`` (``lib/benchpark/experiment.py``) * Checks if Ramble and Spack exist in home directory - * Clones and installs them if they don't exist + * Clones Ramble and Spack libraries in the home directory (`~/.benchpark`) if they do not exist. 2. **Command Processing:** @@ -133,10 +133,10 @@ Benchpark Command Workflow * ``ramble.yaml``: - * Defines problems and experiments to run + * Defines experiment variables, which ramble uses to set up experiments. * Contains required modifiers and packages to be installed -3. Setup Workspace +3. Setup Benchpark Workspace ------------------ **Command:** @@ -151,7 +151,7 @@ Benchpark Command Workflow * Based on definitions in ``test-ruby-system`` and ``test-amg2023-benchmark`` * Contains everything as defined before this step -* Creates necessary scripts for configuring Ramble and Spack installation +* Clones Ramble and Spack into the Benchpark workspace 4. Environment Setup -------------------- @@ -184,14 +184,14 @@ Benchpark Command Workflow **What Happens:** * **Spack Role:** Installs all needed software -* **Ramble Role:** Sets up all experiments/problems defined previously for execution +* **Ramble Role:** Sets up all experiments/problems defined in the experiment `ramble.yaml` * **Benchpark Role:** Complete at this point **Generated Content:** -* New directory ``experiments`` is created +* New directory ``experiments`` is created for each experiment in the `ramble.yaml` * Contains script ``execute_experiment`` for the next step -* This script is the main one that Ramble runs to fire jobs for experiments on the HPC cluster +* This is a job script submitted to the scheduler * Script includes all needed steps to run experiments using: * Requested benchmark @@ -211,8 +211,7 @@ Benchpark Command Workflow **Purpose:** -* Runs the ``execute_experiment`` script generated from the previous step -* Execution occurs on the chosen LLNL cluster (e.g., Ruby or Dane) +* Submits the ``execute_experiment`` job script generated from the previous step to the scheduler Debugging Benchpark with VSCode =============================== diff --git a/docs/docs-requirements.txt b/docs/docs-requirements.txt deleted file mode 100644 index 9c7626e23..000000000 --- a/docs/docs-requirements.txt +++ /dev/null @@ -1,4 +0,0 @@ -sphinx -sphinx_rtd_theme -sphinxcontrib-programoutput -pandas