From 34ed42c6dbb60456c3d2f18e756d7eabd1033865 Mon Sep 17 00:00:00 2001 From: louispt1 Date: Fri, 27 Feb 2026 15:43:03 +0100 Subject: [PATCH 1/2] Extend myetm readme to cover local database set up and import-scenarios usage --- README.md | 273 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 272 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 875a1d26..03856da5 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ docker-compose build ```sh docker-compose run --rm web bash -c 'chmod +x bin/setup && bin/rails db:drop && bin/setup' ``` -⚠️ This command drops any existing database. Use only during initial setup! +This command drops any existing database. Use only during initial setup! For updates, install new dependencies using: ```sh @@ -120,3 +120,274 @@ After setting up MyETM, configure it to communicate with ETEngine and ETModel: 3. Create new applications for **ETEngine (Local)**, **ETModel (Local)** and **Collections (Local)**. 4. Copy the generated configuration into `config/settings.local.yml` for both ETEngine and ETModel. Copy it into `.env.local` for Collections. + +--- + +## Database Setup and Importing Scenarios (Admin Only) + +This guide covers setting up your local ETM databases and importing scenarios using the dump-n-load architecture. + +**Important:** MyETM must be set up **first** because it's the OAuth provider for ETEngine and ETModel. Follow the steps in order. + +### Initial Database Setup + +**Note: If using Docker these steps are not necessary** + +#### 1. Install and Start MySQL + +```bash +brew install mysql +brew services start mysql +``` + +#### 2. Set Up MyETM Database (FIRST!) + +MyETM is the authentication provider for the entire ETM ecosystem, so it must be configured before ETEngine or ETModel. + +```bash +cd myetm +bundle install +bin/rails db:prepare +``` + +This creates the database, runs migrations, and seeds an admin user. **Save the password displayed in the output:** + +``` ++------------------------------------------------------------------------------+ +| Created admin user 'Seeded Admin' with password: aBc123Xy | +| and email: seeded_admin@localdevelopment.com. | ++------------------------------------------------------------------------------+ +``` + +#### 3. Start MyETM + +```bash +bin/dev -p 3002 +# Or: rails s -p 3002 +``` + +Wait for "Listening on..." before proceeding. + +#### 4. Set Up ETEngine Database + +```bash +cd etengine +bundle install +bin/rails db:prepare +``` + +**Warning:** ETEngine requires ETSource to be cloned in the same parent directory. If you get errors about missing datasets, ensure etsource is cloned alongside etengine. + +#### 5. Start ETEngine + +```bash +bin/dev -p 3000 +# Or: rails s -p 3000 +``` + +**Note:** The first time you create a scenario, ETEngine calculates the base dataset (Netherlands). This takes 5-15 seconds. Be patient! + +#### 6. Verify Both Servers Are Running + +- MyETM: http://localhost:3002 +- ETEngine: http://localhost:3000 + +You should be able to access both URLs in your browser. + +### Getting .etm Files for Import + +To import scenarios, you first need a `.etm` file. There are two ways to get one: + +#### Option 1: Export from MyETM Admin Interface (Recommended) + +If you have access to a production or staging MyETM instance: + +1. **Log in as admin** + +2. **Navigate to admin section** + - Click "Admin" in the navigation + - Click "Saved Scenarios" in the admin menu + - URL: `https://my.energytransitionmodel.com/admin/saved_scenarios` + +3. **Filter for scenarios** + - Check "Featured" to show only featured scenarios + - Filter by version, end year, or area code as needed + +4. **Select scenarios to export** + - Click checkboxes next to desired scenarios + - Or click "Select all" to export all filtered scenarios + +5. **Export** + - Click the "Export" button + - A `.etm` file will download (format: `YYYYMMDDHHMM_env.etm`) + +6. **Save the file** + - File downloads to your Downloads folder by default + - Ready to import to your local environment + +### Importing Scenarios + +With both servers running and a `.etm` file ready: + +#### 1. Run the Import Script + +```bash +cd myetm +bin/import-scenarios +``` + +The script will: +1. Detect any `.etm` file +2. Warn you that import will modify your database +3. Prompt for confirmation +4. Load scenarios via ETEngine API +5. Create SavedScenario records in MyETM +6. Display results and ID mappings + +#### 2. Verify the Import + +Check the output for: +- Number of scenarios loaded +- Scenario ID mappings (production ID → local ID) +- Any warnings about missing data + +#### 3. Access Imported Scenarios + +- View in MyETM: http://localhost:3002 +- Open in ETModel: http://localhost:3001 (requires ETModel setup) + +### Command Options + +#### Specify Owner + +```bash +bin/import-scenarios --user your.email@example.com +``` + +By default, scenarios are owned by the first admin user. + +#### Handle Duplicates + +Control what happens when a scenario ID already exists: + +```bash +# Always update existing scenarios (default) +bin/import-scenarios --on-dup update + +# Always create new scenarios +bin/import-scenarios --on-dup create + +# Prompt for each duplicate +bin/import-scenarios --on-dup prompt +``` + +#### Search in Different Directory + +```bash +# Look for .etm files in db/dumps instead of ~/Downloads +bin/import-scenarios db/dumps + +# Or specify exact file path +bin/import-scenarios /path/to/scenarios.etm +``` + +#### Get Full Help + +```bash +bin/import-scenarios --help +``` + +### Troubleshooting + +#### ETEngine Connection Errors + +**Problem:** Import fails with "Connection refused" or timeout errors + +**Solution:** Ensure ETEngine is running on port 3000: + +```bash +cd etengine +bin/dev -p 3000 +# Wait for "Listening on..." +``` + +Check that you can access http://localhost:3000 in your browser. + +#### MyETM Not Running + +**Problem:** Script error: "Database connection failed" or similar + +**Solution:** The import-scenarios script runs within MyETM's context. Ensure the database is set up: + +```bash +cd myetm +bin/rails db:prepare +``` + +#### No Admin User + +**Problem:** "No users found in database!" + +**Solution:** Re-run the seeds: + +```bash +cd myetm +bin/rails db:seed +# Save the password displayed! +# You can then create an account with your own credentials and make yourself an admin if you like! +``` + +#### Duplicate Scenario Prompts + +**Problem:** Import keeps asking about duplicate scenarios + +**Solution:** Use `--on-dup update` to automatically update existing scenarios: + +```bash +bin/import-scenarios --on-dup update ~/Downloads/scenarios.etm +``` + +#### File Not Found + +**Problem:** "No .etm scenario dump files found" + +**Solution:** +- Ensure the file has a `.etm` extension +- Check you're looking in the right directory (defaults to `~/Downloads`) +- Specify the file path directly: `bin/import-scenarios /path/to/file.etm` + +### What Gets Imported + +Each `.etm` file contains: + +**From ETEngine:** +- Slider values (user_values) +- Custom hourly curves +- Merit order configurations +- Area code, end year, privacy settings + +**From MyETM:** +- Scenario title and description +- User permissions (owner, collaborators, viewers) +- Version tags +- Scenario ID history (tracks origin from production) + +### Exporting Scenarios to Share + +To create a `.etm` file from your local scenarios to share with your team: + +1. **Access MyETM admin:** http://localhost:3002/admin/saved_scenarios +2. **Select scenarios** you want to export using checkboxes +3. **Click "Export"** button +4. **Share the downloaded `.etm` file** with your team + +They can then import it using `bin/import-scenarios`. + +### Next Steps + +After importing scenarios: + +1. Access them in MyETM at http://localhost:3002 +2. Set up ETModel to view and modify scenarios in the UI +3. Use imported scenarios as starting points for development +4. Export modified scenarios to share with your team From 09ccc03af00c0e6099916935385ac9191df3bea8 Mon Sep 17 00:00:00 2001 From: louispt1 Date: Mon, 2 Mar 2026 09:32:19 +0100 Subject: [PATCH 2/2] Improvements based on review --- README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 03856da5..5b80358a 100644 --- a/README.md +++ b/README.md @@ -206,11 +206,11 @@ If you have access to a production or staging MyETM instance: 2. **Navigate to admin section** - Click "Admin" in the navigation - - Click "Saved Scenarios" in the admin menu + - Click "All Scenarios" in the admin menu - URL: `https://my.energytransitionmodel.com/admin/saved_scenarios` 3. **Filter for scenarios** - - Check "Featured" to show only featured scenarios + - Check "Show Featured" to show only featured scenarios - Filter by version, end year, or area code as needed 4. **Select scenarios to export** @@ -218,7 +218,7 @@ If you have access to a production or staging MyETM instance: - Or click "Select all" to export all filtered scenarios 5. **Export** - - Click the "Export" button + - Click the "Export selected" button - A `.etm` file will download (format: `YYYYMMDDHHMM_env.etm`) 6. **Save the file** @@ -363,8 +363,9 @@ Each `.etm` file contains: **From ETEngine:** - Slider values (user_values) - Custom hourly curves -- Merit order configurations +- User sortables - Area code, end year, privacy settings +- Balanced values **From MyETM:** - Scenario title and description @@ -376,9 +377,9 @@ Each `.etm` file contains: To create a `.etm` file from your local scenarios to share with your team: -1. **Access MyETM admin:** http://localhost:3002/admin/saved_scenarios +1. **Access MyETM admin:** https://my.energytransitionmodel.com/admin/saved_scenarios 2. **Select scenarios** you want to export using checkboxes -3. **Click "Export"** button +3. **Click "Export selected"** button 4. **Share the downloaded `.etm` file** with your team They can then import it using `bin/import-scenarios`.