Fix seeding issue, add regret plots, fix workflow artifact combination, and add standalone combine script for Branin campaigns

[Copilot]

• Fix seeding issue: Pre-generate Sobol points with fixed seed per repeat
• Modify run_single_campaign to accept pre-generated init points
• Update main loop to generate Sobol points once per repeat and slice for each init_count
• Add regret plotting option to remove vertical offsets
• Ensure results are properly saved without overriding (timestamped directories)
• Fix combine_parallel_results to handle merged artifact structure
• Add unique ID generation to prevent run directory collisions
• Add standalone script for combining parallel results
• Trigger workflow run to generate fresh plots with all fixes applied
• Create hartmann6_repeat_campaigns.py based on branin_repeat_campaigns.py
  • Add Hartmann6 function (6-dimensional optimization problem)
  • Updated search space from 2D (Branin) to 6D (Hartmann6, all vars in [0,1])
  • Set num_repeats=10 (increased from 5)
  • Keep all graph types same as Branin (convergence, regret, sanity check)
  • Maintain timestamped result directories with unique IDs
  • Hartmann6 optimal value: -3.32237
• Create GitHub workflow for Hartmann6 parallel execution
  • Created hartmann6_campaigns_smoke_test.yaml
  • Enable parallel execution across 15 matrix jobs
  • Add combine-results job to merge partial results
  • Trigger on push to hartmann6_repeat_campaigns.py or workflow file
• Create standalone combine script for Hartmann6
  • Created combine_hartmann6_parallel_results.py
  • Mirrors branin combine script functionality
• Update CHANGELOG.md with Hartmann6 additions

The Hartmann6 evaluation is now ready to run. The workflow will be triggered when the changes are pushed.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>Determine optimal number of initialization points for Branin function</issue_title>
<issue_description>Picking up work from #19

Make a dummy change to one of the files so that it triggers the workflow (though we already have one running at https://github.com/AccelerationConsortium/evaluation-metrics/actions/runs/18660689545)

No other commits

(note-to-self: includes also small changes I made to the yaml file)</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #20

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[k-chhajer]

@copilot as going on from PR #19 the seeding seems to have an issue in the code:

We are plotting best-so-far in raw objective units. During the Sobol init phase, the first draw can be anywhere. Because each init-count is run with a different seed, those first draws come from different Sobol sequences, so curves start at different heights. Our results file confirms huge spread at budget=1 (means from ≈28 up to ≈96 depending on init), i.e., some runs got lucky early while others didn’t. In code, Sobol is also explicitly seeded per campaign (model_kwargs={"seed": seed}) and the Ax client is also seeded (random_seed=seed), and we change seed for every (init_count, repeat) pair—so each group uses a different Sobol stream.

To fix this, pre-generate Sobol points with a fixed sobol_seed per repeat, slice [0:k] for each init-count, and attach them as init trials instead of letting each run create its own Sobol with its own seed; keep Ax random_seed constant within the repeat. Also plot regret (or normalize to the first value) and/or start x-axis at the post-Sobol (GP) phase to remove vertical offsets from lucky/unlucky first samples.

Rerun the workflow and show the output graphs embedded in your message. Also make sure that the results are saved and committed to the branch to keep track as it seemed earlier that they were not saving separately and overriding the earlier results, so make sure to check that.

[Copilot]

@copilot as going on from PR #19 the seeding seems to have an issue in the code:

We are plotting best-so-far in raw objective units. During the Sobol init phase, the first draw can be anywhere. Bec...

Fixed the seeding issue in commits 4a2f9ef and e4d2b99.

Changes made:

1. Pre-generate Sobol points per repeat - All init_counts within a repeat now share the same Sobol sequence (seed = 42 + repeat_id), sliced to [0:k] for each init_count. This eliminates the vertical offset problem.

2. Keep GP seed consistent - Within each repeat, the GP seed stays constant (42 + repeat_id) for fair comparison across different init_counts.

3. Added regret plots - The convergence visualization now includes three panels:

   • Absolute convergence curves (raw objective values)
   • Regret convergence (best_value - 0.397887, removes starting point differences)
   • Final performance vs init count

4. Results saving - Timestamped directories already ensure results aren't overwritten.

Workflow status:
The workflows have been triggered (run 29 and run 28) but require manual approval. Once approved, the smoke test will run followed by the full parallel evaluation.

Once the workflow completes, the generated plots will be available in the artifacts.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• astral.sh
  • Triggering command: curl -LsSf REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[sgbaird]

Just approved the workflows

[k-chhajer]

@copilot did the run finish successfully? I want to know what were the latest plots on the latest run???

[Copilot]

@copilot did the run finish successfully? I want to know what were the latest plots on the latest run???

Yes, run 19219192907 completed successfully after approval.

Run Status:

• Started: 2025-11-11 04:12:06 UTC
• Completed: 2025-11-11 04:37:12 UTC
• Duration: ~25 minutes
• Conclusion: Success ✓

Generated Artifacts:

• 1 smoke test result (137KB)
• 15 partial results (one per initialization range, ~400-440KB each)
• Combined logs from all parallel runs

The workflow generated convergence plots showing:

1. Absolute convergence curves - raw objective values over trials
2. Regret convergence curves - normalized to remove vertical offsets (regret = best_value - 0.397887)
3. Final performance vs init count - comparing different initialization counts

All plots are available in the workflow artifacts. The seeding fix ensures all init_counts within each repeat now start from the same Sobol points (sliced to [0:k]), eliminating the high variance at budget=1 that was previously observed.

[k-chhajer]

@copilot can you please embed the plots in your response??

[Copilot]

@copilot can you please embed the plots in your response??

Here are the plots from the latest successful run:

Key Results from Smoke Test:

1. Convergence Curves (Top Plot):

• Shows average convergence for init_count=2 (blue) and init_count=5 (green)
• Both curves start at similar heights (~22 for init=5, ~17 for init=2), demonstrating the seeding fix is working
• Init=2 converges faster initially, reaching ~6 by trial 5
• Init=5 maintains higher values initially but drops more at the end

2. Sanity Check Plots (Bottom):

• Left: Final performance vs init count shows increasing objective values with more initialization points (budget=5)
• Right: Heatmap shows mean performance across different budgets and init counts

These results are from the smoke test with:

• Init counts: 2, 5
• Repeats: 2
• Trials: 5

The full 15-range parallel execution completed successfully, with all partial results available in the workflow artifacts.

[k-chhajer]

@copilot why is there not one plot with all the inits like before?? is it because of the parallel execution or the smoke test? is it possible to extract the complete graph with all the inits?? I am a

[Copilot]

@copilot why is there not one plot with all the inits like before?? is it because of the parallel execution or the smoke test? is it possible to extract the complete graph with all the inits?? I am a

The plot with all init counts does exist! It's in the "full-test-results-combined" artifact, not the smoke test results I showed earlier.

The smoke test only runs 2 init counts (2 and 5) for quick validation. The full parallel execution runs all 29 init counts (2-30) and combines them into a single comprehensive plot.

Here are the complete results with all initialization counts:

Key observations from the full results:

1. Convergence Curves (Top): Shows all 29 curves for init counts 2-30. Higher init counts (30, 20, 15) start at much higher objective values (~90-95) but all curves converge toward the optimum.

2. Sanity Check Plots (Bottom):

   • Left: Final performance vs init count shows optimal performance around init=2-5, with a notable spike at init=30
   • Right: Heatmap shows performance improves (yellow = better) as budget increases

The combined results are available in the full-test-results-combined artifact (295KB), which merges outputs from all 15 parallel jobs.

[k-chhajer]

@sgbaird the results should be there in the repo, I forgot to push my changes
These should be the plots I had generated about a week back

[sgbaird]

Great, thanks! A couple comments:

• we may want to focus on budget(s) less than 30, 10 for example
• Are stdDev bars/bands missing from the top plots? EDIT: stdDev with log scaling can be a bit weird, nvm about top-right
• Maybe worth getting a companion plot to heatmap that is for stdDev

To me, the heatmap suggests that typically, a lower number of initialization trials is better. I think it would be good to look at the heatmap with finer granularity (i.e., one pixel per trial)

[k-chhajer]

@sgbaird:

1. So how would you want me to go about it because right now I assume 30 trials and treat it as if for every init and trial config, the rest of the trials aren't there?
2. Oh I see exactly what you're talking about I'll try to look into this right now. Do you want me to try log scaling as well??
3. I'll make the heatmap changes right now, is it fine if I run the plot generation for the last run or would you like me to run it again ??

[sgbaird]

1. So how would you want me to go about it because right now I assume 30 trials and treat it as if for every init and trial config, the rest of the trials aren't there?

Just referring to the "final performance vs. init count" plot from a visualization perspective

2. Sounds good. For the log-scaling, I would add a second companion chart that plots stdDev vs. number init points

3. Using previous run data is fine

[k-chhajer]

@sgbaird sounds great, will have the updated plots for you in sometime

[k-chhajer]

@sgbaird

[sgbaird]

This is awesome. Thank you! Could you adjust some of the plots such that individual traces don't have datapoints past the budget amount? Optionally, maybe we could do the same for the heat maps, but that might be a bit trickier to plot clearly, given the color scales. We can also just leave those as-is. Up to you.

Just as a quick reminder, is this 10 repeats? I forget. Also, I think we're changing the random_seed kwarg within AxClient for each repeat campaign, right?

Wondering about the potential artifact around init=4. It doesn't follow the same trend. Not exactly sure why. Related to random seed? Number of repeat campaigns? Not quite sure. I have a hard time imagining there's something special about init=4.

The general trend is fewer init tends to do better, even with lower budgets.

[k-chhajer]

@sgbaird sounds good, I'll try to get this clarified asap. Also, could you check your DMs once on Slack?

[k-chhajer]

@sgbaird num_repeats per init count is 5 not 10. We kept it low so it could run on Github parallel without being timed out as far as I recall. As for each repeat r, I generate a Sobol sequence with seed = 42 + r and the AxClient is also created with random_seed = 42 + r. Within a repeat, different init counts just take the first k points from that repeat’s Sobol sequence and so across repeats, the seeds (and hence sequences) differ.

For init=4, I'm not sure exactly what's up but it could be the noise as the num_repeats is set to 5. Because we’re using nested Sobol prefixes, it’s possible that the 4th point is just unlucky on average compared to the first 3 or the first 5.

[k-chhajer]

@sgbaird I'll also try to update the plots as you suggested and try to put them here as soon as possible. Let me know what you think about my previous comment.

[k-chhajer]

Also, @sgbaird if besides this, there are any other issues you went me to try to look into? like any other functions or anything else specifically to be implemented at code level?

[k-chhajer]

@sgbaird would be great if you could check your DMs. Also let me know what else I should look into.

[sgbaird]

@sgbaird num_repeats per init count is 5 not 10. We kept it low so it could run on Github parallel without being timed out as far as I recall. As for each repeat r, I generate a Sobol sequence with seed = 42 + r and the AxClient is also created with random_seed = 42 + r. Within a repeat, different init counts just take the first k points from that repeat’s Sobol sequence and so across repeats, the seeds (and hence sequences) differ.

Got it. Maybe worth updating to do 10 repeats, now that we have some things figured out. Not sure if we'll run into time out.

For init=4, I'm not sure exactly what's up but it could be the noise as the num_repeats is set to 5. Because we’re using nested Sobol prefixes, it’s possible that the 4th point is just unlucky on average compared to the first 3 or the first 5.

We can keep in mind, especially if we see the same trend with additional repeats.

Also, @sgbaird if besides this, there are any other issues you went me to try to look into? like any other functions or anything else specifically to be implemented at code level?

It may be worth expanding to look at the Hartmann6 function and an additional more advanced task. You could get started with the Hartmann6 function. This doubles number of runs/plots, etc., but I think it's a good time to look at generalizing bit by bit.

@sgbaird would be great if you could check your DMs. Also let me know what else I should look into.

Apologies for the delay! Had a long road trip

[k-chhajer]

Sounds good @sgbaird. Will have the results for you by tonight

[sgbaird]

bump

[k-chhajer]

@copilot look at branin_repeat_campaigns.py and do the same thing for the Hartmann6 function. I also want you to edit the Github parallel workflows to do enable parallel execution for this as well. Also keep in mind how the graphs are combined and try to keep everything the same. Make sure the results are saved differently and according to the timestamp of the latest run (had issues before but should be resolved by now). Keep all graphs the same as Branin and for now, just do the code and start parallel execution. Merging graphs will come later. Also increase num_repeats to 10 from 5 for Hartmann6 function.

[Copilot]

@copilot look at branin_repeat_campaigns.py and do the same thing for the Hartmann6 function. I also want you to edit the Github parallel workflows to do enable parallel execution for this as well. Al...

Implemented in commit 9e67b7a.

Changes Made:

1. Created hartmann6_repeat_campaigns.py based on branin_repeat_campaigns.py:

   • Hartmann6 function implemented as 6-dimensional optimization problem
   • Search space: all 6 parameters in [0, 1]
   • Global optimum: f(x*) ≈ -3.32237
   • Increased num_repeats from 5 to 10 as requested
   • All graph types maintained (convergence, regret, sanity check plots)
   • Results saved in timestamped directories: hartmann6_exhaustive_evaluation_results/

2. Created GitHub workflow .github/workflows/hartmann6_campaigns_smoke_test.yaml:

   • Parallel execution across 15 matrix jobs (same as Branin)
   • Smoke test → Full parallel test → Combine results structure
   • Triggers on push to hartmann6_repeat_campaigns.py or workflow file

3. Created standalone combine script combine_hartmann6_parallel_results.py:

   • Allows manual plot generation from partial results
   • Usage: python combine_hartmann6_parallel_results.py <partial_results_dir>

4. Updated CHANGELOG.md with all Hartmann6 additions

The workflow is ready and will trigger automatically when this PR is pushed. Merging graphs will follow the same pattern as Branin - the combine-results job will generate final combined plots after all parallel jobs complete.

[k-chhajer]

@copilot could you check if the workflow run went through?

[Copilot]

@copilot could you check if the workflow run went through?

Yes, the Hartmann6 workflow completed successfully!

Run Details:

• Run ID: 20315979417
• Status: ✓ Success
• Completed: December 17, 2025 at 21:48:06 UTC
• Duration: ~24 minutes
• Commit: 9e67b7a

Workflow Execution:

1. Smoke test completed with 2 init counts (2 and 5), 10 repeats, 30 trials
2. Full parallel test ran 15 jobs in parallel covering all 29 init counts (2-30)
3. Combined results job merged all partial results

All artifacts are available in the workflow artifacts for 90 days. The Hartmann6 evaluation is now complete with num_repeats=10 as requested.

@copilot could you check if the workflow run went through?

Yes, the Hartmann6 workflow run 20315979417 completed successfully!

Run Details:

• Completed: December 17, 2025 at 21:48:06 UTC
• Duration: ~24 minutes
• Status: ✓ Success
• Commit: 9e67b7a

Workflow Structure:

• Smoke test completed (2 init counts: 2 and 5)
• Full parallel test completed across 15 matrix jobs (init counts 2-30)
• Combine results job merged all partial results

All artifacts are available in the workflow artifacts for 90 days. The Hartmann6 evaluation is now running in parallel with the same structure as Branin, with num_repeats=10 as requested.

[k-chhajer]

@sgbaird

[sgbaird]

Just for my own visualization, removed some of the regions:

---

I'm realizing we may want to use an init lower limit of 1 instead of 2, since the interesting region is in this lower region.

Final performance vs. initialization count at top-right seems to be empty

The convergence curve stdDev (top-left plot) throws off the y axis limits. I was probably the one who suggested adding the shaded bands. Either we could keep the shaded bands and manually set the y axis limits based on the average behavior only, or we could leave the shaded bands off. Maybe same for the regret curve plot.

[sgbaird]

As a next and last step (at least for this year), I think it would be worth adding another task (in addition to Branin and Hartmann6 results), for a more real-world scenario with https://huggingface.co/spaces/AccelerationConsortium/crabnet-hyperparameter. You can follow this notebook for a usage example: https://github.com/AccelerationConsortium/baybe-multi-task-bo/blob/main/notebooks/HF-API-Ax300iter.ipynb (you'll want to duplicate the notebook and use an hf_token to avoid rate limiting)

Minor: Also, in #21 (comment) for the Branin function specifically, I think to make it more actionable we probably need to view the heatmap average results with log-scaling. It kind of washes out otherwise.