Simplify _Q_opt in parmest with block scenario structure.

[sscini]

Fixes # .

Summary/Motivation:

_Q_opt, the main model building and solving function within parmest, is still heavily dependent on MPISPPY, and the code is becoming outdated and difficult to interpret. The goal of this PR is to redesign _Q_opt to work without this dependence, retaining all the current functionality

Changes proposed in this PR:

-_Q_opt redesign, using a block structure, with minimal changes needed to functions that utilize it.

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

1. I agree my contributions are submitted under the BSD license.
2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

[sscini]

@djlaky @adowling2 Here are my initial thoughts on the redesign. Please provide feedback on code layout to this point.

[sscini]

Dan and I had a good conversation today about the functional things needed in the redesign, and how to go about adding components to the overall block and sub-models. Working to implement changes, closely related to what is done currently in doe's _generate_scenario_blocks.

[adowling2]

Nice progress

[sscini]

@adowling2 @djlaky Added case for bootstrap, now works with theta_est, theta_est_bootstrap, and theta_est_leaveNout if I replace _Q_opt with _Q_opt_blocks. Please review.

I am aware of the duplicated code, but when I attempted to unify them using n_scenarios = len(self.exp_list) or len(bootlist), I am getting an error for bootstrap I am still working out.

Would we want to fully transition to using theta_est for obj and theta, and cov_est for covariance and add an error/warning? Or should I make it work to calculate cov if calc_cov=True?

Thanks!

[codecov]

Codecov Report

❌ Patch coverage is 4.76190% with 100 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.31%. Comparing base (13facca) to head (d91ce3f).
⚠️ Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
pyomo/contrib/parmest/parmest.py	4.76%	100 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3789      +/-   ##
==========================================
- Coverage   89.40%   89.31%   -0.10%     
==========================================
  Files         909      909              
  Lines      105541   105646     +105     
==========================================
- Hits        94364    94360       -4     
- Misses      11177    11286     +109     

Flag	Coverage Δ
builders	29.07% <4.76%> (-0.04%)	⬇️
default	85.94% <4.76%> (?)
expensive	35.73% <4.76%> (?)
linux	86.63% <4.76%> (-2.55%)	⬇️
linux_other	86.63% <4.76%> (-0.10%)	⬇️
osx	82.78% <4.76%> (-0.09%)	⬇️
win	84.88% <4.76%> (-0.09%)	⬇️
win_other	84.88% <4.76%> (-0.09%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[sscini]

@adowling2 @djlaky Should I tag larger team on this? Please review when available. Thanks!

[sscini]

Thursday night (03/05/26): Made changes to architecture based on additional recommendations from ND Pyomo Dev team, and added a separate testing file for tests I am considering adding to review.

All feedback is appreciated. Main items of interest in review:

1. How to handle the import structure that only applies to the deprecated interface
2. Are there any changes needed in the new implementation I did not consider?
3. Now that this will be Pyomo 6.10.1, should I remove deprecation warnings for covariance structural change? (separate theta_est and cov_est)
4. Can elements from the new interface be updated into the deprecated and the old structural utility files be removed?
5. All new tests in the separate file. Once reviewed, I will move into test_parmest.py

Thanks!

[adowling2]

I think this is ready to merge after review from @blnicho or @mrmundt

[adowling2]

This makes sense to me.

[sscini]

From development meeting conversation, I moved all additional tests to test_parmest.py, and will work on removing deprecated theta_est arguments for covariance. Some tests may need to be adjusted.

Review still would be greatly appreciated, and recommendation on whether the full MPI-SPPY dependent interface can be cleanly removed here, or address in a follow-up PR.

Thank you!

[blnicho]

@sscini I'm still going through parmest.py but here are my review comments so far on the rest of the files.

In response to your specific review questions

1. How to handle the import structure that only applies to the deprecated interface

Could you leave a comment pointing to the "import structure" you're referring to. I'm not following. By deprecated interface do you mean the ParmEst interface before the Experiment class or the old interface for calculating the covariance?

2. Are there any changes needed in the new implementation I did not consider?

Not sure yet.

3. Now that this will be Pyomo 6.10.1, should I remove deprecation warnings for covariance structural change? (separate theta_est and cov_est)

Yes, I think you can remove the deprecated covariance functionality.

4. Can elements from the new interface be updated into the deprecated and the old structural utility files be removed?

Are create_ef.py and mpi_utils.py the structural utility files you're referring to? I think we should leave these files for now and let's address the rest of the mpisppy infrastructure in a separate PR. Could you be more specific with which "elements" and which deprecated interface you're referring to?

[sscini]

@blnicho Thank you very much for your comments so far!

I will add a specific comment in the code for the import clarification.

Can elements from the new interface be updated into the deprecated and the old structural utility files be removed? Are create_ef.py and mpi_utils.py the structural utility files you're referring to? I think we should leave these files for now and let's address the rest of the mpisppy infrastructure in a separate PR. Could you be more specific with which "elements" and which deprecated interface you're referring to?

Leaving create_ef.py and mpi_utils makes sense.

To clarify this: I was referring to the _DeprecatedEstimator class as the deprecated interface. I updated _Q_opt in the Estimator class to use blocks and combined it with _Q_at_theta. The _DeprecatedEstimator still uses both _Q_opt and _Q_at_theta, which rely on the _experiment_instance_creation_callback function (line 95), and have separate booleans to connect to either mpisppy, or create_ef files.

If I updated the _DeprecatedEstimator to use my _Q_opt implementation, since we previously discussed that the main point of the deprecated estimator is to preserve old user facing functions, then _experiment_instance_creation_callback, and the mentioned files could be removed. The main deprecated function to preserve now would be the use of theta_est for covariance calculations. Otherwise, I tried my best to avoid the user interface in this PR.

Hopefully this clarifies, but please let me know if more information is still needed. As you mentioned, this will most likely be done in its own PR.

[sscini]

@blnicho I removed the covariance support from theta_est and adjusted testing. Please review changes. Thank you!