Skip to content

Conversation

Iroy30
Copy link

@Iroy30 Iroy30 commented Jun 4, 2025

Fixes #3626

Summary/Motivation:

Add cuOpt math optimization (includes LP and MILP) solver backend to Pyomo so users can solve pyomo models with cuOpt

Changes proposed in this PR:

  • Add cuopt_direct solver plugin

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the BSD license.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

Copy link

codecov bot commented Jun 17, 2025

Codecov Report

❌ Patch coverage is 20.00000% with 204 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.95%. Comparing base (7deb912) to head (6cfa3e4).
⚠️ Report is 349 commits behind head on main.

Files with missing lines Patch % Lines
pyomo/solvers/plugins/solvers/cuopt_direct.py 20.00% 204 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3620      +/-   ##
==========================================
- Coverage   89.19%   88.95%   -0.24%     
==========================================
  Files         892      893       +1     
  Lines      103100   103339     +239     
==========================================
- Hits        91956    91922      -34     
- Misses      11144    11417     +273     
Flag Coverage Δ
builders 26.71% <19.21%> (-0.03%) ⬇️
default 85.41% <19.21%> (?)
expensive 34.04% <20.00%> (?)
linux 86.78% <19.21%> (-2.14%) ⬇️
linux_other 86.78% <19.21%> (-0.17%) ⬇️
osx 83.13% <19.21%> (-0.17%) ⬇️
win 85.01% <19.21%> (-0.16%) ⬇️
win_other 85.01% <19.21%> (-0.16%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Contributor

@mrmundt mrmundt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are working on making cuopt available in our testing infrastructure; can you please add tests to this PR?

@mrmundt
Copy link
Contributor

mrmundt commented Jul 1, 2025

@Iroy30 - We've been able to make cuopt available on our internal testing machines. Can you please add tests to this PR?

@Iroy30 Iroy30 force-pushed the add_cuopt_direct_solver_plugin branch from 5dbf9dd to 6f64094 Compare July 2, 2025 12:33
@Iroy30
Copy link
Author

Iroy30 commented Jul 2, 2025

@mrmundt Thanks! We have added tests by enabling testing cuopt with LP and MILP capabilities in tests/solvers.py. Let us know if:

  1. There is any other testing that needs to be enabled.
  2. For testing cuOpt backend in CI, it needs a GPU, is that something pyomo team intends to add or will it be disabled in github CI?

The following is the testing output I get relevant to cuOpt

Testing scenario generation
------------------------------
LP_block, cuopt, python
   ok: 
LP_block_kernel, cuopt, python
   ok: 
LP_compiled, cuopt, python
   ok: 
LP_compiled_dense_kernel, cuopt, python
   ok: 
LP_compiled_sparse_kernel, cuopt, python
   ok: 
LP_constant_objective1, cuopt, python
   ok: 
LP_constant_objective1_kernel, cuopt, python
   ok: 
LP_constant_objective2, cuopt, python
   ok: 
LP_constant_objective2_kernel, cuopt, python
   ok: 
LP_duals_maximize, cuopt, python
   ok: 
LP_duals_maximize_kernel, cuopt, python
   ok: 
LP_duals_minimize, cuopt, python
   ok: 
LP_duals_minimize_kernel, cuopt, python
   ok: 
LP_inactive_index, cuopt, python
   ok: 
LP_inactive_index_kernel, cuopt, python
   ok: 
LP_infeasible1, cuopt, python
   ok: 
LP_infeasible1_kernel, cuopt, python
   ok: 
LP_infeasible2, cuopt, python
   ok: 
LP_infeasible2_kernel, cuopt, python
   ok: 
LP_piecewise, cuopt, python
   ok: 
LP_piecewise_nosuffixes, cuopt, python
   ok: 
LP_simple, cuopt, python
   ok: 
LP_simple_kernel, cuopt, python
   ok: 
LP_trivial_constraints, cuopt, python
   ok: 
LP_trivial_constraints_kernel, cuopt, python
   ok: 
LP_unbounded, cuopt, python
   ok: 
LP_unbounded_kernel, cuopt, python
   ok: 
LP_unused_vars, cuopt, python
   ok: 
LP_unused_vars_kernel, cuopt, python
   ok: 
MILP_discrete_var_bounds, cuopt, python
   ok: 
MILP_discrete_var_bounds_kernel, cuopt, python
   ok: 
MILP_infeasible1, cuopt, python
   ok: 
MILP_infeasible1_kernel, cuopt, python
   ok: 
MILP_simple, cuopt, python
   ok: 
MILP_simple_kernel, cuopt, python
   ok: 
MILP_unbounded, cuopt, python
   ok: 
MILP_unbounded_kernel, cuopt, python
   ok: 
MILP_unused_vars, cuopt, python
   ok: 
MILP_unused_vars_kernel, cuopt, python
   ok: 

Comment on lines +60 to +62
t0 = time.time()
self.solution = cuopt.linear_programming.solver.Solve(self._solver_model)
t1 = time.time()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fine, but just so you're aware, we have this lovely little utility called TicTocTimer that you may want to consider using: https://pyomo.readthedocs.io/en/latest/api/pyomo.common.timing.TicTocTimer.html

@whart222
Copy link
Member

whart222 commented Jul 8, 2025

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

@mrmundt
Copy link
Contributor

mrmundt commented Jul 8, 2025

@mrmundt What do you think about including this solver interface in pyomo.contrib.solvers? Would it make sense to pull-in new solver interfaces there, since that's where the new solver API is evolving?

@whart222 - I am evenly split. Because we are still messing with what the new solver interfaces are going to actually do / how they will handle input and present output, I don't know if we want to put "new" solvers there or just "well-established" ones that we can robustly test / really know what they are supposed to do and return.

@mrmundt
Copy link
Contributor

mrmundt commented Jul 8, 2025

@Iroy30 - I forgot to post this last week, but all of the failures are of the variety:

self = <pyomo.solvers.tests.checks.test_no_solution_behavior.Test_LP_unbounded_kernel testMethod=test_cuopt_python>

    def return_test(self):
>       return failed_solve_test(self)

/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:92: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/tester/python/python39/pyomo/pyomo/solvers/tests/checks/test_no_solution_behavior.py:61: in failed_solve_test
    opt, results = model_class.solve(
/tester/python/python39/pyomo/pyomo/solvers/tests/models/base.py:123: in solve
    results = opt.solve(
/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:99: in solve
    self._solver_error('solve')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <pyomo.opt.base.solvers.UnknownSolver object at 0x7f86ad9ab040>
method_name = 'solve'

        def _solver_error(self, method_name):
>           raise RuntimeError(
                """Attempting to use an unavailable solver.
    
    The SolverFactory was unable to create the solver "%s"
    and returned an UnknownSolver object.  This error is raised at the point
    where the UnknownSolver object was used as if it were valid (by calling
    method "%s").
    
    The original solver was created with the following parameters:
    \t"""
                % (self.type, method_name)
                + "\n\t".join("%s: %s" % i for i in sorted(self._kwds.items()))
                + "\n\t_args: %s" % (self._args,)
                + "\n\toptions: %s" % (self.options,)
            )
E           RuntimeError: Attempting to use an unavailable solver.
E           
E           The SolverFactory was unable to create the solver "cuopt"
E           and returned an UnknownSolver object.  This error is raised at the point
E           where the UnknownSolver object was used as if it were valid (by calling
E           method "solve").
E           
E           The original solver was created with the following parameters:
E           	executable: cuopt
E           	solver_io: python
E           	type: cuopt
E           	_args: ()
E           	options: {}

/tester/python/python39/pyomo/pyomo/opt/base/solvers.py:116: RuntimeError

@mrmundt
Copy link
Contributor

mrmundt commented Jul 8, 2025

@Iroy30 - Two more things:

  1. Please run black -S -C on your files
  2. Something in our testing environment isn't loading correctly. I'm looking into it.

Iroy30 and others added 2 commits July 9, 2025 13:15
@Iroy30
Copy link
Author

Iroy30 commented Aug 12, 2025

There are active improvements and fixes in cuOpt that are being worked on for the next release. I have skipped two above mentioned tests temporarily. The CI tests are passing for me locally. Hopefully all runs successfully in the pipeline as well.

@Iroy30
Copy link
Author

Iroy30 commented Aug 12, 2025

@mrmundt How do I trigger the CI ?

@mrmundt
Copy link
Contributor

mrmundt commented Aug 12, 2025

@mrmundt How do I trigger the CI ?

I needed to click "Approve" on it, which I have now done!

@Iroy30
Copy link
Author

Iroy30 commented Aug 12, 2025

Looks like it is failing as cuOpt is not available in CI? I can reproduce the CI error when cuOpt package is not available in the environment.

@jsiirola
Copy link
Member

cuOpt is not available on GHA because the GHA runners (to our knowledge) do not have GPUs. The Jenkins job will run on machines with GPUs and cuOpt, and should exercise this interface. Regardless, solver interfaces must be able to be imported and instantiated (to check for availability) without a hard dependency on any "external" packages.

A quick look at you implementation:

  • you will need to guard the numpy import in cuopt_direct.py (the easy way it to import it from pyomo.common.sdependencies
  • you should defer resolving the _version and checking if the python api exists until after (outside) the CUOPTDirect.__init__().

@Iroy30
Copy link
Author

Iroy30 commented Aug 13, 2025

@jsiirola Thanks John, aded the guards and updated version, hopefully this should fix the fails when cuopt isn't present. Should we re-run CI?

@Iroy30
Copy link
Author

Iroy30 commented Aug 26, 2025

@mrmundt @jsiirola How can I go ahead and re-run the CI?

@Iroy30
Copy link
Author

Iroy30 commented Sep 4, 2025

@mrmundt I don't think the fail we see is cuOpt related. Can you confirm?

@blnicho blnicho requested a review from michaelbynum September 9, 2025 18:55
@Iroy30
Copy link
Author

Iroy30 commented Sep 18, 2025

We are gearing up for launch of cuopt 25.10, it would be great if we could review this PR before the new release so that pyomo & cuopt are in sync

@michaelbynum
Copy link
Contributor

This is very close to the top of my list. When is the cuopt release?

@Iroy30
Copy link
Author

Iroy30 commented Sep 18, 2025

Thanks @michaelbynum ! It is October 10th

Copy link
Contributor

@mrmundt mrmundt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, @Iroy30 , for this PR! I have done an initial review. Some high-level things to consider:

  1. Logging - logger is created but then never used. Consider places it may be helpful
  2. "Magic" numbers - there are quite a few instances of numbers that aren't defined being used. We love documentation!

extract_reduced_costs = False
for suffix in self._suffixes:
flag = False
if re.match(suffix, "dual"):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could very well be wrong, but isn't this backwards? I thought it worked like:

# re.match(pattern, string), e.g.,
re.match("Hello", some_string)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is fragile, and is being reworked in the contrib.solver effort. It really doesn't need to use re at all and can just use ==.

That said, this logic was copied from other direct solvers (I saw it in CPLEX.py and GUROBI.py). It would be good to rework this, but given that it is consistent with the (questionable) behavior in other solvers, I will not complain if we leave it as is.


conname = self._symbol_map.getSymbol(con, self._labeler)
self._pyomo_con_to_solver_con_map[con] = con_idx
con_idx += 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to actually be incrementing.

Comment on lines +169 to +175
except Exception:
e = sys.exc_info()[1]
msg = (
"Unable to create CUOPT model. "
"Have you installed the Python "
"SDK for CUOPT?\n\n\t" + "Error message: {0}".format(e)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is never raised or logged, so a user will never see this message.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worse, the user will get a strange exception later ... probably from _add_var.

Comment on lines +84 to +85
if not con.active:
return None
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there are multiple constraints, one that is active and all others that aren't, and this hits an inactive one first, wouldn't that cause the loop to immediately exit / it wouldn't get to the active constraint? I think you probably want continue instead of return None.

Comment on lines +213 to +217
if not flag:
raise RuntimeError(
"***The cuopt_direct solver plugin cannot extract solution suffix="
+ suffix
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It might be better to log loudly than to raise an error here (no strong feelings either way, just wanted to offer an alternative option).

Comment on lines +228 to +256
if status in [1]:
self.results.solver.status = SolverStatus.ok
self.results.solver.termination_condition = TerminationCondition.optimal
soln.status = SolutionStatus.optimal
elif status in [3]:
self.results.solver.status = SolverStatus.warning
self.results.solver.termination_condition = TerminationCondition.unbounded
soln.status = SolutionStatus.unbounded
elif status in [8]:
self.results.solver.status = SolverStatus.ok
self.results.solver.termination_condition = TerminationCondition.feasible
soln.status = SolutionStatus.feasible
elif status in [2]:
self.results.solver.status = SolverStatus.warning
self.results.solver.termination_condition = TerminationCondition.infeasible
soln.status = SolutionStatus.infeasible
elif status in [4]:
self.results.solver.status = SolverStatus.aborted
self.results.solver.termination_condition = (
TerminationCondition.maxIterations
)
soln.status = SolutionStatus.stoppedByLimit
elif status in [5]:
self.results.solver.status = SolverStatus.aborted
self.results.solver.termination_condition = (
TerminationCondition.maxTimeLimit
)
soln.status = SolutionStatus.stoppedByLimit
elif status in [7]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a lot of "magic numbers" here - what is 1? What is 3? What is 7? Can you give them names or at least add comments/explanation?

from pyomo.opt.results.solution import Solution, SolutionStatus
from pyomo.opt.results.solver import TerminationCondition, SolverStatus
from pyomo.opt.base import SolverFactory
from pyomo.core.base.suffix import Suffix
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You import Suffix twice - once here and once on line 19

except ImportError:
self._python_api_exists = False
except Exception as e:
print("Import of cuopt failed - cuopt message=" + str(e) + "\n")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We generally log/raise rather than print. Not a rule, but generally a suggestion.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will argue that it is a rule: as Pyomo is a library and not an application (with the exception of the pyomoscript), we should 100% avoid print() (outside the pyomo script)

from pyomo.core.base.suffix import Suffix
import time

logger = logging.getLogger("pyomo.solvers")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use the logger more liberally!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we are really trying to move to a paradigm where we log using

logger = logging.getLogger(__name__)

Comment on lines +29 to +34
try:
import cuopt

cuopt_available = True
except:
cuopt_available = False
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can use the attempt_import logic here instead of this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed: I would just import cuopt and cuopt_available from pyomo.solvers.plugins.solvers.cuopt_direct

Copy link
Member

@jsiirola jsiirola left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is actually pretty close, but there are a number of things that need to be resolved before we can merge.

import logging
import re
import sys

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please consider alphabetizing the imports

from pyomo.solvers.plugins.solvers.direct_or_persistent_solver import (
DirectOrPersistentSolver,
)
from pyomo.core.kernel.objective import minimize, maximize
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not import from kernel.

Suggested change
from pyomo.core.kernel.objective import minimize, maximize
from pyomo.common.enums import minimize, maximize

from pyomo.core.base.suffix import Suffix
import time

logger = logging.getLogger("pyomo.solvers")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, we are really trying to move to a paradigm where we log using

logger = logging.getLogger(__name__)

except ImportError:
self._python_api_exists = False
except Exception as e:
print("Import of cuopt failed - cuopt message=" + str(e) + "\n")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will argue that it is a rule: as Pyomo is a library and not an application (with the exception of the pyomoscript), we should 100% avoid print() (outside the pyomo script)

Comment on lines +50 to +58
try:
import cuopt

self._version = tuple(int(k) for k in cuopt.__version__.split('.'))
self._python_api_exists = True
except ImportError:
self._python_api_exists = False
except Exception as e:
print("Import of cuopt failed - cuopt message=" + str(e) + "\n")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This entire block of code is unnecessary and can run afoul of the cuopt imported at the module scope using attempt_import. I think the entire thing can be replaced with

self._version = cuopt.__version__.split('.')
self._python_api_exists = cuopt_available

(Note that cuopt.__version__ will trigger the import and cause cuopt_available to be resolved to a bool)

Comment on lines +189 to +199
for sub_block in block.block_data_objects(descend_into=True, active=True):
obj_counter = 0
for obj in sub_block.component_data_objects(
ctype=Objective, descend_into=False, active=True
):
obj_counter += 1
if obj_counter > 1:
raise ValueError(
"Solver interface does not support multiple objectives."
)
self._set_objective(obj)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This could be made simpler / more efficient:

Suggested change
for sub_block in block.block_data_objects(descend_into=True, active=True):
obj_counter = 0
for obj in sub_block.component_data_objects(
ctype=Objective, descend_into=False, active=True
):
obj_counter += 1
if obj_counter > 1:
raise ValueError(
"Solver interface does not support multiple objectives."
)
self._set_objective(obj)
objectives = list(
block.component_data_objects(Objective, descend_into=True, active=True)
)
if len(objectives) > 1:
raise ValueError("Solver interface does not support multiple objectives.")
elif objectives:
self._set_objective(objectives[0])

self._solver_model.set_constraint_lower_bounds(np.array(c_lb))
self._solver_model.set_constraint_upper_bounds(np.array(c_ub))

def _add_var(self, variables):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to _add_constraint above, I would recommend renaming:

Suggested change
def _add_var(self, variables):
def _add_vars(self, variables):

extract_reduced_costs = False
for suffix in self._suffixes:
flag = False
if re.match(suffix, "dual"):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is fragile, and is being reworked in the contrib.solver effort. It really doesn't need to use re at all and can just use ==.

That said, this logic was copied from other direct solvers (I saw it in CPLEX.py and GUROBI.py). It would be good to rework this, but given that it is consistent with the (questionable) behavior in other solvers, I will not complain if we leave it as is.


prob_type = solution.problem_category

if status in [1]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should not create lists / use in:

Suggested change
if status in [1]:
if status == 1:

(here and all the subsequent tests)

Comment on lines +273 to +274
self.results.problem.upper_bound = solution.get_primal_objective()
self.results.problem.lower_bound = solution.get_primal_objective()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This implies convergence to a MIP gap of 0... I think this needs to be something like (assuming get_dual_objective() is a real thing):

Suggested change
self.results.problem.upper_bound = solution.get_primal_objective()
self.results.problem.lower_bound = solution.get_primal_objective()
if self._solver_model.maximize:
self.results.problem.upper_bound = solution.get_dual_objective()
self.results.problem.lower_bound = solution.get_primal_objective()
else:
self.results.problem.upper_bound = solution.get_primal_objective()
self.results.problem.lower_bound = solution.get_dual_objective()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support NVIDIA cuOpt solver
6 participants