Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adds ONNX Surrogate support from OMLT #1308

Open
wants to merge 26 commits into
base: main
Choose a base branch
from

Conversation

avdudchenko
Copy link

Fixes

This adds ONNX surrogate model object and slightly refactors the current Keras surrogate model to support multiple omlt model types.

Summary/Motivation:

Currently, if one generates a NN model using PyTorch, or any other method for use in IDAES and depended project (e.g. WaterTAP) we must export the model to ONNX format, and then to keras format to use in IDAES, which makes little sense.

This PR adds native support to onnx model format provided through OMLT.

Changes proposed in this PR:

-move out part of Keras functions into omlt_surrogate_base.py to enable simple support for multiple omlt import types
-adds ONNXSurrogate object that can load onnx model directly in IDAES flowsheet block

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

  1. I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
  2. I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

@avdudchenko avdudchenko marked this pull request as ready for review December 21, 2023 06:22
@ksbeattie ksbeattie added WaterTAP Priority:Normal Normal Priority Issue or PR labels Dec 21, 2023
@avdudchenko avdudchenko changed the title Addes ONNX Surrogate support from OMLT Adds ONNX Surrogate support from OMLT Jan 4, 2024
Copy link

codecov bot commented Jan 25, 2024

Codecov Report

Attention: 18 lines in your changes are missing coverage. Please review.

Comparison is base (d45e2ad) 77.43% compared to head (7ba2ba6) 77.43%.

Files Patch % Lines
idaes/core/surrogate/onnx_surrogate.py 78.26% 7 Missing and 8 partials ⚠️
idaes/core/surrogate/omlt_base_surrogate_class.py 94.64% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1308   +/-   ##
=======================================
  Coverage   77.43%   77.43%           
=======================================
  Files         390      392    +2     
  Lines       63758    63844   +86     
  Branches    11737    11749   +12     
=======================================
+ Hits        49373    49440   +67     
- Misses      11841    11852   +11     
- Partials     2544     2552    +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ksbeattie ksbeattie requested review from rundxdi and removed request for andrewlee94, eslickj, lbianchi-lbl, ksbeattie and bpaul4 February 8, 2024 19:52
@ksbeattie
Copy link
Member

@avdudchenko can you look into the pylint and rtd test failures?

Copy link
Contributor

@rundxdi rundxdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@avdudchenko Generally this all looks good. I have a couple very minor questions and test failures need to be resolved, then I can approve it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes to this file all look good

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes to this file all look good


# TODO: remove this once new OMLT 1.2 is made available and includes tanh support
# overrides default available activation functions for ONNX, tanh is not listed in 1.1 but is supported
omltio.onnx_parser._ACTIVATION_OP_TYPES = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this need catches for anything?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure best catches to implement here, thoughts on just checking for OMLT version? (It should not be needed with future OMLT releases.

)
self.populate_block_with_net(block, formulation_object)

def evaluate_surrogate(self, inputs):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this present but only raises NotImplementError?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left it here for future implementation, but setting up ONNX model to solve outside idaes did not seem trivial at time of first commit. (e.g. there did not seem to be something as simple as _keras_model.predict.

@ksbeattie
Copy link
Member

@avdudchenko @rundxdi should this be part of the next (now May) release?

@ksbeattie
Copy link
Member

@avdudchenko @rundxdi now would be a good time to get this on the Aug release, if you can find a reviewer for it

@ksbeattie ksbeattie requested review from Robbybp and removed request for carldlaird September 5, 2024 18:18
@codecov-commenter
Copy link

codecov-commenter commented Sep 5, 2024

Codecov Report

Attention: Patch coverage is 85.71429% with 18 lines in your changes missing coverage. Please review.

Project coverage is 76.98%. Comparing base (0e05ed1) to head (1084f5c).

Files with missing lines Patch % Lines
idaes/core/surrogate/onnx_surrogate.py 77.94% 7 Missing and 8 partials ⚠️
idaes/core/surrogate/omlt_base_surrogate_class.py 94.44% 1 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1308      +/-   ##
==========================================
- Coverage   76.98%   76.98%   -0.01%     
==========================================
  Files         382      384       +2     
  Lines       61913    61995      +82     
  Branches    10130    10136       +6     
==========================================
+ Hits        47665    47728      +63     
- Misses      11848    11859      +11     
- Partials     2400     2408       +8     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@ksbeattie
Copy link
Member

@avdudchenko, this is now on the Nov 2024 release board. Can progress be picked up on this now?

@ksbeattie ksbeattie requested a review from bpaul4 October 10, 2024 18:37
@ksbeattie
Copy link
Member

ksbeattie commented Oct 10, 2024

@OOAmusat, @bpaul4, @rundxdi can you review or finish your review?

Copy link
Contributor

@rundxdi rundxdi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finalizing my review -- looks good at this point

Copy link
Contributor

@bpaul4 bpaul4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the changes implemented here are good. Not critical to fix, there are a few places in the ONNX file that refer to "keras" instead of "onnx".

idaes/core/surrogate/onnx_surrogate.py Outdated Show resolved Hide resolved
idaes/core/surrogate/onnx_surrogate.py Outdated Show resolved Hide resolved
idaes/core/surrogate/onnx_surrogate.py Outdated Show resolved Hide resolved
idaes/core/surrogate/tests/test_onnx_surrogate.py Outdated Show resolved Hide resolved
@dangunter
Copy link
Member

dangunter commented Oct 17, 2024

@avdudchenko Can you look at the test failures and see if they can be fixed?
Also could you address Brandon's comments

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority:Normal Normal Priority Issue or PR WaterTAP
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants