Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a full OV testing GHA workflow for nightly #985

Conversation

nikita-savelyevv
Copy link
Collaborator

@nikita-savelyevv nikita-savelyevv commented Nov 4, 2024

What does this PR do?

Changes:

  1. Introduce a new GHA workflow for full OV testing on 4 possible combinations of OV and NNCF stable/nightly versions.
  2. Remove OV and NNCF stable/nightly options from slow test.

Reason for changes:
During slow testing almost none quantization tests are run which is critical for assessing NNCF version impact. That is why all OV tests need to be run on a nightly basis.

Instead of expanding the current slow test workflow to do full testing, a new workflow is added for that. This is to conserve computational resources, as different OS and transformers version is not strictly required to assess OV nightly / NNCF develop impact.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

@nikita-savelyevv nikita-savelyevv added the openvino-test Trigger OpenVINO slow tests label Nov 4, 2024
@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@nikita-savelyevv nikita-savelyevv force-pushed the separate-ov-nightly-workflow branch 2 times, most recently from 380c4fd to e79768b Compare November 4, 2024 12:44
@nikita-savelyevv nikita-savelyevv mentioned this pull request Nov 4, 2024
3 tasks
Update workflow name

Trigger Tests

Fix workflow file

Fix

Introduce more readable job names

Revert basic test changes
@nikita-savelyevv nikita-savelyevv force-pushed the separate-ov-nightly-workflow branch from e79768b to 9c05797 Compare November 6, 2024 09:23
@nikita-savelyevv nikita-savelyevv marked this pull request as ready for review November 26, 2024 18:26
Copy link
Collaborator

@eaidova eaidova left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

accuracy issues for runs with openvino nightly not related to this PR. Will be addressed in separated PR

Comment on lines 54 to 56
- name: Install lowest compatible transformers version
if: ${{ matrix.transformers-version != 'latest' }}
run: pip install transformers==${{ matrix.transformers-version }}
Copy link
Member

@IlyasMoutawwakil IlyasMoutawwakil Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

installing transformers before pip install .[tests] might result in the later overriding the intended version

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Fixed

@IlyasMoutawwakil IlyasMoutawwakil merged commit 958eb6e into huggingface:main Dec 5, 2024
20 of 28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
openvino-test Trigger OpenVINO slow tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants