Vendor eudsl in wheels/build (aie.extras)

[hunhoffe]

TL;DR

This PR vendors eudsl into the mlir-aie wheels and fixes some minor flaws in the CI.

Full Explanation

This PR is associated with the desire for a cleaner install of eudsl (Part of this issue: #2805). This should benefit those that use the wheels to install mlir-aie.

This was more difficult than expected. To include eudsl in the normal requirement.txt, I had to ensure pip was updated. This was fairly straightforward.

But what was not straightforward was:

• Instrumenting the setup.py that installed mlir-aie wheels dependencies upon install to automatically install the extras
• Instrumenting the cmake files to include the vendored files for tests from local builds e.g., ninja

The difficulties were a combination of:

• setup.py does not support parsing the install options required for eudsl
• because eudsl is installed into the same namespace (aie) installing it directly into aie.extras didn't work correctly; so I have to use pip to download it and then copy it into the correct place
• Updating pip changed the version of LLVM used in the MLIR AIE Distro workflow, which resulting in a different version of llvm. I change this workflow to use the same version of LLVM used in the clone_llvm utility, which should ensure future consistency.

[ypapadop-amd]

This would be amazing if you figure it out!

[hunhoffe]

This is a super hacky addition that should be able to be deleted in a follow-on PR. It only exists to satisfy the CI. Here's some explanation.

cloudpickle is a dependency of eudsl. Since (in this PR) we start to vendor eudsl, we need to include cloudpickle as a dependency. We do this in requirements.txt. This is not hacky and is correct.

However, the "Build and Test on Ryzen AI" job uses latest wheels to install dependencies. So new dependencies are not reflected in the CI job. So I added cloudpickle here until wheels can be built with the new dependency, at which point it should not be necessary.

I didn't try to fix the workflow dependencies because there is value in testing that dependencies are correctly reflected in the wheels, and I didn't know how to retain that property while fixing this issue.

[Copilot]

Pull request overview

This PR vendors the eudsl-python-extras package into mlir-aie wheels to simplify installation. Previously, eudsl required manual installation with special environment variables, which is now handled automatically during the wheel build process.

Changes:

• Removed manual eudsl installation steps from documentation and CI workflows
• Created vendor_eudsl.py script to download and package eudsl during wheel builds
• Updated build system to ensure consistent MLIR version across workflows

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
utils/mlir_aie_wheels/vendor_eudsl.py	New script to vendor eudsl-python-extras into wheel packages
utils/mlir_aie_wheels/setup.py	Integrated eudsl vendoring into the build process and filtered eudsl from install_requires
python/requirements.txt	Added eudsl-python-extras with config settings (previously in separate file)
python/requirements_extras.txt	Deleted - contents moved to requirements.txt
python/requirements_dev.txt	Added cloudpickle dependency required by vendored eudsl
python/CMakeLists.txt	Added build step to vendor eudsl for local builds
utils/mlir_aie_wheels/scripts/download_mlir.sh	Updated to use consistent MLIR version from clone-llvm.sh
Multiple workflow files	Removed manual eudsl installation steps
Multiple documentation files	Removed manual eudsl installation instructions
utils/setup_python_packages.sh	Removed eudsl installation from setup script
utils/quick_setup.sh	Removed eudsl installation from quick setup

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hunhoffe]

I created an alpha release to test the new installation mechanism in projects that build on mlir-aie wheels. I've use this test release in amd/IRON (amd/IRON#68), and it appears to be working well. There is just one thing I've noticed that I am not sure is a feature or a bug.

When you run pip list, you see:

eudsl-python-extras 0.1.0.20251215.1715+3c7ac1b

This is an artifact of how I hacked setup.py in the wheels to ignore the extras dependency, but pip still thinks it was installed via pip.

This seems like a feature because it accurately reflects the extras version, and it may even be possible to install a different version via normal pip commands (although I haven't tried this as it seems like an edge use case).

This seems like a bug because the eudsl-python-extras weren't actually installed via pip.

But, seems more useful than not, so I'm leaving it as-is unless I get feedback to do otherwise!

[jgmelber]

• Updating pip changed the version of LLVM used in the MLIR AIE Distro workflow, which resulting in a different version of llvm. I change this workflow to use the same version of LLVM used in the clone_llvm utility, which should ensure future consistency.

With this change are do we now use multiple versions of LLVM for MLIR-AIE?

[hunhoffe]

@jgmelber it should be all the same version, as defined in the clone-llvm.sh script