fix(esp_ulp): Add support for multiple ULP program embedding without name collision (IDFGH-14152)

[X-Ryl669]

Description

Currently, the ULP's auto variable name embedding script has multiple issues:

1. It names variable in the global namespace (in extern "C" mode) so multiple ULP program using the same variable name results in impossible-to-solve access and memory corruption (see linked issue)
2. It doesn't support generating arrays (so one has to trick/hack around a declared variable, rendering the whole process useless)
3. It create a useless linker script in the build process
4. It creates a lots of useless symbols declaration (like all functions and global included objects), which, if used in the HP core, crashes the HP core

This PR fixes it:

1. It changes from using nm cross compilation tool to readelf tool for extracting symbols. This allows keeping the symbol's scope (global or local), the symbol type (object or function or ...) and the symbol size (for arrays).
2. It only extract the meaningful symbols and doesn't pollute the global scope with unusable symbols like ulp_lp_core_lp_ana_peri_intr_handler
3. It allows to specify the prefix to use for generating the ULP's variable name (defaults to the current ulp_ prefix, but now, you can specify it)
4. You can use C++ prefix too (like a namespace ULPmain:: and it'll mangle the name and work without extern "C"): perfect to move the shorter variables in their own namespace and avoid collisions without dumb long names
5. It removes the generated linker script (useless to specify the symbol location, it can be done in the header file directly).
6. It add support for array uint32_t myVariable[32] is correctly exported so can be used like a native variable in HP core, no more hacky ((uint32_t*)myVariable)[0] like specified in the documentation.

From a user perspective, nothing changes (the useless symbol will be removed, but they couldn't be used anyway previously).
However, she can now replace the line:

set(ulp_app_name ulp_${COMPONENT_NAME})
set(ulp_sources "ulp/ulp_main.c")
set(ulp_exp_dep_srcs ${srcs})

ulp_embed_binary(${ulp_app_name} "${ulp_sources}" "${ulp_exp_dep_srcs}")

set(ulp_app_name2 ulp_ds_cal)
set(ulp_sources2 "ulp/ulp_ds_cal.c")

ulp_embed_binary_prefix(${ulp_app_name2} "${ulp_sources2}" "${ulp_exp_dep_srcs}")

to this:

set(ulp_app_name ulp_${COMPONENT_NAME})
set(ulp_sources "ulp/ulp_main.c")
set(ulp_exp_dep_srcs ${srcs})

ulp_embed_binary_prefix(${ulp_app_name} "ULPmain::" "${ulp_sources}" "${ulp_exp_dep_srcs}")

set(ulp_app_name2 ulp_ds_cal)
set(ulp_sources2 "ulp/ulp_ds_cal.c")

ulp_embed_binary_prefix(${ulp_app_name2} "ULPds_cal::" "${ulp_sources2}" "${ulp_exp_dep_srcs}")

And build collision-free and working multiple ULP binary.

Related

Fix #14945 and other issues too related to misusage of ULP symbols in HP core.

Testing

I've tested on my system with ESP32C6. I haven't tested with other CPU types.
However, the tools used in this PR are common cross compilation tools, it shouldn't depend on the CPU architecture.

Checklist

Before submitting a Pull Request, please ensure the following:

• 🚨 This PR does not introduce breaking changes.
• All CI checks (GH Actions) pass.
• Documentation is updated as needed.
• Tests are updated or added as necessary.
• Code is well-commented, especially in complex areas.
• Git history is clean — commits are squashed to the minimum necessary.

[github-actions]

	Messages
📖	🎉 Good Job! All checks are passing!

👋 Hello X-Ryl669, we appreciate your contribution to this project!

---

📘 Please review the project's Contributions Guide for key guidelines on code, documentation, testing, and more.

---

🖊️ Please also make sure you have read and signed the Contributor License Agreement for this project.

---

Click to see more instructions ...

This automated output is generated by the PR linter DangerJS, which checks if your Pull Request meets the project's requirements and helps you fix potential issues.

DangerJS is triggered with each push event to a Pull Request and modify the contents of this comment.

Please consider the following:
- Danger mainly focuses on the PR structure and formatting and can't understand the meaning behind your code or changes.
- Danger is not a substitute for human code reviews; it's still important to request a code review from your colleagues.
- To manually retry these Danger checks, please navigate to the Actions tab and re-run last Danger workflow.

Review and merge process you can expect ...

We do welcome contributions in the form of bug reports, feature requests and pull requests via this public GitHub repository.

This GitHub project is public mirror of our internal git repository

1. An internal issue has been created for the PR, we assign it to the relevant engineer.
2. They review the PR and either approve it or ask you for changes or clarifications.
3. Once the GitHub PR is approved, we synchronize it into our internal git repository.
4. In the internal git repository we do the final review, collect approvals from core owners and make sure all the automated tests are passing.
- At this point we may do some adjustments to the proposed change, or extend it by adding tests or documentation.
5. If the change is approved and passes the tests it is merged into the default branch.
5. On next sync from the internal git repository merged change will appear in this public GitHub repository.

Generated by 🚫 dangerJS against a71e63e

[igrr]

@X-Ryl669 Thank you for the PR. We can definitely accept the changes related to customization the prefix (ulp_). The part related to replacement of the linker script with inline assembly looks more risky for the reason mentioned above.

[X-Ryl669]

I've reworked the PR to fit the review. I've reverted the linker script removal and instead changed its format so if the declared symbols already exists, it errors out with a meaningful error. This should prevent silent building and runtime crash on colliding name for multiple ULP programs. I've also removed the ulp_embed_binary_prefix and instead added a named parameter to the function (called PREFIX). I've fixed the documentation to match the changes and describe the new prefix option.

[X-Ryl669]

@igrr Can you review the change to see if they fit your requirements, please?

[sudeep-mohanty]

Thanks for the updates @X-Ryl669. Final changes LGTM. I shall pull in the updates and run it through our CI pipeline. Thanks!

[sudeep-mohanty]

sha=a71e63e266e25309d96d89e9555f3b6739fd4de6

[X-Ryl669]

Should I resolve the new conflict with master branch, and if yes, via merging or rebasing?

[sudeep-mohanty]

Should I resolve the new conflict with master branch, and if yes, via merging or rebasing?

I can take care of it :)

[sudeep-mohanty]

Hi @X-Ryl669,
To keep you updated on this work, I had to make a few changes before I could merge the changes -

• Removed the link time symbol name collision detection since it flagged all globals being included from common files across ULP binaries. Primarily, globals defined in the ULP component library.
• Added the extraction of NOTYPE symbols to not break global variables defined in the assembly code for ULP-FSM type of the coprocessor (available in the esp32, esp32s2, esp32s3)
• Some cmake changes and general clean up to avoid any breaking changes and make the code a bit more readable.
• Added a unit test for the feature.

Thank you for the contribution.

[X-Ryl669]

Ok, thank you very much.

About: "Removed the link time symbol name collision detection since it flagged all globals being included from common files across ULP binaries. Primarily, globals defined in the ULP component library."

I think that was the point of this PR. I'm probably misunderstanding here what you've done.

I have a global log library I'm using in my ULP program. It defines an array of logs (u32 logBuffer[64]). Without symbol collision detection, when it's included and built in program 1, it'll be mapped to, let's say 0x50000100. In the second program 2, it'll be mapped to 0x50000200.

If included in the main HP core, only one address will be used (likely the first linked, since the linker will just ignore the second one). If your code is trying to read from the buffer, it'll read from the wrong address. If the data read is used to read elsewhere, (like, in my case, when I'm storing a number of argument from the buffer), it'll make your main program crash.

So, yes, moving the symbols in their own namespace is one way to prevent this, but since it isn't forced to do so, it'll still cause issues for developers that don't know about the problem.

Using the linker to flag that it has already found this symbol makes sure that it can't happen. I've tried this on the multiple ULP program I'm using and it never conflicted wrongly.

If it's causing issues, maybe it would be a good idea to warn the user that symbol collision will happen (I think it's possible in CMake to actually check a variable (the last prefix name?) and if set, warn the user about potential collisions, else set it).