bindgen: use bindgen to provide Rust bindings to C - v4 #12450
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a minimal integration of bindgen to the build. Bindgen is integrated at compile time with a "build.rs" file that for now only generates AppLayerEventType Rust bindings. This required some refactoring of the C so app-layer-events.h did not also include "rust.h", which causes issues for bindgen, probably related to circular references. AppLayerEventType was chosen as the first step as its an argument type some app-layer functions that we may want to use bindgen to export Rust, and one of the requirements of bindgen might be that C functions should only use datatypes defined in C, and not Rust. Following such a rule also prevents circular dependencies between Rust and C code. Bindgen generates the bindings in a file named "bindings.rs" in the target/ direction, which "sys.rs" will statically "include" at compiling name, making these bindings available under package "crate::sys". "build.rs" had to be placed in the non-standard location of "src/build.rs" (its usually alongside Cargo.toml) to satisfy the out-of-tree build requirements of distcheck. Note that bindgen is also available as a command line tool which could be used instead of integrating this at compile time, however the tool requires a newer version of Rust than our MSRV, and may present additional issues with respect to autoconf and distcheck. Ticket: OISF#7341
Follow the naming scheme for public exports.
This lets us remove decode.h from app-layer-events.h as pulling in app-layer-events.h shouldn't result in pulling in dpdk, and other includes not related to app-layer-events. decode.h also doesn't need those forward declarations anymore due to previous changes.
Instead of defining this function pointer in type in Rust, and having it in C signatures, create a type and export it to Rust. To facilitate this, and new header has been creates, "app-layer-types.h", this is to avoid the circular reference of C headers pulling in "rust.h" which are required to generate Rust bindings.
This exposes the C define ALPROTO values to Rust without having to perform some runtime initialization with init_ffi. As app-layer-protos.h was clean of a circular reference to rust.h we could use it directly, it just needed the addition of suricata-common.h.
Replace bindgen usage from build.rs with the bindgen-cli program much like we use cbindgen. As bindgen can only accept one header file, we construct a pseudo-header file of all the headers that need to be consumed. Makefile dependency checking is done to make sure the pseudo-header is only generated as needed to avoid rebuilding every time. Special handling is required for Windows to use the Windows path.
This simplifies building, as we don't have to worry about path and such under autoconf/automake. It does however mean when we update C headers that are exposed to Rust, we manually have to re-generate the bindings, but this is a check we can do in CI. It also eliminates the need for everyone who wants to build Suricata to have bindgen and clang tools installed.
This required us to remove the auto-generated by header as it can change depending on what version is used. I suppose different versions could generate slightly different output that would cause this error out. In which case we may want to use a custom script that can be a bit smarter output, or simply error out if any of the headers used in the bindings is newer than the generated _sys.rs file.
If we have bindgen, and we're building in-tree, rebuild the bindings and compare to the previous bindings and print a warning if there is a difference. Only a warning for now, as I'm not sure where this might fail, and I don't want it to become a nuisance.
This was referenced Jan 22, 2025
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12450 +/- ##
=======================================
Coverage 80.63% 80.63%
=======================================
Files 920 921 +1
Lines 258704 258704
=======================================
+ Hits 208595 208606 +11
+ Misses 50109 50098 -11
Flags with carried forward coverage won't be shown. Click here to find out more. |
|
CI issue fixed here: #12451 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previous work:
bindgen: use bindgen to provide Rust bindings to C - v3 #12062: This first PR used the bindgen library as a build-time plugin. This requires that clang/clang-devel be available to all builders of Suricata. Additionally there was some path issues on Windows where Rust a cygpath have very different ideas of what a path should look like and I was unable to get it to work on Windows. However, its the most "seamless" way.
bindgen-cli: alternate use of bindgen - v1 #12139: This converted to bindgen-cli to generate the bindings. This lets us control it from Makefiles. There are issues like dependency tracking to automatically rebuild the bindings and such. And it also required clang/clang-devel type packages to be install.
Changes from previous work:
bindgenis installedbindgenand related tools installed.This is just the very beginnings of using
bindgeneffectively, but once foundation work like this is commited, it becomes more about header cleanup, ideally to the point where we have no manually maintainedextern "C"code in our Rust.Ticket: https://redmine.openinfosecfoundation.org/issues/7341