Migration path for projects using Managed Build System

[ruspl-afed]

For many years, Eclipse CDT Managed Build System has been the main approach to integrating custom tool chains and targets, and there is a lot of code strictly based on it, including Eclipse Embedded CDT

With raising of Core Build System it is not clear how to deal with these implementations as well as with user projects produced by them.

What is recommended migration path from Managed Build System to Core Build System?

[jantje]

FYI AutoBuild (a new implementation of managed build) has been ready for some time.

[ruspl-afed]

Yes, I understand. I'm working in the same area, but with a bit different drivers.
My goal is to come up with a model (EMF, of cause) that will allow to easily support new forks of (GCC/LLVM) toolchains (with new option values) done by hardware vendors. For now (from about 5th attempt) I have more or less good model that I can contribute back to MBS through dynamic extension and then generate valid makefile. With pretty and tunable EMF Forms UI for it.

[jantje]

I don't know what EMF is nor what you mean with dynamic extension.
Currently MBS gets the tools and there parameters from the plugin.xml based on the buildDefinitions.exsd and copies this information over to .cproject.
Autobuild gets the tools and there parameters from the plugin.xml based on a structural very similar buildDefinitions.exsd and keeps the ID of the projectType.
I do not know how your dynamic extension fits in this story but I would like to know.

[ruspl-afed]

I don't know what EMF is

https://eclipse.dev/modeling/emf/
So, I have a model of GCC and other things and can manage them as I like with good API, GUI, validations and all the luxury around EMF, instead of editing huge plugin.xml contributions manually and spending hours trying to figure out where exactly in these thousands of lines I did a misprint.

Currently MBS gets the tools and there parameters from the plugin.xml based on the buildDefinitions.exsd and copies this information over to .cproject.

Correct. And buildDefinitions.exsd also supports dynamicElementProvider. Well, these providers are not really "dynamic", but at least they allow to programmatically assembly buildDefinitions data from another source.
Moreover, MBS was implemented so sloppy that one can modify a lot of things stored in ManagedBuildManager, but I'm not using these lacks of design.

[jantje]

instead of editing huge plugin.xml contributions manually and spending hours trying to figure out where exactly in these thousands of lines I did a misprint.

I agree and disagree here.
I agree in the sense that the huge plugin.xml is a real pain in the ass. (pardon my french).
I disagree because I think the fact that the plugin.xml is so huge is a "management problem" and less a "result of the buildDefinitions.exsd".

IMHO The management problem is mostly caused by the fact CDT is open source and everyone contributes and nobody can really take responsibility of the whole thing.
Therefore people copy and past and copy paste pastes of a copy even if there is technically no need to copy there still remains the ownership problem. Given that CDT MBS has been pretty stable for the last years I think the owner problem has reduced in importance and having one person taking responsability over the whole thing can be a reality.
Next to that there are 2 limitations in buildDefinitions.exsd that contribute to the huge plugin.xml

1. The fact that the toolchain (as defined in buildDefinitions.exsd ) is at the level of the configuration. I have not found one single project type that had differences between the toolchains apart from default options which can easily be handled in other ways.
   As nearly all project types have 2 configurations (debug and release) this means a doubling of toolchains. (most of them use superclass so they are basically empty )
2. I seems like it was decided to have a project type per toolchain (here used as gcc;gnu;mingw ) Resulting in basically tripling the the size the plugin.xml (not in number of bytes due to the superclass but in the hours spend "trying to figure out where exactly in these thousands of lines I did a misprint/misread/miswrite")

For autoBuild I added toolprovider to buildDefinitions.exsd and moved the toolchain from the configuration level to the project type level.
I also removed plenty of the "make related fields".
To make sure I would get this to work I imported all the build definitions from the plugin.xml's I could find and cleaned them up.
I never got the clang and the MVS to work (not even in plain CDT) so I ended up dumping these.

I ended up with 4 project types 4 toolchains (1 for each project type) 4 tool providers and 2 builders.

Using enablement default options can be turned on

Options can also be turned on or off based on the tool provider

It is pretty crazy what you can do with these enablements. Even disabling tools (like the MVC resource compiler) is possible.

Because I do not have the knowledge of which options are available for which toolchain (and sloeber doesn't use them) I only added some differences for testing purposes.
As you can see from the screenshots I kept the ID's the same not to make migration impossible.

[jantje]

Correct. And buildDefinitions.exsd also supports dynamicElementProvider.

I have no clue how this works so I assume this will not work with autobuild.

Well, these providers are not really "dynamic", but at least they allow to programmatically assembly buildDefinitions data from another source.

Because MBS copies the plugin.xml to the .cproject and adds "project configuration information" to the xml (like compile option X for file Y) buildDefinitions.exsd contains way more fields then are needed to describe a build process.
Also therefore MBS has writing capabilities in plugin.xml of your java project (which I found out because the Sloeber plugin.xml kept changing while I was debugging Sloeber)
Also therefore all the encapsulating classes of the objects in buildDefinitions.exsd are read and write which confused me a lot and makes things really complex.
AutoBuild is purely plugin.xml read. It stores ID's of the needed plugin.xml content and gets the actual values from the plugin.xml.
So lets assume we have 2 toolchains (gcc and mingw or gcc on windows and linux) which have the same option (encoded with the same ID in plugin.xml) but one should contribute X and the other to x.
As Autobuild always reads from the plugin.xml based on the ID changing the toolchain in the GUI will change the commandline contribution for this option from X to x.
The exact same thing happens with changing the tooltype for example from static lib to exe.

What I'm trying to say is that for AutoBuild the source is the plugin.xml. I have no clue whether you can "dynamically" do something there.
Autobuild accesses the plugin.xml through interfaces which you can find here
https://github.com/Sloeber/arduino-eclipse-plugin/tree/master/io.sloeber.autoBuild/src/io/sloeber/autoBuild/schema/api
Which have been implemented here
https://github.com/Sloeber/arduino-eclipse-plugin/tree/master/io.sloeber.autoBuild/src/io/sloeber/autoBuild/schema/internal

[jonahgraham]

I think we need a "new" plan of record for this, because as of now the current plan of record is:

CDT "deprecated" MBS many many years ago - see this release review from 2017 as an example https://projects.eclipse.org/projects/tools.cdt/releases/9.3.0/review - quoted here:

We have a few systems that are duplicated legacy and new. We have the new Core Build system that should be a long term replacement for the Managed Build system. Until we can do everything with the new one, we have both systems remaining.

And the follow on release review quoted here https://projects.eclipse.org/projects/tools.cdt/releases/9.4.0/review:

As Core Build matures, and with Managed Build largely unmaintained, we have started planning for Managed Build's deprecation. Integrators adding in new build integrations should look at Core Build.

[jonahgraham]

I think part of the new plan should be to remove the deprecated tag that is semi-attached to MBS and define it as legacy, but with a long shelf-life. i.e. we don't plan to remove it, probably not ever. But I don't think we should encourage new projects to start with MBS, but not because of the technical drawbacks of MBS, but rather because of its proprietary nature. (As I mentioned in #960 (review) I am unsure that proprietary is the correct term)

[jld01]

I think we are still at the "planning for deprecation" stage and may be there for quite some time. We might reasonably describe Managed Build as a legacy build system, but there are still many users, multiple downstream products and some ongoing contributions in the area of Managed Build.

[jonahgraham]

legacy build system

I think this is the best description.

[ewaterlander]

Hi @ruspl-afed ,
Ideas for Core Build started in 2010. Implementation started in 2015. See also #758 (comment)

[ruspl-afed]

thank you for the references @ewaterlander
I have explained my position here

[jantje]

I think we need a "new" plan of record for this,

I think CDT should take a look at AutoBuild before making new plans.

These are the 2 plugins you need to run it
https://github.com/Sloeber/arduino-eclipse-plugin/tree/master/io.sloeber.autoBuild
https://github.com/Sloeber/arduino-eclipse-plugin/tree/master/io.sloeber.autoBuild.ui
This is the plugin I use to run tests
https://github.com/Sloeber/arduino-eclipse-plugin/tree/master/io.sloeber.autoBuild.test

What is better for the user?

1. Consistent results between internal build and make build (I made internal build default)
2. Support for compound projects ( multiple libraries and one exe in one project; could be extended easily to multiple libraries and multiple exes in one project)
3. Support for changing projectType of created projects (static lib; dynamic lib; exe;compound;sloeber ..)
4. Support for changing buildTools of created projects (mingw; gcc flavours;)
5. AutoBuild allows easier migration of projectTypes
6. Better support for version control
7. Support for space in filenames
8. When your project no longer has configurations based on a configuration in the plugin.xml you can change a configuration to be based on a configuration in the plugin.xml

What is better for the developer?

1. Cleaner code (lots of stuff has been taken out of managed Build but the scars still show)
2. easier extensibility like Eclipse Embedded CDT and Sloeber. (FI autobuild loads the extension meaning there is no need for lifecycle management in the extension)
3. Cleanup of the buildDefinitions.exsd
4. Replaced the custom enablement to custyomize plugin.xml reading by the eclipse enablement
5. The project info and projectType info (from the plugin.xml) is read and put in a data structure (providing sequence and FQN commands to execute) that can be used to by any builder. This allows for easy integration of new builders (like cmake) or the generation of the json needed for the new editor.
6. The data structure creation/reading mentioned in the previous points has extensions to allow for customization (like inserting pre/post build, compile or link commands) by the extension provider.

What is less good?

1. removal of creation of project types in the gui (as discussed on the mail list)
2. I rearanged the projecttypes and toolchains so the options right now may not map to the toolchain (I kept all id's equal to allow for easier migration) some effort to get all (and only the) supported options in the gui is needed. (The biggest problem is having the knowledge of the compiler options and the enablement system in one person)
3. It is seriously modified and new code; there will be bugs.
4. I do not have graphical skills so graphics are missing or are reuse
5. Most usage has been through the extension sloeber so the gui for project creation is rough around the edges and not 100% working (you can select configuration but all configs are created anyways)
6. I think many java programmers will look down on how I implemented point 2 in "What is better for the developer?"

[ruspl-afed]

Let me try to explain. MBS is a kind of CDT "vendor lock", i.e. one cannot get such an IDE project and go build it without CDT, using VSCode or another IDE, or just a plain make.
This is because MBS relies on variables and other things,
that can be determined only in a context of MBS IConfiguration
that needs a lot of data to be constructed correctly
and this data is distributed across many manifests of various origins.

The question is: does AutoBuild support building outside of CDT?
From what I can read, the answer is rather no than yes

Regarding build system that I'm working on, the answer is rather yes than no, because one of my goals is to share project manifests with Theia-based IDEs: at least my manifests persisted with "standard" EMF JSON.
But, to be honest, I cannot answer "yes, sure" today, because I cannot proof such an answer. Hopefully, next year.

[jantje]

Thanks for the explanation.
In my book MBS is CDT "vendor lock" and Autobuild is not intended to be different.
However as autobuild always uses FQN the generated makefiles (if you select to build using external make) can be run outside of CDT.
If there is a interest and a "standard format" to remove "the vendor lock" I'm quite willing to see if it is possible to generate that format.
I assume compile_commands.json is such a "standard format"; I plan to implement the creation/update of that file at build time to support the new editor. (This should be really easy)

project manifests with Theia-based IDEs: at least my manifests persisted with "standard" EMF JSON.

I have no clue what you are talking about. That is why working together is a good thing. Each with their own knowledge skills the result becomes way bigger than the sum of parts.

[jonahgraham]

The question is: does AutoBuild support building outside of CDT?
From what I can read, the answer is rather no than yes

My read talking to people is that is a deal breaker for adopting the technology. People are only taking on a build system that works in their favourite IDE (be it CDT, VSCode, CLion, etc) and the command line. CMake, meson, Makefiles all fit that bill, and a new tool can fit that bill too.

[jantje]

I have read this comment multiple times but I do not understand what you are trying to say.
It sounds like on the one hand you say people use the build system of their favorite IDE while on the other hand saying AutoBuild not supporting build systems of other IDE's is a dealbreaker.
On top of that I do not see AutoBuild as a new adoption of technology. I see AutoBuild as a way to get out of dead end CDT is in regards to MBS. I therefore went into great lengths to make AutoBuild feel and behave like MBS.
You really should take a look at it.
The easiest way I think is to install sloeber product; switch to CDT perspective and create a autobuild project.
https://github.com/Sloeber/arduino-eclipse-plugin/actions/runs/12300446683

[jonahgraham]

I have read this comment multiple times but I do not understand what you are trying to say.

Sorry for being verbose before. I don't think there is any interest in CDT adopting AutoBuild. Based on the feedback when you asked two years ago, there was little to no interest in an updated MBS that is incompatible with MBS

I have no doubt that AutoBuild is great, feels like MBS, etc. But existing projects are not redoing their MBS build to another IDE proprietary builder. They want CMake and other IDE independent builders. That is why our focus at CDT should be making sure that the common builders work well with CDT and try to ensure we don't break the wide existing install base that uses MBS like it is today - even with all its quirks.

[jonahgraham]

There is another key part to the discussion - a key purpose of Core Build is to connect the external build system (CMake, Makefile, Meson, etc) to CDT's parsers and indexers.

With CDT LSP, this mostly goes away as the Java parts of CDT (parser, indexer, etc) don't need to know as much about a project's structure anymore.

[jantje]

FYI
Sloeber uses the indexer results as part of the build process (conversion of .ino to .cpp based on method declarations and includes in the .ino file) and project update (inclusion of missing Arduino libraries based on missing includes triggered by indexer events).
If CDT LSP does not provide a querying API Sloeber will have to stick to the indexer. Even if CDT LSP provides this info some migration will be needed.
I do not know if other integrators query the indexer results.

As for Autobuild. The only integration that can be dropped is the request for discovery commands ( at least what I can think of right now).

[ruspl-afed]

a key purpose of Core Build is to connect the external build system (CMake, Makefile, Meson, etc) to CDT's parsers and indexers.

Interesting. Do you mean by analyzing build system logs to extract actual build command for every compilation unit @jonahgraham ?

[jonahgraham]

@ruspl-afed yes that is what I mean, although it is core build toolchain dependent. the cmake one uses compile_commands.json, the makefile one examines build logs, not sure what the meson one does. They also use discovery (running the compiler) to get built-in defines and includes once they have identified what the compiler command is.

[jantje]

As far as I understood cmake is build on top of MBS and feeds MBS the data it needs.
I would assume that means the discovery is run by the MBS language provider that feeds the data into the CDT framework.
CDT/LSP will need a similar discovery if you do not want user experience degradation (like color coding #ifdefed code based on macro's supplied by the compiler during discovery ).
As to examining build logs: The error parsers do that. Are you sure cmake has build log parsers that are not registered as error parsers?

[jonahgraham]

cmake integration does not use build logs from user's build, it uses discovery to launch gcc (or equiv) to find out built in values of the compiler.

[ruspl-afed]

@jonahgraham @jld01 I understood the reasoning behind why Core Build made MBS obsolete (legacy).
However my initial question still stands:
What is the recommended migration path from Managed Build System to Core Build System for existing solutions like Eclipse Embedded CDT ?

[jonahgraham]

migration path

If you mean something that helps users or ISVs migrate from MBS to CMake, I don't think there is more than that statement. FTR I don't think embed-cdt can migrate off MBS, they can only support new in addition to MBS. I don't see any future where we have a tool to migrate from MBS to anything else.

[jld01]

@ruspl-afed, I think the first step towards adoption of Core Build within Embedded CDT would be to analyse the current Core Build feature set for embedded use, identify any missing capabilities and discuss with the CDT project where it would be most appropriate to implement those missing capabilities.

There is no easy way to migrate an existing Managed Build project to Core Build at present. Ideally CDT could provide a migration wizard that generates a CMake description of all the source files to be built and their tool settings.

[ruspl-afed]

I think this is the best answer so far, thank you @jld01

To give a little more context for the initial question.
I'm actively working on a new IDE product focused on RISC-V embedded development. Although it was started from what Embedded CDT already does (many thanks to this project and @ilg-ul, who has done a great job for so many people in this world) now I'm far from its codebase and need it only to support these users who already created projects with Embedded CDT and want to stay with it.

The specific of RISC-V is very active evolution of compiler arguments near -march -mabi and -mtune triplet. Vendors are improving there compilers and users are trying different combination to find the best memory footprint and performance for the concrete combination of device and engineering problem. Thus, the arguments of the compiler and linker are not more or less "static", as for a mature C/C++ project, but they change very often. Moreover, they should change according to the active toolchain capabilities. And this is where generation of manifest (either with MBS or with something else) can help a lot.

Also, users may have very different scenario of compiling, linking and debugging for the same project sources since it depends on HW/SW target environment. And this is where the build configuration as a kind of "profile for build settings" is really useful.

Core Build says that both (c)make generation and the build configuration are in the past. And to me it sounds like a degradation of the IDE toolkit for the described scenarios. However, I understand, that for other areas it may be a notable user experience improvement.

[ilg-ul]

The specific of RISC-V is very active evolution of compiler arguments near -march -mabi and -mtune triplet.

In my opinion, trying to support all possible RISC-V compiler arguments in a UI is no longer realistic. Many years ago when SiFive asked for the RISC-V plugin, the specs defined only a handful of ISA extensions; now there are tons of them, including vendor specific ones. If I'd have to redo the RISC-V plugin, I'd use plain strings for the -march, -mabi and -mtune options and ignore all those combinations of letters to define the extensions.

[ruspl-afed]

trying to support all possible RISC-V compiler arguments in a UI is no longer realistic

Yes, you are right, with the current MBS approach this is no longer realistic. But since we have different approach that involves our own extendable model of GCC spec, RISC-V extension, vendor-specific ToolChains and lot of over data

we have an ability to transform this data to MBS in the way we like programmatically. This way, each vendor can focus on the supported extensions.

[ilg-ul]

Sure, you can make things as complicated as you want to please the vendors, but, from the user point of view, is this extreme configurability really needed?

I'm no longer involved in active RISC-V development, and I have no idea what all those new extensions do, so I might be wrong, but how often are all those options changed during the project life cycle? My understanding is that the long -march string is actually device dependent, you set it when you create the project according to the device you have, then you do not need to touch it at all. If so, then this very elaborate configuration mechanism is not really needed.

[ruspl-afed]

Yes, you are right, this is vendor-oriented feature.
To be more precise, it is "core vendor" oriented feature. When it goes to MCU vendor - the set of core extension is already kind of frozen. But there is also another scenario: some vendors are customizing compiler quite deep and to try all their optimizations one need to specify the exact -march, -mabi, -mtune combination. And this combination may be not valid for another compiler fork.