sloeber.cfg is not enough for version control

[robzed]

Bug Summary

Sloeber.cfg does not include information about folders or files excluded from the build.

Environment

Sloeber 4.4.0.202102222152 io.sloeber.product null
OS: MacOS 11.6

Bug Description

To get a project previously built under Arduino IDE to build in Sloeber, some Library folders that need to be excluded. This is both entire libraries and some sub-folders of libraries (e.g. examples). This is not obvious, but it possible to resolve with some thought in Sloeber.

However a simple project in version control cannot be created that can be built without a lot of work or instructions because sloeber.cfg doesn't contain the information about excluded files/folders. This is the bug reported here. (although the library problem might be a separate bug or something to consider.)

Additionally user source files and folders that should be excluded are also not recorded. This basically the same bug as above.

To Reproduce

1. To get a simple Neopixel project building Sloeber e.g. https://github.com/robzed/LED_Display2, I had to exclude folders that are in the necessary libraries - these examples and other libraries give lots of errors

To reproduce clone this project and add new project to Sloeber. The configure to mbed_nano/2.6.1 with the Board as 'Arduino Nano 33 BLE'.

See the folders here that are excluded:

2. Additionally in a separate project I needed to exclude specific source files from the build.

This is the same in any project, in this case it's a pForth build where some files and folders are not applicable (for other platforms).

Expected behaviour

We should record excluded files/folders in the project settings so other users can easily rebuild from a cloned project.

Potential Workaround

Add .project and .cproject into version control.

References to Existing issues about version control

• What to include in Version control? #1307
• Better support for version control #1268
• Sloeber 4.4 removing a configuration via version control does not work #1350
• https://www.youtube.com/watch?v=bqdw1L_8Gjs
• https://github.com/Sloeber/arduino-eclipse-plugin/issues?q=is%3Aissue++label%3A%22status%3A+known+limitation%22+

[jantje]

First of all: it is great to see the version control functionality is used

sloeber.cfg is not enough for version control

I agree to that. This is why #1268 is named "Better support for version control"
Things I was aware of to improve are

1. Add platform installation information to the sloeber.cfg file
2. optionally install the platform if not present
3. add used libraries and libraries versions to sloeber.cfg (this requires support for multiple versions of the libraries)

Given this issue consider to add

1. Add exclusion path to sloeber.cfg

I'm saying "consider to add" because of following reasons

1. If point 3 above is added on should not have "unwanted libraries"
2. Arduino libraries should -with the default behaviour- not need additional excluded folders
3. As this is CDT storage territory it will be really hard to implement.

So in the LED_Display2 snapshot the examples and extra folders should be excluded by default.
The extra libraries should be fixed by point 3 of the aware improvements.

I do not understand the pForth problem you mention

FYI I tried to record the importing of the project but it did not contain a sloeber.cfg :-(

[robzed]

First - let me say thank you for your reply, and for your work on Sloeber. As a long time Eclipse user on embedded systems, I really appreciate the effort that has gone into this, and I'd prefer not to have to move to VSCode :-)

So in the LED_Display2 snapshot the examples and extra folders should be excluded by default. The extra libraries should be fixed by point 3 of the aware improvements.

Agreed. It would be better not to have to exclude library example folders manually. Even though I've been doing this for a long time, it still took me several hours to realise what was going on; I almost gave up.

I do not understand the pForth problem you mention

Sorry for my bad explanation. Let me try again.

There might be "user" sources files or folders with source files that cannot be built on the Arduino. For example in pForth there is:

• A folder with some Windows, and Unix specific source ... this will not built on the Arduino because we don't have those operating system calls.
• A file (pf_main.c) with command line argument processor, used when operating in from a terminal environment/command line, e.g. Windows Command Prompt or Linux/Mac terminal. This is not applicable for an embedded target, and hence is replaced with a Arduino specific caller.

These need to be excluded in the Eclipse build. Just deleting these files/folders is possible, but generally is a bad idea because you are effectively forking the code for different use cases. Hence the normal option is to use Resource Configuration->Exclude form build in Eclipse.

There is another "user" use-case for this:

• You might have one source code with multiple resource configurations - where some files are excluded from the build. Variants, if you like.

FYI I tried to record the importing of the project but it did not contain a sloeber.cfg :-(

I haven't added the sloeber.cfg to GitHub. I wanted you to convert this from the native Arduino IDE format to Sloeber, and hence it would add the libraries, etc. and duplicate what I saw. I don't think this matters too much - you understand the problems I saw (example library files).

As this is CDT storage territory it will be really hard to implement.

I understand.

On a standard embedded Eclipse CDT project I commit the .project and .cproject files into Git. Although they are not very easy to review (and on big projects causes merge conflicts), this is standard practice - I'm sort of surprised Sloeber is taking a different approach. (cproject especially is pain, but basically is unavoidable).

References:

• http://wiki.eclipse.org/FAQ_How_do_I_set_up_a_Java_project_to_share_in_a_repository%3F
• https://stackoverflow.com/questions/337304/which-eclipse-files-belong-under-version-control

[jantje]

I haven't added the sloeber.cfg to GitHub. I wanted you to convert this from the native Arduino IDE format to Sloeber, and hence it would add the libraries, etc. and duplicate what I saw. I don't think this matters too much - you understand the problems I saw (example library files).

Worked great on my system. rofl
Arduino took some decision around library management and inclusion that makes things .... well .... difficult and unpredictable.
Basically it depends on which libraries you have installed and in which order and in which order missing include files are reported what libraries are added.
I noticed that the more capable the board (like wifi on esp) the more likely Sloeber will add wrong libraries (like generic wifi stuff to esp) leading to more libraries being pulled in.

On a standard embedded Eclipse CDT project I commit the .project and .cproject files into Git. Although they are not very easy to review (and on big projects causes merge conflicts), this is standard practice - I'm sort of surprised Sloeber is taking a different approach. (cproject especially is pain, but basically is unavoidable).

Well .... lets say ... I do not always agree with CDT/eclipse policies views.
I used to check in .cprojects and .projects as well but I had lots of issues. In my opinion .cproject is not designed to be checked in and shared over windows/linux/mac. More over the Sloeber data used to be stored in the .settings files so more files need to be checked in for it to work.
In my personal opinion one should not check in .* files. .* files are by default invisible in Eclipse/Linux and why would one check in files that are so unimportant that you want them to be invisible by default? It simply doesn't make sense to me.
That being said I think it is safe to store the .project. But why would I? To safe some time on the import action?

Given that Arduino IDE and Sloeber both can start from a ino and add the boards and menu options I thought storing this info is a first good step to get version control easier to use. Given that it is a human readable text file it is less of a problem to convince repository owners to add it to the repository (compared to .project .cproject .sproject and the settings folder).

There might be "user" sources files or folders with source files that cannot be built on the Arduino.

We are here in the area of what Sloeber calls private libraries and private hardware.
Personally I do not see how one can 100% support these with version control.
What I mean is that libraries managed by the library manager and platforms managed by the platform manager are released and as such are identified by their json file specification and their version number.
Sloeber can find them and install them.
Moreover they are supposed to comply with the Arduino framework as if they don't they will not work in Arduino IDE nor in Sloeber for which they are made.

Private libraries and hardware on the other hand can by anything from a zip file to a version controlled repository.
Sloeber can not make assumptions on how you got to these and as such can not reproduce your project at import as it is dependent on -on the version server unknown- private libraries or hardware.
And private stuff does not have to comply with the Arduino framework as such forcing you to exclude folders.

You might have one source code with multiple resource configurations - where some files are excluded from the build. Variants, if you like.

I see this as a realistic scenario. I'd doubt it is frequent though and probably a consequence of bad housekeeping. And less priority than multiple versions of the same library and support for libraries in the sloeber.cfg file.