CodeFormatter produces inconsistent results when used through CodeFormatterApplication

[GeekOffTheStreet]

Describe the bug
We have some workflows that use the CDT code formatter application to auto-format source code and noticed some unexpected results when run on projects using C++11 or later features. When using the same ruleset and formatting from the GUI, the format was consistent with our expectations and different from the output when running from CodeFormatter application from command line.

To Reproduce
Steps to reproduce the behavior:

1. Create a new CDT C++ managed build project using Linux GCC toolchain with "hello world" template
2. Right click on project properties and navigate to C/C++ General -> Formatter. Click "Enable project specific settings" and select a ruletset. It doesn't matter, but following output will use "GNU [built-in]". Then click apply and close.
3. Open src/test_format.cpp and paste in the following:

#include <tuple>

int main() {
    std::tuple<int, int> blah;
    std::get<0>(blah) = 2;
    return 0;
}

5. Format the file (e.g. ctrl+shift+f). You should see output resembling:

#include <tuple>

int
main ()
{
  std::tuple<int, int> blah;
  std::get<0> (blah) = 2;
  return 0;
}

6. Now from a terminal run the CodeFormatter application on the same file e.g.; eclipse -nosplash -application org.eclipse.cdt.core.CodeFormatter -verbose -config .settings/org.eclipse.cdt.core.prefs src/test_format.cpp -vmargs -Dosgi.dataAreaRequiresExplicitInit=false

#include <tuple>

int
main ()
{
  std::tuple<int, int> blah;
  std::get < 0 > (blah) = 2;
  return 0;
}

Note std::get <0> turned into std::get < 0 >

Expected behavior
Formatting code using the same ruleset in Eclipse GUI and from CodeFormatter command line application produce identical results.

Screenshots
If applicable, add screenshots to help explain your problem.

Version Information (please complete the following information):

• OS and OS Version/extra details: Linux (RHEL8 x86_64)
• Eclipse Version from Help -> About: 2023-12 (4.30.0). Also tested with Version: 2024-03 (4.31.0)
• CDT Version from Help -> About -> Installation Details -> Features tab: 11.4.0

Additional context
The issue seems to be that the CodeFormatter application is configuring a default toolchain that doesn't understand modern C++. I tried cloning the CDT source code and hacking at it. I have never looked at any Eclipse source code before, so some of this may be incorrect, but here are my observations anyways. The formatting starts here:

cdt/core/org.eclipse.cdt.core/src/org/eclipse/cdt/internal/formatter/CCodeFormatter.java

Line 189 in 98e86ff

ast = language.getASTTranslationUnit(content, scanInfo, contentProvider, null, 0,

Things that seem important for future processing is that we're always creating a blank ScanInfo (line 180) and ILanguage language (line 183) will always be null as that option stores an object instance that gets set at runtime when opening projects and isn't going to be present in the project's .settings/org.eclipse.cdt.core.prefs .

Moving on to creating the IASTTranslationUnit

cdt/core/org.eclipse.cdt.core/parser/org/eclipse/cdt/core/dom/parser/AbstractCLikeLanguage.java

Line 145 in 98e86ff

final IScanner scanner = createScanner(reader, scanInfo, fileCreator, log);

Two bits seem to be important - the scanner and parser. The scanner when run CodeFormatter applications ends up returning the default GCC implementation, which lacks keywords in versioned companions:

cdt/core/org.eclipse.cdt.core/parser/org/eclipse/cdt/core/dom/parser/cpp/GPPScannerExtensionConfiguration.java

Line 48 in 98e86ff

private static GPPScannerExtensionConfiguration CONFIG = new GPPScannerExtensionConfiguration();

I tried changing the default private static GPPScannerExtensionConfiguration CONFIG = new GPPScannerExtensionConfiguration(); to instead instantiate newer versions (e.g. new GPPScannerExtensionConfiguration(VERSION_10_0);), but that didn't have any observable effect on behavior. So that alone isn't sufficient.

When running the formatter from the GUI, the parser is created with an ExtendedScanInfo object that has parser settings:

cdt/core/org.eclipse.cdt.core/parser/org/eclipse/cdt/core/dom/parser/AbstractCLikeLanguage.java

Lines 149 to 152 in 98e86ff

	IParserSettings parserSettings = null;
	if (scanInfo instanceof ExtendedScannerInfo) {
	ExtendedScannerInfo extendedScannerInfo = (ExtendedScannerInfo) scanInfo;
	parserSettings = extendedScannerInfo.getParserSettings();

I can't see any way to hack those in without having actually imported a CDT Project, and not sure if I'm going down a rabbit hole. However, there seems to be something wrong with how the CodeFormatter is parsing modern C++, and there should be some way to configure it to understand modern syntax to prevent conflicting output with GUI.

[GeekOffTheStreet]

Ok, I tried a few things but still wasn't able to get things working.

I tried to change:

cdt/core/org.eclipse.cdt.core/src/org/eclipse/cdt/internal/formatter/CCodeFormatter.java

Lines 189 to 190 in 98e86ff

	ast = language.getASTTranslationUnit(content, scanInfo, contentProvider, null, 0,
	ParserUtil.getParserLogService());

To pass in 0x30 for options instead of 0', which is what is being passed in when running from the GUI. I also pre-populated scanInfo` with all of the macros that are getting set from the GUI. This resembled:

			Map<String, String> macros = new HashMap<>();
			macros.put("__LDBL_DECIMAL_DIG__", "21");
			macros.put("__INT64_MAX__", "0x7fffffffffffffffL");
                        // <snip />
                       // pass in macro map instead of calling default constructor
                       IScannerInfo scanInfo = new ScannerInfo(macros);

This had the effect of getting all of the macros and predefined macros getting added to CPreprocessor:

cdt/core/org.eclipse.cdt.core/parser/org/eclipse/cdt/internal/core/parser/scanner/CPreprocessor.java

Lines 539 to 557 in 98e86ff

	IMacro[] toAdd = config.getAdditionalMacros();
	if (toAdd != null) {
	for (final IMacro macro : toAdd) {
	addMacroDefinition(macro.getSignature(), macro.getExpansion());
	}
	}
	final Map<String, String> macroDict = info.getDefinedSymbols();
	if (macroDict != null) {
	for (Map.Entry<String, String> entry : macroDict.entrySet()) {
	final String key = entry.getKey();
	final String value = entry.getValue().trim();
	addMacroDefinition(key.toCharArray(), value.toCharArray());
	}
	}
	Collection<PreprocessorMacro> predefined = fMacroDictionary.values();
	for (PreprocessorMacro macro : predefined) {
	fLocationMap.registerPredefinedMacro(macro);

Unfortunately, that wasn't sufficient and I'm out of obvious differences to try and hack in.

[polfeliu]

Experiencing something similar, very annoying