Error in JDT Core during AST creation due to spurious recovery of String template expression by DiagnoseParser

[Madjosz]

With the following code three messages in the Error Log View pop up

package mwe;

import java.util.List;
import java.util.function.Predicate;
import java.util.stream.IntStream;

public class CompilationUnitConversion {
    private Predicate<String> matches() {
        return s -> {
            List<Object> results = List.of();
            return IntStream.range(0, 10).allMatch(results.get(i).);
        };
    }
}

Exception occurred during compilation unit conversion:

Error in JDT Core during AST creation

Problems occurred when invoking code from plug-in: "org.eclipse.jdt.core.manipulation".

java.lang.IllegalArgumentException
	at org.eclipse.jdt.core.dom.ASTNode$NodeList.add(ASTNode.java:1619)
	at java.base/java.util.AbstractList.add(AbstractList.java:111)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2528)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2046)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3021)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3092)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:1370)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2607)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:2055)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3021)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3092)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:738)
	at org.eclipse.jdt.core.dom.ASTConverter.buildBodyDeclarations(ASTConverter.java:208)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:3439)
	at org.eclipse.jdt.core.dom.ASTConverter.convert(ASTConverter.java:1521)
	at org.eclipse.jdt.core.dom.CompilationUnitResolver.convert(CompilationUnitResolver.java:344)
	at org.eclipse.jdt.core.dom.ASTParser.internalCreateASTCached(ASTParser.java:1269)
	at org.eclipse.jdt.core.dom.ASTParser.lambda$0(ASTParser.java:1120)
	at org.eclipse.jdt.internal.core.JavaModelManager.cacheZipFiles(JavaModelManager.java:5812)
	at org.eclipse.jdt.core.dom.ASTParser.internalCreateAST(ASTParser.java:1120)
	at org.eclipse.jdt.core.dom.ASTParser.createAST(ASTParser.java:868)
	at org.eclipse.jdt.core.manipulation.CoreASTProvider$1.run(CoreASTProvider.java:294)
	at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:47)
	at org.eclipse.jdt.core.manipulation.CoreASTProvider.createAST(CoreASTProvider.java:286)
	at org.eclipse.jdt.core.manipulation.CoreASTProvider.getAST(CoreASTProvider.java:199)
	at org.eclipse.jdt.core.manipulation.SharedASTProviderCore.getAST(SharedASTProviderCore.java:138)
	at org.eclipse.jdt.internal.ui.viewsupport.SelectionListenerWithASTManager$PartListenerGroup.calculateASTandInform(SelectionListenerWithASTManager.java:166)
	at org.eclipse.jdt.internal.ui.viewsupport.SelectionListenerWithASTManager$PartListenerGroup$1.run(SelectionListenerWithASTManager.java:151)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:63)

It also messes with postfix completions on the trailing . after the get() call:

The 'org.eclipse.jdt.ui.postfixCompletion' proposal computer from the 'org.eclipse.jdt.ui' plug-in did not complete normally. The extension has thrown a runtime exception.

System information

Eclipse SDK
Version: 2024-03 (4.31)
Build id: I20231207-1800

openjdk 21 2023-09-19 LTS
OpenJDK Runtime Environment Temurin-21+35 (build 21+35-LTS)
OpenJDK 64-Bit Server VM Temurin-21+35 (build 21+35-LTS, mixed mode, sharing)

[srikanth-sankaran]

Interesting.

The parse tree for the method comes out to be:

private Predicate<String> matches() {
  return (<no type> s) -> {
  List<Object> results = List.of();
  return IntStream.range(0, 10).allMatch(results.get(i)."");
};
}

See the "" at the end of the return statement inside the lambda!

Basically we have parsed a string template expression where none existed. I suspect DiagnoseParser's black magic having a finger in the pie

[srikanth-sankaran]

Yep. DiagnoseParser inserts a "" at the syntax error site and then during recovery we see a String template. This itself is not the problem per se - org.eclipse.jdt.core.dom.ASTConverter.convert(Expression) is lacking a case for StringTemplates and returns null which then results in IAE.

We may want to steer the DiagnoseParser from inserting "" into the Recovery Scanner's input stream - otherwise code completion may break as reported above

[Madjosz]

Latest build of Eclipse (I20240205-1800) does not throw anymore and shows

Syntax error on token ".", StringLiteral expected after this token

when hovering the last . and on results.get(i) it shows

String Template is a preview feature and disabled by default. Use --enable-preview to enable

This is better (as it does not hard-fail the build) but not at all what this code is about. Content assist also suggests List methods after last . but that might be due to the missing declaration of the i variable and thus the compiler not being able to find a suitable method.

[Madjosz]

aa1a02d fixed the errors appearing in the Error Log view and I would consider this issue fixed.
The spurious recovery of String template might be worth its own issue but there might already be enough issues about String templates so we could revisit this behaviour when the implementation is more mature.

[Madjosz]

... or wait until they drop String Templates at all from the JDK.

With that feature removed the error message is back to

Syntax error on token ".", Identifier expected after this token

which is the expected error message from before. I'll close this, thanks to all the people involved.