[23] DOM changes for markdown Javadoc comments #2738

[jarthana]

What it does

Partially fixes #2738

How to test

Author checklist

• I have thoroughly tested my changes
• The change is following the coding conventions
• I have signed the Eclipse Contributor Agreement (ECA)

[jarthana]

@stephan-herrmann , there's a long way to go for this PR. Plenty of new tests still failing. I already see that the flatteners are going to need some changes to the existing DOM Javadoc nodes to differentiate markdown comments. I would like you to try this out and let me know what you would need in terms of the DOM changes for rendering.

[github-actions]

Test Results

    11 files  ±0      11 suites  ±0   19m 23s ⏱️ -41s
77 969 tests ±0  77 969 ✅ ±0  0 💤 ±0  0 ❌ ±0 
78 029 runs  ±0  78 029 ✅ ±0  0 💤 ±0  0 ❌ ±0 

Results for commit 02d70fc. ± Comparison against base commit 707447a.

♻️ This comment has been updated with latest results.

[stephan-herrmann]

@stephan-herrmann , there's a long way to go for this PR. Plenty of new tests still failing. I already see that the flatteners are going to need some changes to the existing DOM Javadoc nodes to differentiate markdown comments. I would like you to try this out and let me know what you would need in terms of the DOM changes for rendering.

Thanks @jarthana. I do get some markdown Javadoc DOM nodes, great! As soon as links are involved, some details don't yet look right, but I need to take a closer look to find out if that is caused in DOM or during rendering. I'll let you know.

[stephan-herrmann]

@jarthana once you feel somewhat comfortable with existing changes, despite any incompleteness merging this change would let me push some real changes in JDT/UI. That would establish a basis for investigating into details and corner cases.

As mentioned before a new API Javadoc.isMarkdown() or Javadoc.getCommentStyle() would allow for a cleaner check, but currently the workaround to test the raw javadoc for leading /// is functional.

[stephan-herrmann]

@jarthana the following two changes help a great deal to pass correct IDocElements into the renderer:

diff --git a/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/AbstractCommentParser.java b/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/AbstractCommentParser.java
index 70a929a..bcba898 100644
--- a/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/AbstractCommentParser.java
+++ b/org.eclipse.jdt.core.compiler.batch/src/org/eclipse/jdt/internal/compiler/parser/AbstractCommentParser.java
@@ -504,7 +504,7 @@
 				}
 				refreshInlineTagPosition(textEndPosition);
 				setInlineTagStarted(false);
-			} else if (this.lineStarted && this.textStart != -1 && this.textStart <= textEndPosition && (this.textStart < this.starPosition || this.starPosition == lastStarPosition)) {
+			} else if (this.lineStarted && this.textStart != -1 && this.textStart <= textEndPosition && (this.textStart < this.starPosition || this.starPosition == lastStarPosition || this.markdown)) {
 				pushText(this.textStart, textEndPosition);
 			}
 			updateDocComment();
diff --git a/org.eclipse.jdt.core/dom/org/eclipse/jdt/core/dom/DocCommentParser.java b/org.eclipse.jdt.core/dom/org/eclipse/jdt/core/dom/DocCommentParser.java
index c125d44..bdd0b86 100644
--- a/org.eclipse.jdt.core/dom/org/eclipse/jdt/core/dom/DocCommentParser.java
+++ b/org.eclipse.jdt.core/dom/org/eclipse/jdt/core/dom/DocCommentParser.java
@@ -761,7 +761,7 @@
 						// Both tag elements must get the same source range.
 						TagElement previousTag = (TagElement) this.astStack[this.astPtr];
 						int parentStart = previousTag.getStartPosition();
-						int length = this.index - previousPosition + 1;
+						int length = this.index - previousPosition ;
 						previousTag.setSourceRange(parentStart, this.index - parentStart + 1);
 						List fragments = previousTag.fragments();
 						int size = fragments.size();
@@ -774,7 +774,7 @@
 							// If last fragment is a tag, then use it as previous tag
 							ASTNode lastFragment = (ASTNode) fragments.get(size-1);
 							if (lastFragment.getNodeType() == ASTNode.TAG_ELEMENT) {
-								//lastFragment.setSourceRange(lastFragment.getStartPosition(), length);
+								lastFragment.setSourceRange(lastFragment.getStartPosition(), length);
 								previousTag = (TagElement) lastFragment;
 							}
 						}

• DocCommentParser: the same change discussed above, which avoids showing the closing ] of a link
• AbstractCommentParser: this change fixes a bug whereby the last text element of a doc comment was dropped.

I would love to see these merged as part of this PR, whereas other findings can be conveniently handled separately, notably:

• javadoc: Explicit link to package without package-info.java does not work eclipse.jdt.ui#1579 filed against JDT/UI and not specific to markdown
• [23] markdown javadoc should detect code regions #2808 -- for a full solution we actually need clarification regarding an observed diff spec vs. javadoc tool

[jarthana]

In the interest of getting ourselves unblocked on other issues, I have removed the ASTConverterMarkdownTest from the convert test suite and planning to release this today. The ASTConverterMarkdownTest takes after ASTConverterJavadocTest, which has another ad-hoc parser/scanner that parses the Javadoc and compares with the DOM output. I am yet to completely have that under control. Will take that up immediately after this.