Add MDX language

[wooorm]

Description

MDX is a literate programming language that combines markdown with several JavaScript language features, such as JSX, expressions, and ESM.

• markdown: https://commonmark.org/
• JavaScript: https://tc39.es/ecma262/
• JSX: https://facebook.github.io/jsx/

Previously, there was some temporary duct-tape in place, to map .mdx extensions to markdown: GH-4416.
This improved the situation from nothing to something, but has issues, because the syntax that MDX brings wasn’t highlighted.

With this grammar, those can be properly highlighted.

Related-to: GH-4416.

Checklist:

• I am adding a new language.
  • The extension of the new language is used in hundreds of repositories on GitHub.com.
    • Search results for each extension:
      • https://github.com/search?q=path%3A*.mdx+NOT+nothack&type=code
  • I have included a real-world usage sample for all extensions added in this PR:
    • Sample source(s):
      • https://wooorm.com/markdown-tm-language/, based on examples from tests of the MDX parser, and on https://mdxjs.com
    • Sample license(s): ?
  • I have included a syntax highlighting grammar: https://github.com/wooorm/markdown-tm-language
  • I have added a color
    • Hex value: #fcb32c
    • Rationale: color from the MDX logo :) https://mdxjs.com
  • I have updated the heuristics to distinguish my language from others using the same extension. (not needed?)

---

P.S. the URL in the issue template doesn’t work with the new GH search:

https://github.com/search?utf8=%E2%9C%93&type=Code&ref=searchresults&q=extension%3AMDX+KEYWORDS+NOT+nothack

[lildude]

P.S. the URL in the issue template doesn’t work with the new GH search:

Yup. The new search still in beta and I'd rather keep things simple for now, especially as the new search does give a clue on how to correct things if you use the old syntax. The template will be changed when it becomes the default search for all users.

[wooorm]

Good point! ID added.

---

Ah, it’s still in beta, I forgot that I toggled it on!

---

I have some further Qs btw, hoping someone can help me!

1. The grammar in question uses \g<name> and later (?<name>...){0} patterns. That way, recursion can be supported: markdown has a couple cases where matching parens are fine, but unmatched ones aren’t. These are oniguruma extensions (https://github.com/kkos/oniguruma/blob/e35581d6568a47a981a57806f239f9de401c6511/doc/RE#LL460C14-L460C14). From what I can read on PCRE, they should also be supported here by GH: http://www.pcre.org/original/doc/html/pcrepattern.html#SEC25. I also find a couple of such uses in grammars that already exist here. But as I can’t easily test GHs internal highlighting, I wanted to make sure: are they supported?

2. As MDX, the language in question here, is literate programming, github/markup can’t really render (it would need to evaluate JavaScript to do so). So it’s important that github/markup doesn’t think it’s markdown. Can someone confirm whether this is enough (I think so?), or whether that’s done based on type: markup or ace_mode: markdown or so?

3. In the grammar repo, I also have a text.md grammar, for plain markdown. It’s a lot better than other grammars that I checked, including atom/language-gfm. My grammar has less bugs (take the sample in this PR for an example of lists and fenced code with tildes going haywire), supports more GFM features (tables, footnotes, plain URLs), and is maintained (atom/language-gfm is archived). I understand markdown is quite important to GitHub. Would it be of interest to swap out atom/language-gfm for this new repo? If there is potential interest, I can open a new discussion for this.

[lildude]

1. The grammar in question uses \g<name> and later (?<name>...){0} patterns. That way, recursion can be supported: markdown has a couple cases where matching parens are fine, but unmatched ones aren’t. These are oniguruma extensions (https://github.com/kkos/oniguruma/blob/e35581d6568a47a981a57806f239f9de401c6511/doc/RE#LL460C14-L460C14). From what I can read on PCRE, they should also be supported here by GH: http://www.pcre.org/original/doc/html/pcrepattern.html#SEC25. I also find a couple of such uses in grammars that already exist here. But as I can’t easily test GHs internal highlighting, I wanted to make sure: are they supported?

GitHub uses PCRE for performance reasons, so anything that PCRE supports, GitHub supports. The grammar compiler will flag anything that isn't PCRE compatible and any regex errors. You can see examples of this in #3924

2. As MDX, the language in question here, is literate programming, github/markup can’t really render (it would need to evaluate JavaScript to do so). So it’s important that github/markup doesn’t think it’s markdown. Can someone confirm whether this is enough (I think so?), or whether that’s done based on type: markup or ace_mode: markdown or so?

It won't. github/markup will only render whole markup files as HTML if they have one of these extensions. All ```mdx codeblocks within markdown files or GitHub comments, issues and PRs will be rendered as the raw code with syntax highlighting applied.

3. In the grammar repo, I also have a text.md grammar, for plain markdown. It’s a lot better than other grammars that I checked, including atom/language-gfm. My grammar has less bugs (take the sample in this PR for an example of lists and fenced code with tildes going haywire), supports more GFM features (tables, footnotes, plain URLs), and is maintained (atom/language-gfm is archived). I understand markdown is quite important to GitHub. Would it be of interest to swap out atom/language-gfm for this new repo? If there is potential interest, I can open a new discussion for this.

Most definitely! We're always on the lookout for actively maintained grammars, especially if they're better than anything else we're using. Feel free to skip the discussion and go straight to submitting a PR. Include examples where your grammar does things better than the current grammar too.

[wooorm]

github/markup will only render whole markup files as HTML if they have one of these extensions

Hmm, that source seems incorrect: MDX files, since the introduction of .mdx here, also started being rendered. For example: https://github.com/mdx-js/mdx-analyzer/blob/main/fixtures/node16/mixed.mdx. That’s why I wanted to be extra sure!

Feel free to skip the discussion and go straight to submitting a PR. Include examples where your grammar does things better than the current grammar too.

Awesome, will do!
I thought about waiting for this PR to land, as there might be a conflict.
But I think you wait with merging PRs like this until the next release cycle?
So I’ll just go ahead and make a PR :)

[lildude]

Hmm, that source seems incorrect: MDX files, since the introduction of .mdx here, also started being rendered. For example: https://github.com/mdx-js/mdx-analyzer/blob/main/fixtures/node16/mixed.mdx.

I've looked into this and it this would be our fault when we merged #4416 😁. Markup uses the language Linguist tells it to and if it matches one of the expected "markups", it'll attempt to render it using the appropriate rendered. Of course, this means when this PR is merged that rendering will disappear.

[lildude]

LGTM. Thanks.

Note: this PR will not be merged until close to when the next release is made. See here for more details.