Requirements

The updated grammar should support syntax highlighting for the following:

• Binary integer literals: The prefix 0b or 0B indicates a binary integer literal such as 0b1011.
• Octal integer literals: The prefix 0o or 0O indicates an octal integer literal such as 0o660. The existing octal notation indicated by a leading 0 followed by octal digits remains valid.
• Hexadecimal floating point literals The prefix 0x or 0X may now be used to express the mantissa of a floating-point number in hexadecimal format such as 0x1.0p-1021. A hexadecimal floating-point number must always have an exponent, written as the letter p or P followed by an exponent in decimal. The exponent scales the mantissa by 2 to the power of the exponent.
• Imaginary literals: The imaginary suffix i may now be used with any (binary, decimal, hexadecimal) integer or floating-point literal.
• Digit separators: The digits of any number literal may now be separated (grouped) using underscores, such as in 1_000_000, 0b_1010_0110, or 3.1415_9265. An underscore may appear between any two digits or the literal prefix and the first digit.

(taken from Go 1.13 Release Notes)

Description of the Change

All the regular expressions are manually converted from the EBNF (Extended Backus-Naur Form) expressions for each number literal type found in The Go Programming Language Specification:

• Integer literals
• Floating-point literals
• Imaginary literals

Playground containing code snippet to test with: https://play.golang.org/p/IuWyvmHjHs0

EBNF to Regular Expression Conversion

Decimal literals

Binary literals

Octal literals

Hexadecimal literals

Decimal floating-point literals

Hexadecimal floating-point literals

Imaginary literals

Since the imaginary suffix i may now be used with any (binary, decimal, hexadecimal) integer or floating-point literal, we just need to append an i to the various set of regular expressions above.

There's only a slight difference in the pattern for imaginary literals (with octal integer literal), which we modify it to not collide with the old imaginary literals pattern (pre Go 1.12):

  {
    'comment': 'Imaginary literals (with octal integer literal)'
-   'match': '(^|(?<=[^_.\\da-zA-Z]))0[oO]?_?[0-7](_?[0-7])*i(?=[^_.\\da-zA-Z])'
+   'match': '(^|(?<=[^_.\\da-zA-Z]))0[oO]_?[0-7](_?[0-7])*i(?=[^_.\\da-zA-Z])'
    'name': 'constant.numeric.imaginary.octal-integer.go'
  }
  ...
+ {
+   'comment': 'Imaginary literals (for backwards compatibility)'
+   'match': '(^|(?<=[^_.\\da-zA-Z]))0+(_?\\d)*i(?=[^_.\\da-zA-Z])'
+   'name': 'constant.numeric.imaginary.backwards-compatibility.go'
+ }

Alternate Designs

Benefits

Patterns & scopes have been broken down by each number literal types for the following benefits:

• Smaller regular expressions are easier to maintain & audit
• Imaginary literals becomes a first class citizen like in tree-sitter/tree-sitter-go:
  • https://github.com/tree-sitter/tree-sitter-go/blob/master/corpus/literals.txt#L77
  • https://github.com/atom/language-go/blob/master/grammars/tree-sitter-go.cson#L85
• More detailed scopes would allows themes able to provide more "granular" syntax highlighting (eg. binary literals can be colored differently from decimal literals etc.)

Possible Drawbacks

This change introduces the following scopes:

• constant.numeric.integer.decimal.go
• constant.numeric.integer.binary.go
• constant.numeric.integer.octal.go
• constant.numeric.integer.hexadecimal.go
• constant.numeric.floating-point.decimal.go
• constant.numeric.floating-point.hexadecimal.go
• constant.numeric.imaginary.decimal-integer.go
• constant.numeric.imaginary.binary-integer.go
• constant.numeric.imaginary.octal-integer.go
• constant.numeric.imaginary.hexadecimal-integer.go
• constant.numeric.imaginary.decimal-floating-point.go
• constant.numeric.imaginary.hexadecimal-floating-point.go
• constant.numeric.imaginary.backwards-compatibility.go

The only "breaking" change I'm aware is that imaginary literal expressions currently matched under constant.numeric.integer would now be matched with scopes prefixed with constant.numeric.imaginary.*.

But as far as I know, majority of the color themes don't use anything beyond constant.numeric: https://github.com/search?l=Less&q=syntax--integer&type=Code

Applicable Issues

Fixes #174
Related tree-sitter/tree-sitter-go#31 (PR: tree-sitter/tree-sitter-go#32)