[Bug][Monarch] A newline is not supported in Monarch token

[BeishaoCao-db]

Reproducible in vscode.dev or in VS Code Desktop?

• Not reproducible in vscode.dev or VS Code Desktop

Reproducible in the monaco editor playground?

• Reproducible in the monaco editor playground

Monaco Editor Playground Code

// Create your own language definition here
// You can safely look at other samples without losing modifications.
// Modifications are not saved on browser refresh/close though -- copy often!
return {
  // Set defaultToken to invalid to see what you do not tokenize yet
  // defaultToken: 'invalid',

  keywords: [
    'abstract', 'continue', 'for', 'new', 'switch', 'assert', 'goto', 'do',
    'if', 'private', 'this', 'break', 'protected', 'throw', 'else', 'public',
    'enum', 'return', 'catch', 'try', 'interface', 'static', 'class',
    'finally', 'const', 'super', 'while', 'true', 'false'
  ],

  typeKeywords: [
    'boolean', 'double', 'byte', 'int', 'short', 'char', 'void', 'long', 'float'
  ],

  operators: [
    '=', '>', '<', '!', '~', '?', ':', '==', '<=', '>=', '!=',
    '&&', '||', '++', '--', '+', '-', '*', '/', '&', '|', '^', '%',
    '<<', '>>', '>>>', '+=', '-=', '*=', '/=', '&=', '|=', '^=',
    '%=', '<<=', '>>=', '>>>='
  ],

  // we include these common regular expressions
  symbols:  /[=><!~?:&|+\-*\/\^%]+/,

  // C# style strings
  escapes: /\\(?:[abfnrtv\\"']|x[0-9A-Fa-f]{1,4}|u[0-9A-Fa-f]{4}|U[0-9A-Fa-f]{8})/,

  // The main tokenizer for our languages
  tokenizer: {
    root: [
      [
        /LANGUAGE\s+PYTHON\s+AS\s+\$\$/, // Note: does not handle new lines
        { token: 'keyword', bracket: '@open', next: '@python_block', nextEmbedded: 'python' },
      ],
      // identifiers and keywords
      [/[a-z_$][\w$]*/, { cases: { '@typeKeywords': 'keyword',
                                   '@keywords': 'keyword',
                                   '@default': 'identifier' } }],
      [/[A-Z][\w\$]*/, 'type.identifier' ],  // to show class names nicely

      // whitespace
      { include: '@whitespace' },

      // delimiters and operators
      [/[{}()\[\]]/, '@brackets'],
      [/[<>](?!@symbols)/, '@brackets'],
      [/@symbols/, { cases: { '@operators': 'operator',
                              '@default'  : '' } } ],

      // @ annotations.
      // As an example, we emit a debugging log message on these tokens.
      // Note: message are supressed during the first load -- change some lines to see them.
      [/@\s*[a-zA-Z_\$][\w\$]*/, { token: 'annotation', log: 'annotation token: $0' }],

      // numbers
      [/\d*\.\d+([eE][\-+]?\d+)?/, 'number.float'],
      [/0[xX][0-9a-fA-F]+/, 'number.hex'],
      [/\d+/, 'number'],

      // delimiter: after number because of .\d floats
      [/[;,.]/, 'delimiter'],

      // strings
      [/"([^"\\]|\\.)*$/, 'string.invalid' ],  // non-teminated string
      [/"/,  { token: 'string.quote', bracket: '@open', next: '@string' } ],

      // characters
      [/'[^\\']'/, 'string'],
      [/(')(@escapes)(')/, ['string','string.escape','string']],
      [/'/, 'string.invalid']
    ],

    python_block: [[/\$\$/, { token: 'keyword', next: '@popall', nextEmbedded: '@pop' }]],

    comment: [
      [/[^\/*]+/, 'comment' ],
      [/\/\*/,    'comment', '@push' ],    // nested comment
      ["\\*/",    'comment', '@pop'  ],
      [/[\/*]/,   'comment' ]
    ],

    string: [
      [/[^\\"]+/,  'string'],
      [/@escapes/, 'string.escape'],
      [/\\./,      'string.escape.invalid'],
      [/"/,        { token: 'string.quote', bracket: '@close', next: '@pop' } ]
    ],

    whitespace: [
      [/[ \t\r\n]+/, 'white'],
      [/\/\*/,       'comment', '@comment' ],
      [/\/\/.*$/,    'comment'],
    ],
  },
};

Reproduction Steps

1. Using the same code in monarch playground https://microsoft.github.io/monaco-editor/monarch.html
2. add one token in tokenizer.root:
   [ /LANGUAGE\s+PYTHON\s+AS\s+\$\$/, { token: 'keyword', bracket: '@open', next: '@python_block', nextEmbedded: 'python' }, ],
   and one python block:
   python_block: [[/\$\$/, { token: 'keyword', next: '@popall', nextEmbedded: '@pop' }]],
3. We can see that LANGUAGE PYTHON AS $$ will open a python block, but if we enter a newline inside the token like LANGUAGE PYTHON AS \n $$ and it won't be consider as a token.
4. Question is: why we have \s in the matching rule but not include \n? And is there a way to achieve that we define a token and support newline? Currently my guess is monaco will tokenize the code line by line and it cause this behavior

Actual (Problematic) Behavior

We can see that LANGUAGE PYTHON AS $$ will open a python block, but if we enter a newline inside the token like LANGUAGE PYTHON AS $$ and it won't be consider as a token.

Expected Behavior

EXPECT:
Add a newline inside won't affect LANGUAGE PYTHON AS \n $$ to become a token

Additional Context

why we have \s in the matching rule but not include \n? And is there a way to achieve that we define a token and support newline? Currently my guess is monaco will tokenize the code line by line and it cause this behavior and is there any workaround?

[Prinzhorn]

Maybe you're looking for includeLF: true? https://microsoft.github.io/monaco-editor/api/interfaces/monaco.languages.IMonarchLanguage.html#includeLF

Edit: no, that probably won't work since the tokenizer goes line-by-line. You'll never get a token with multiple lines.

[BeishaoCao-db]

@Prinzhorn Thanks for your clarification!