lexer: incorrect tokenization

[sunshineco]

According to POSIX section 2.3, paying special attention to rules 7 & 8, input such as nothing#42'was not'here"with me $(echo flash)"!right@now should be scanned as a single token. For instance (using dash):

% x=nothing#42'was not'here"with me $(echo flash)"!right@now
% echo "[$x]"
[nothing#42was notherewith me flash!right@now]

However, flash's lexer misinterprets it badly. For example, input x=nothing#here should be scanned as assignment of the single token nothing#here to variable x, however, flash incorrectly splits off #here as a comment:

List {
    statements: [
        Assignment {
            name: "x",
            value: StringLiteral(
                "nothing",
            ),
        },
        Comment(
            "#here",
        ),
    ],
    operators: [
        "",
    ],
}

Similarly, echo 42'was not'here"with me"!right@now should present echo with the single argument 42was notherewith me!right@now, but the lexer incorrectly scans it as separate tokens which the parser considers separate argument, and incorrectly splits off right@now as a history expansion:

List {
    statements: [
        Command {
            name: "echo",
            args: [
                "42",
                "was not",
                "here",
                "with me",
            ],
            redirects: [],
        },
        HistoryExpansion {
            pattern: "right@now",
        },
    ],
    operators: [
        "",
    ],
}

The assignment x=42'was not'here"with me"!right@now should assign the single value 42was notherewith me!right@now to variable x, however, due to the lexer scanning it incorrectly as multiple tokens, the parser incorrectly interprets it as the much simpler assignment x=42 followed by a command invocation, and loses the ! altogether:

List {
    statements: [
        Assignment {
            name: "x",
            value: StringLiteral(
                "42",
            ),
        },
        Command {
            name: "was not",
            args: [
                "herewith meright@now",
            ],
            redirects: [],
        },
    ],
    operators: [
        "",
    ],
}

The similar input x='was not'here"with me"!right@now should assign the value was notherewith me!right@now to variable x but instead is incorrectly interpreted as an assignment followed by a command with one argument, as well as a history expansion:

List {
    statements: [
        Assignment {
            name: "x",
            value: StringLiteral(
                "was not",
            ),
        },
        Command {
            name: "here",
            args: [
                "with me",
            ],
            redirects: [],
        },
        HistoryExpansion {
            pattern: "right@now",
        },
    ],
    operators: [
        "",
        "",
    ],
}