VSCode doesn't send textDocument & semantic highlighting requests

[wakaztahir]

Does this issue occur when all extensions are disabled?: Yes

• VS Code Version: 1.86.1
• OS Version: Windows

I am trying to build a Language Server, For that I used LspCpp in C++ to build the language server, so it would be really fast. I started out the basic implementation and I am not receiving the textDocument requests (didOpen, change), also not receiving the semantic highlighting request for which I am interested in. I tried looking for other issues and made similar changes to my configuration file but nothing fixed it.

This doesn't seem like server issue, The request is just not being sent !

Notes

• Initialize request received
• Text Document requests not received didOpen, change
• Initialized notification received with semantic highlighting not supported
• Yes I'm sending true in capabilities along with token types and modifiers

What I've tried

• Tried enabling semantic highlighting in VSCode settings !
• Changing configuration file of LSP extension
• Tried opening & closing the files
• I've checked the Uncaught exceptions and there are no uncaught exceptions, I tried checking caught exceptions and there were too many.
• The output / debug console don't contain any informative information, The developer tools also don't contain any helpful information about what's happening
• Updated to latest version of VSCode

Projects

Here's my LSP client extension :
https://github.com/Qinetik/chemical-vscode

Here's my LSP server implementation :
https://github.com/Qinetik/chemical

Chat with the owner of LSPCpp :
kuafuwang/LspCpp#35 (comment)

[RedCMD]

the client needs to receive the requests and forward them onto the server

context.subscriptions.push(
	vscode.workspace.onDidOpenTextDocument((document: vscode.TextDocument) => {
		// forward onto LSP
	})
);
context.subscriptions.push(
	vscode.languages.registerDocumentSemanticTokensProvider(DocumentSelector, DocumentSemanticTokensProvider, SemanticTokensLegend)
);

[wakaztahir]

Can you tell me how exactly do I forward these requests to the server.

I've written this code

    const provider : DocumentSemanticTokensProvider = {
        provideDocumentSemanticTokens: function (document: TextDocument, token: CancellationToken): ProviderResult<SemanticTokens> {
            throw new Error("Function not implemented.");
        },
        onDidChangeSemanticTokens : null,
        provideDocumentSemanticTokensEdits : null,
    }

    const legend : SemanticTokensLegend = {
        tokenTypes: [   
            SemanticTokenTypes.namespace,
            SemanticTokenTypes.type,
            SemanticTokenTypes.class,
            SemanticTokenTypes.enum,
            SemanticTokenTypes.interface,
            SemanticTokenTypes.struct,
            SemanticTokenTypes.typeParameter,
            SemanticTokenTypes.parameter,
            SemanticTokenTypes.variable,
            SemanticTokenTypes.property,
            SemanticTokenTypes.enumMember,
            SemanticTokenTypes.event,
            SemanticTokenTypes.function,
            SemanticTokenTypes.method,
            SemanticTokenTypes.macro,
            SemanticTokenTypes.keyword,
            SemanticTokenTypes.modifier,
            SemanticTokenTypes.comment,
            SemanticTokenTypes.string,
            SemanticTokenTypes.number,
            SemanticTokenTypes.regexp,
            SemanticTokenTypes.operator,
            SemanticTokenTypes.decorator
        ],
        tokenModifiers: [
            SemanticTokenModifiers.declaration,
            SemanticTokenModifiers.definition,
            SemanticTokenModifiers.readonly,
            SemanticTokenModifiers.static,
            SemanticTokenModifiers.deprecated,
            SemanticTokenModifiers.abstract,
            SemanticTokenModifiers.async,
            SemanticTokenModifiers.modification,
            SemanticTokenModifiers.documentation,
            SemanticTokenModifiers.defaultLibrary
        ]
    }

    context.subscriptions.push(
        languages.registerDocumentSemanticTokensProvider(['ch'], provider, legend)
    );

[RedCMD]

context.subscriptions.push(languages.registerDocumentSemanticTokensProvider(['ch'], provider, legend));

putting that in activate() will work

throw new Error("Function not implemented.");

here you will need to contact your server and then return the reply
I haven't setup an LSP before, I've only custom wrote the parser directly in that function
I would recommend looking at other extensions that use an LSP
like C, JS, JSON, HTML etc
or maybe some smaller 3rd extensions that'll be easier to comprehend

[wakaztahir]

I can't seem to find a implementation of an LSP that has such code. Plus they are so complicated !

[wakaztahir]

In the semantic-tokens-sample The parse text is implemented on the client side, So its still not going to send the request to server !

https://github.com/microsoft/vscode-extension-samples/blob/2eb7bad1195e4270ce93a179aa45f18efa0f953e/semantic-tokens-sample/src/extension.ts#L67

There should be a sample that provides semantic tokens through server by making request using the language server client !

The semantic tokens request is still not being sent, even with this Code, I've added console logs, This is what I get in console

I am not getting the [Request] textDocument/semanticTokens/full

It seems the extension is not even attempting to send the request, even after hooking the document semantic tokens provider

export function activate(context: ExtensionContext) {

    let serverOptions = () => {
        // Connect to language server via socket
        let socket = net.connect({ port: 5007 })
        let result: StreamInfo = {
            writer: socket,
            reader: socket
        };
        return Promise.resolve(result);
    };

    let clientOptions: LanguageClientOptions = {
        documentSelector: ['ch'],
        synchronize: {
            fileEvents: workspace.createFileSystemWatcher('**/*.ch')
        }
    };

    // Create the language client and start the client.
    lc = new LanguageClient('Chemical Server', serverOptions, clientOptions);

    context.subscriptions.push(
        workspace.onDidOpenTextDocument((document: TextDocument) => {
            console.log("[Request] onDidOpenTextDocument Not Being Sent To LSP Server")
        })
    );

    context.subscriptions.push(vscode.languages.registerDocumentSemanticTokensProvider(['ch'], new DocumentSemanticTokensProvider(), legend));

    lc.setTrace(Trace.Verbose);
    lc.start().then(() => {
        console.log("[Debug] ChemicalLSP Running")
    }).catch((e) => {
        console.error("[Debug] Error running ChemicalLSP", e)
    });

}

export function deactivate() {
    return lc.stop().then(() => {
        console.log("[Debug] ChemicalLSP Stopped")
    }).catch((e) => {
        console.error("[Debug] Error Stopping ChemicalLSP", e)
    });
}

// interface IParsedToken {
// 	line: number;
// 	startCharacter: number;
// 	length: number;
// 	tokenType: string;
// 	tokenModifiers: string[];
// }

class DocumentSemanticTokensProvider implements vscode.DocumentSemanticTokensProvider {

    async provideDocumentSemanticTokens(document: vscode.TextDocument, token: vscode.CancellationToken): Promise<vscode.SemanticTokens> {
        console.log("[Request] textDocument/semanticTokens/full");
        return lc.sendRequest("textDocument/semanticTokens/full", { textDocument : document }).catch(e => {
            console.error("Error sending semantic tokens request", e)
            return e
        })
    }
    
    // TODO
    // onDidChangeSemanticTokens ?: vscode.Event<void> = function (e) {
            // Not implemented Yet
            // Also the return type is Disposable
    // }

    // TODO
    // async provideDocumentSemanticTokensEdits(document: vscode.TextDocument, previousResultId: string, token: vscode.CancellationToken): vscode.ProviderResult<vscode.SemanticTokens | vscode.SemanticTokensEdits> {
           // Not implemented Yet
    // }

    private _encodeTokenType(tokenType: string): number {
        if (tokenTypes.has(tokenType)) {
            return tokenTypes.get(tokenType)!;
        } else if (tokenType === 'notInLegend') {
            return tokenTypes.size + 2;
        }
        return 0;
    }

    private _encodeTokenModifiers(strTokenModifiers: string[]): number {
        let result = 0;
        for (let i = 0; i < strTokenModifiers.length; i++) {
            const tokenModifier = strTokenModifiers[i];
            if (tokenModifiers.has(tokenModifier)) {
                result = result | (1 << tokenModifiers.get(tokenModifier)!);
            } else if (tokenModifier === 'notInLegend') {
                result = result | (1 << tokenModifiers.size + 2);
            }
        }
        return result;
    }

    // private async _convertParsedTokens(allTokens : IParsedToken[]) : Promise<vscode.SemanticTokens> {
    //     const builder = new vscode.SemanticTokensBuilder();
    //     allTokens.forEach((token) => {
    //         builder.push(token.line, token.startCharacter, token.length, this._encodeTokenType(token.tokenType), this._encodeTokenModifiers(token.tokenModifiers));
    //     });
    //     return builder.build();
    // }

}

[wakaztahir]

Ok, I've found the error, The error is that you need onLanguage:semanticLanguage in package.json in activationEvents field

microsoft/vscode-extension-samples#898

[wakaztahir]

I guess I'm still not done, I'm getting the request on server, I can also console.log before sending the request and its also showing in the debug console

BUT -> no syntax highlighting !

Here's what my server is returning for the full semanticTokens Request :

Server Returns:

std::vector<SemanticToken> toks;
toks.push_back(SemanticToken{
        0, 0, 4, SemanticTokenType::ls_keyword, 0
});
toks.push_back(SemanticToken{
        0, 5, 1, SemanticTokenType::ls_variable, 0
});
toks.push_back(SemanticToken{
        0, 2, 1, SemanticTokenType::ls_operator, 0
});
toks.push_back(SemanticToken{
        0, 2, 1, SemanticTokenType::ls_number, 0
});
toks.push_back(SemanticToken{
        0, 1, 1, SemanticTokenType::ls_operator, 0
});

Client Console

Sample Code

#var x = 5;

Update : Server Exception Occurred

Reason:Exception  when process requestnotification message:
std::string
content:
{"jsonrpc":"2.0","id":2,"method":"textDocument/semanticTokens/full","params":{"textDocument":{"uri":{"$mid":1,"fsPath":"
d:\\Programming\\Chemical\\chemical-vscode\\sample\\language.ch","_sep":1,"external":"file:///d%3A/Programming/Chemical/
chemical-vscode/sample/language.ch","path":"/d:/Programming/Chemical/chemical-vscode/sample/language.ch","scheme":"file"
},"fileName":"d:\\Programming\\Chemical\\chemical-vscode\\sample\\language.ch","isUntitled":false,"languageId":"chemical
","version":1,"isClosed":false,"isDirty":false,"eol":2,"lineCount":1}}}

[wakaztahir]

Fixed With :

    async provideDocumentSemanticTokens(document: vscode.TextDocument, token: vscode.CancellationToken): Promise<vscode.SemanticTokens> {
        console.log("[Request] textDocument/semanticTokens/full");

        const params : SemanticTokensParams = {
            textDocument : {
                uri : document.uri.toString()
            }
        }
              // @ts-ignore
        return lc.sendRequest("textDocument/semanticTokens/full", params).catch(e => {
            console.error("Error sending semantic tokens request", e)
            return Promise.reject(e)
        })
    }

Oooooh this looks good !

[wakaztahir]

Microsoft : Hire me please...

[wakaztahir]

VSCode is not sending didOpen , didChange & didClose requests as well, To fix this I tried these things

with semantic highlighting full request, I can provide the semantic tokens when the user opens the file for the first time, BUT making changes to source code is known to VSCode only which is not saved in the file on disk, which is read by my server providing tokens.

To provide the very fresh tokens for fresh edits user made, I must track the source code changes done by the user, vscode will send these changes to my server, I'll keep it in memory as overridden source then when vscode asks for tokens I'll provide it from overridden source instead of file on disk

1 - Catch and send requests from client to the server in the extension, however this results in errors

2 - VSCode has lsp user input sample, which has a registerProposedFeatures call and then LSP Client sends the didOpen, didChange & didClose requests automatically, However that doesn't work as well

https://github.com/microsoft/vscode-extension-samples/blob/a7e7e7f3a214c0fa3da1ff8d4e60104c06713787/lsp-user-input-sample/client/src/extension.ts#L43

3 - This extension also has a onLanguage in package.json here, Adding this also doesn't work

https://github.com/microsoft/vscode-extension-samples/blob/a7e7e7f3a214c0fa3da1ff8d4e60104c06713787/lsp-user-input-sample/package.json#L15C3-L15C25

[wakaztahir]

Okay I fixed it, The way you do it this :

onLanguage:plaintext means when the language is plaintext, so I wrote onLanguage:chemical, so use your own language in this one

I also used this as the document selector so vscode knows the language

{ pattern : "**/*.ch", scheme: 'file', language: 'chemical' }