Add a language server support in stlite

[bdimitrijoski]

Hi there,

First off, big kudos for creating stlite! Your work is truly impressive. It provides a rock-solid and flexible foundation for running Streamlit apps directly in the browser.

At Cognite, we've been using a fork of this repo to inject authentication token, configure startup settings, and implement a language server on top of the existing kernel. The users on our platform can create and run Streamlit apps seamlessly.

To enhance the user experience, especially in dev mode, we've integrated the Monaco code editor. It communicates with stlite via postMessage to offer language server features like autocomplete and hover support.

We're hoping to move away from our fork and contribute our language server logic to the upstream repo, if that’s alright with you. 😊

I’ll outline the potential solution below, and if it looks good to you, I'd be happy to start working on a draft PR (only autocomplete part first, later we can add other features like hover support).

Modify the Kernel and Worker to load the libraries needed for the language server

To be able to provide a language server features, we used a third party package (jedi).
Jedi is a static analysis tool for Python that is already used in few IDEs/editors plugins, but it is also lightweight.

To load and use it in stlite, we’ll need to do some orchestration. This involves loading the Jedi wheel file and registering it in the kernel.

export class StliteKernel {
    // ...rest of the code
    constructor(options: StliteKernelOptions) {
        //... register the jedi wheel file in constructor
        wheels = {
            stliteServer: stliteServerWheelUrl,
            streamlit: streamlitWheelUrl,
            jedi: jediWheelUrl, // points to the local jedi.whl file
        };
    }
}

And modify the worker-runtime to inject the jedi wheel

/**
 * Imports the necessary python packages to enable language server features
 */
export const importLanguageServerPythonLibraries = async (
  pyodide: Pyodide.PyodideInterface,
  micropip: Pyodide.PyProxy
) => {
  try {
    await micropip.install.callKwargs(["jedi"], {
      keep_going: true,
    });
    await pyodide.runPythonAsync(`import jedi`);
    console.debug("Importing jedi Interpreter");
  } catch (err) {
    console.error("Error while importing jedi", err);
  }
};

async function loadPyodideAndPackages() {
    // load pyodide and packages code
    postProgressMessage("Importing Language Server");
    await importLanguageServerPythonLibraries(pyodide, micropip);

    postProgressMessage("Booting up the Streamlit server.");
    // ...rest of the code in loadPyodideAndPackages
}

Create new MessageIn and MessageOut types for autocomplete

export interface InMessageAutocomplete extends InMessageBase {
  type: "language-server:autocomplete";
  data: {
    code: string;
    currentLine: string;
    currentLineNumber: number;
    offset: number;
  };
}


export interface OutMessageLangugeServerAutocomplete {
  type: "language-server:autocomplete";
  data: {
    items: languages.CompletionItem[];
  };
}

Modify the kernel and worker to listen for the new events

Add the event listener in the worker-runtime

// ... rest of the code
 const onmessage = async (event: MessageEvent<InMessage>): Promise<void> => {
    switch (msg.type) {
        case "language-server:autocomplete": {
            // write a util function that will use the jedi languge server
            // and provide the code completion items
            const response = await handleAutoComplete(msg, pyodide);
            messagePort.postMessage(reply);
        }
        // .. remaining code
    }
 }

Add the method in the Kernel

class StliteKernel {

    private _processWorkerMessage(msg: OutMessage, port?: MessagePort): void {

        case "language-server:autocomplete": {
            messagePort.postMessage(msg); // already with the correct structure
        }
    }

}

From this point, you can utilize the language server and extend it with features like hover, validation, and more. You can then connect it to a UI such as a code editor.

I’m more than happy to help integrate it with the Monaco code editor in stlite. However, I’ll need to review the code and get familiar with where changes need to be made.

[whitphx]

Hi @bdimitrijoski, thank you!
All sounds great and I'm 100% positive on that. Of course no against from my side!
This feature must be useful for other Stlite apps as well including Stlite Sharing, and I really appreciate your intension to help the integration too.

I’ll outline the potential solution below, and if it looks good to you, I'd be happy to start working on a draft PR

So, yes, I'm looking forward to it!

[bdimitrijoski]

That's great @whitphx I will start working on a draft PR and hopefully have an initial version tomorrow 😃

[whitphx]

Thank you so much @bdimitrijoski for the amazing contributions for this feature!
They are merged and live on https://edit.share.stlite.net/