Developing extensions

Newelle extensions are simple python files that can extend Newelle capabilities in these ways:

Editing history
Adding new handlers (for LLM, TTS, STT, Embedding, Long Term Memory, RAG and Web Search)
Adding new prompts
Replacing codeblocks with custom GTK widgets or text to be sent to the LLM (for example, mathematical results)
Showing custom tabs / mini applications

Developing an extension does not require deep knowledge of the project codebase or the GTK framework. Generally speaking, every extension is a python file that contains a class that extends NewelleExtension. This class is located in the same path as the project files, meaning that you can use any other function and class.

Getting Started

Every Newelle extension must have:

A name, that is the name displayed in the settings
An ID, a string that uniquely identifies the extension. Please check that it is not already taken form here

You can set ID and name as attributes of your Newelle extension class.

For example, this is a valid extension (that does nothing):

from .extensions import NewelleExtension

class MyCustomExtension(NewelleExtension):
    name = "Custom Extension"
    id = "customextension"

Useful variables

In every Newelle extension, you can use these variables:

self.pip_path  # Path to the pip directory, useful if you want to install new python packages
self.extension_path  # Path for cache, where you can put temporary files (shared with other extensions)
self.settings  # GIO settings of the application, this is generally not intended to be used
# For estensions settings use self.set_setting() and self.get_setting() like in handlers
self.ui_controller # UI Controller instance, explained better later
# Handlers
self.llm # LLM Handler
self.secondary_llm # Secondary LLM handler (can be none)
self.tts # TTS Handler
self.stt # STT Handler
self.rag # RAG Handler
self.memory # Memory Hanlder
self.websearch # Websearch Handler

Settings and Dependency management

Since NewelleExtension is a subclass of the Handler class, you can manage Extension settings and dependencies like you would do with normal handlers.

Very important: The settings about added LLM/TTS/STT... must not be set as extensions settings. Set them as settings in their handlers!

Note: The install method for extensions is always called when the extension is installed!

Example of extension using extra settings: Perchance Image Generator

Adding handlers

Extensions can add custom handlers using the methods

get_llm_handlers
get_tts_handlers
get_stt_handlers
get_rag_handlers
get_memory_handlers
get_embedding_handlers
get_websearch_handlers

For example, you can create an handler like this (assuming HyperbolicHanlder is an handler in the same file):

class MyCustomExtension(NewelleExtension):
...
    def get_llm_handlers(self) -> list[dict]:
        """
        Returns the list of LLM handlers

        Returns: 
            list: list of LLM handlers in this format
            {
                "key": "key of the handler",
                "title": "title of the handler",
                "description": "description of the handler",
                "class": LLMHanlder - The class of the handler,
            } 
        """
        return [{
        "key": "hyperbolic",
        "title": _("Hyperbolic API"),
        "description": _("Hyperbolic API"),
        "class": HyperbolicHandler,
    }]

The handler will appear in settings. The procedure is analog for TTS and STT handlers

Replacing codeblocks

If the llm generates a codeblock, for example

```mmd
x -> y
y -> x
\```

You can replace it with a custom GTK widget or with some text that will be sent to the LLM Let's say you want to replace every mmd codeblock. You can add mmd in the return of get_replace_codeblocks_langs

class MyCustomExtension(NewelleExtension):
    def get_replace_codeblocks_langs(self) -> list:
        return ["mmd"]

Replace codeblock with widget

Now if we want to replace that codeblock with a custom widget, we can override the get_gtk_widget method:

from gi.repository import Gtk, GdkPixbuf

   def get_gtk_widget(self, codeblock: str, lang: str) -> Gtk.Widget | None: 
        """
        Returns the GTK widget to be shown in the chat, optional
        NOTE: it is run every time the message is loaded in chat
        Args:
            codeblock: str: text in the codeblock generated by the llm
            lang: str: language of the codeblock

        Returns:
            Gtk.Widget: widget to be shown in the chat or None if not provided 
        """
        // Do what you need
        return Gtk.Image(...)

Replace codeblock with an Answer

You can also replace the codeblock with a result of an operation to send to the llm by overriding get_answer

   def get_answer(self, codeblock: str, lang: str) -> str | None:
        """
        Returns the answer to the codeblock

        Args:
            codeblock: str: text in the codeblock generated by the llm 
            lang: str: language of the codeblock 

        Returns:
            str: answer to the codeblock (will be given to the llm) or None if not provided
        """
        if lang == "calc"
            return "The result is: " + eval(codeblock)
        return None

Adding custom prompts

Extensions can add custom prompts in order to make the llm use their capabilities. To add a custom prompt you can override the get_additional_prompts method.

    def get_additional_prompts(self) -> list:
        """
        Returns the list of additional prompts

        Returns:
            list: list of additional prompts in this format
            {
                "key": "key of the prompt",
                "setting_name": "name of the settings that gets toggled",
                "title": "Title of the prompt to be shown in settings",
                "description": "Description of the prompt to be shown in settings",
                "editable": bool, If the user can edit the prompt
                "show_in_settings": bool If the prompt should be shown in the settings,
                "default": bool, default value of the setting
                "text": "Default Text of the prompt"
            }            
        """
        return [
           {
                "key": "mermaid",
                "setting_name": "mermaid",
                "title": "Show mermaid graphs",
                "description": "Allow the llm to show mermaid graphs",
                "editable": True,
                "show_in_settings": True,
                "default": True,
                "text": "You can use ```mmd\ngraph\n```\n to show a Mermaid graph"
            }
        ]

Editing history

Extensions can edit the current chat history. You can do it by implementing the methods:

preprocess_history(self, history: list, prompts : list) -> tuple[list, list]:
postprocess_history(self, history: list, bot_response: str) -> tuple[list, str]:

preprocess_history is called before the LLM, the whole chat history (in Newelle format) is given with the list of prompts. The function must return the edited history and prompts.

postprocess_history is called after the LLM response and before the message is redered, the whole chat history (excluding the bot response, in Newelle format) and the bot response is given as a string. You must return the updated history and the updated bot response.

Note: If you add or delete a message, the whole chat history will be reloaded. If you edit a message, only that message will be updated. Console messages and prompts get only updated in history but they don't have changes on the UI.

Building Mini Apps

Mini apps are extensions that create tabs the user can interact with. This requires GTK programming, but you can also show Web Interfaces using the BrowserWidget widget. Good examples of extensions that make use of this functionality are Newelle Calendar and AI Webnavigator.

Adding the mini app to the available tabs

In case you want your mini-app to be launched (also) manually via the new tab menu, you can add it as an entry overriding the add_tab_menu_entries. For example:

from .handlers import TabButtonDescription
...
    def add_tab_menu_entries(self) -> list:
        return [
            TabButtonDescription("Calendar tab", "month-symbolic", lambda x, y: self.open_calendar(x))
        ]

TabButtonDescription is defined as follows:

def TabButtonDescription(title: str, icon: GdkPixbuf.Pixbuf | str | Gtk.IconPaintable, callback):
    """Generate a "new tab button"

    Args:
        title: Title of the button
        icon: Icon of the button
        callback: Callback of the button
    """
    return title, icon, callback

Creating a custom tab

You can add a tab at any point of the code using this method:

tab = self.ui_controller.add_tab(widget)

Where widget is a GTK Widget. That returns an Adw.TabPage.

Tips and Tricks

LSP Autocomplete while coding and extension

To get LSP working correctly while coding, you have to:

Clone Newelle repo:

git clone https://github.com/qwersyk/Newelle

Create your extension file in src/
Write your extension there, the LSP should be able to recognize the right imports

Managing python dependencies

Checking if a module is installed

from .utility.pip import find_module

if find_module("numpy") is None:
    print("Module not found")
else:
    print("Module found")

Installing a module

from .utility.pip import find_module

class MyExtension(NewelleExtension):
   ...
   def install(self):
       install_module("numpy", self.pip_path)

Examples

The full code for these examples are in this repository.

DuckDuckGO extension - Adding an Handler

First of all, we create the base class and add the required metadata:

from .extensions import NewelleExtension

class DDGExtension(NewelleExtension):
    name = "DuckDuckGo"
    id = "ddg"

After that, we just need to override the get_llm_handlers method. DDGHandler was already programmed here. The code for the LLMHandler must be in the same file.

    def get_llm_handlers(self) -> list[dict]:
        return [
            {
                "key": "ddg",
                "title": "DuckDuckGo",
                "description": "DuckDuckGo AI chat, private and fast",
                "class": DDGHandler
            }
        ]

Pollinations Image Generation - Replacing codeblocks

We will now build an Extension that will replace any generate-image codeblock with a generated image by pollinations.ai.

We create the base extension class

from .extensions import NewelleExtension


class PollinationsExtension(NewelleExtension):
    name = "Pollinations Image Generator"
    id = "pollinationsimg"

We override get_replace_codeblocks_langs in order to be able to replace generate-image codeblocks

def get_replace_codeblocks_langs(self) -> list:
    return ["generate-image"]

We override get_additional_prompts in order to add a prompt that tells the AI that it can generate images using those codeblocks

    def get_additional_prompts(self) -> list:
        return [
            {
                "key": "generate-image",
                "setting_name": "generate-image",
                "title": "Generate Image",
                "description": "Generate images using Pollinations AI",
                "editable": True,
                "show_in_settings": True,
                "default": True,
                "text": "You can generate images using: \n```generate-image\nprompt\n```\nUse detailed prompts, with words separated by commas",
            }
        ]

This will make possible for the user to view and edit the prompt

Override the method get_gtk_widget to return the widget you want to replace the codeblock with.

   def get_gtk_widget(self, codeblock: str, lang: str) -> Gtk.Widget | None:
        from threading import Thread
        # Create the box that will be returned
        box = Gtk.Box()
        # Create a spinner while loading the image 
        spinner = Gtk.Spinner(spinning=True)
        # Add the spinner to the box
        box.append(spinner)
        # Create the image widget that will replace the spinner 
        image = Gtk.Image()
        image.set_size_request(400, 400)
        # Add the image to the box
        box.append(image)

        # Create the thread that will load the image in background
        thread = Thread(target=self.generate_image, args=(codeblock, image, spinner, box))
        # Start the thread
        thread.start()
        # Return the box
        return box

Then we add the necessary methods for image generation:

    def generate_image(self, codeblock, image: Gtk.Image, spinner: Gtk.Spinner, box: Gtk.Box):
        import urllib.request
        import urllib.parse
        # Create a pixbuf loader that will load the image
        pixbuf_loader = GdkPixbuf.PixbufLoader()
        pixbuf_loader.connect("area-prepared", self.on_area_prepared, spinner, image, box)
        # Generate the image and write it to the pixbuf loader
        try:
            url = "https://image.pollinations.ai/prompt/" + urllib.parse.quote(codeblock)
            with urllib.request.urlopen(url) as response:
                data = response.read()
                pixbuf_loader.write(data)
                pixbuf_loader.close()
        except Exception as e:
            print("Exception generating the image: " + str(e))

    def on_area_prepared(self, loader: GdkPixbuf.PixbufLoader, spinner: Gtk.Spinner, image: Gtk.Image, box: Gtk.Box):
        # Function runs when the image loaded. Remove the spinner and open the image
        image.set_from_pixbuf(loader.get_pixbuf())
        box.remove(spinner)
        box.append(image)

And this is the result:

ArchWiki - Replacing codeblocks with information for the LLM

Now we will create an extension that will allow the LLM to get information about Arch Linux wiki pages. We want that if the LLM wants to get information from the Arch wiki, it uses an arch-wiki codeblock with the search query.

We create the base class and add the required metadata:

class ArchWikiExtension(NewelleExtension):
    id = "archwiki"
    name = "Arch Wiki integration"

We override get_replace_codeblocks_langs to be able to replace arch-wiki codeblocks

    def get_replace_codeblocks_langs(self) -> list:
        return ["arch-wiki"]

We override get_additional_prompts and add a prompt to inform the LLM that he can do queries to the Arch Wiki

    def get_additional_prompts(self) -> list:
        return [
            {
                "key": "archwiki",
                "setting_name": "archwiki",
                "title": "Arch Wiki",
                "description": "Enable Arch Wiki integration",
                "editable": True,
                "show_in_settings": True,
                "default": False,
                "text": "Use \n```arch-wiki\nterm\n```\nto search on Arch Wiki\nThen do not provide any other information. The user will give you the content of the page"
            }
        ]

We override the method get_answer in order to replace the codeblock with the content of the arch wiki page

   def get_answer(self, codeblock: str, lang: str) -> str | None:
        import requests
        import markdownify
        # Search for pages similar to that query in the wiki
        r = requests.get("https://wiki.archlinux.org/api.php", params={"search": codeblock, "limit": 1, "format": "json", "action": "opensearch"})
        if r.status_code != 200:
            return "Error contacting Arch API"
        # Pick the page
        page = r.json()[1][0]
        # Pick the page name in order to get its content
        name = page.split("/")[-1]
        r = requests.get("https://wiki.archlinux.org/api.php", params={"action": "parse", "page": name, "format": "json"})
        if r.status_code != 200:
            return "Error contacting Arch API"
        # Convert the HTML in Markdown in order to make it more readable for the LLM
        html = r.json()["parse"]["text"]["*"]
        return markdownify.markdownify(html)

As you may have noticed, we used a library that is not shipped with Newelle by default. So we override the install method to install it with pip when the extension is installed.

from .utility.pip import install_module, find_module
from .extensions import NewelleExtension

class ArchWikiExtension(NewelleExtension):
    id = "archwiki"
    name = "Arch Wiki integration"

    def install(self):
        if find_module("markdownify", self.pip_path) is None:
            install_module("markdownify", self.pip_path)

TTS Speaker - Example of Mini App

We will now build a simple mini app that makes use of Newelle's TTS in order to speak text written by the user.

As always, first of all we define the base skeleton of the extension:

from .extensions import NewelleExtension

class TTSSpeaker(NewelleExtension):
    id = "tts_speaker"
    name="TTS Speaker"

We now add a menu entry to the add tab menu, in order to allow the user to open the mini app. Note that this is not required for all mini apps, as you can open them at any trigger (for example a codeblock or a user message). The title of the entry will be "TTS Speaker", the icon will be "audio-volume-high-symbolic", and when it's clicked, the function self.open_tts_tab (that we define next) will be called.

from .handlers import TabButtonDescription

    def add_tab_menu_entries(self) -> list:
        return [
            TabButtonDescription("TTS Speaker", "audio-volume-high-symbolic", lambda x,y : self.open_tts_tab(x))
        ]
...

3. Now we have to create the actual mini app and add it to the tab. 
We want to create a simple `MultilineEntry` on top of a button to trigger the speech.
We create the previously used method:

```python
from .ui.widgets import MultilineEntry
from gi.repository import Gtk, Gio
...
    def open_tts_tab(self, button):
        # Base content of the mini-app
        box = Gtk.Box(hexpand=True, vexpand=True, orientation=Gtk.Orientation.VERTICAL, halign=Gtk.Align.FILL, valign=Gtk.Align.CENTER)
        # Create a MultilineEntry to input text
        entry = MultilineEntry()
        entry.set_margin_end(10)
        entry.set_margin_start(10)
        entry.set_margin_top(10)
        entry.set_margin_bottom(10)
        entry.set_hexpand(True)
        # Add the entry to the box 
        box.append(entry)

        # Add the button to trigger the TTS
        button = Gtk.Button(css_classes=["suggested-action"], label="Speak")
        button.set_margin_end(10)
        button.set_margin_start(10)
        button.set_margin_top(10)
        button.set_margin_bottom(10)
        button.connect("clicked", self.speak, entry)
        box.append(button)
        
        # Key section of the code
        # Create a new tab with the content     
        tab = self.ui_controller.add_tab(box)
        # Set title and icon to the tab 
        tab.set_title("TTS Speaker")
        tab.set_icon(Gio.ThemedIcon(name="audio-volume-high-symbolic"))

We define the self.speak method in order to trigger the speech synthesis.

from .handlers import TabButtonDescription, ErrorSeverity
from threading import Thread 

    def speak(self, button, entry: MultilineEntry):
        text = entry.get_text()
        if self.tts is not None:
            # Start on another thread to not hang the UI
            Thread(target=self.tts.play_audio, args=(text,)).start()
        else:
            self.throw("TTS is not enabled", ErrorSeverity.ERROR)

This method:

Gets the text from the MultilineEntry
Checks that the tts handler is set
- If it's not set, shows a dialog explaining the error
- It it's set, runs play audio on another thread. Remember to always run blocking operations on another thread in order to no freeze the UI.

Also, remember to run UI operations always on the main thread, otherwise crashes may occour. In case you have to edit some things on the UI from another thread, run the changes like this:

def ui_change():
    Gtk..

GLib.idle_add(ui_change)

And here is your stunning mini app:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Developing extensions

Getting Started

Useful variables

Settings and Dependency management

Adding handlers

Replacing codeblocks

Replace codeblock with widget

Replace codeblock with an Answer

Adding custom prompts

Editing history

Building Mini Apps

Adding the mini app to the available tabs

Creating a custom tab

Tips and Tricks

LSP Autocomplete while coding and extension

Managing python dependencies

Checking if a module is installed

Installing a module

Examples

DuckDuckGO extension - Adding an Handler

Pollinations Image Generation - Replacing codeblocks

ArchWiki - Replacing codeblocks with information for the LLM

TTS Speaker - Example of Mini App

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally