Creating handlers
Handlers are the classes that handle some of Newelle's features so that the user can use the best one that suits their needs. Extensions can also add handlers to Newelle.
Supported handlers:
-
LLMHandler: handle the chatbot responses -
TTSHandler: handle the Text to Speech -
STTHandler: handle the Speech Recognition -
WebSearchHandler: handle Web Searches -
RAGHandler: handle document reading -
MemoryHandler: handle long term memory -
EmbeddingHandler: handle text embedding (making sentences into vectors for RAG)
This is an UML diagram of the structure
classDiagram
Handler <|-- LLMHandler
Handler <|-- TTSHandler
Handler <|-- STTHandler
Handler <|-- NewelleExtension
Handler <|-- WebSearchHandler
Handler <|-- RAGHandler
Handler <|-- MemoryHandler
Handler <|-- EmbeddingHandler
RAGHandler "1" *-- "1" RAGIndex : builds / queries
class Handler{
+str key
+require_sandbox_escape()
+get_extra_settings()
+get_extra_requirements()
+install()
+is_installed()
+get_setting(key)
+set_setting(key, value)
+get_default_setting(key)
+settings_update()
}
class LLMHandler{
+list history
+list prompts
+str schema_key
+stream_enabled(): bool
+load_model(model): bool
+set_secondary_settings(secondary: bool)
+is_secondary(): bool
+supports_vision(): bool
+supports_video_vision(): bool
+get_supported_files(): list[str]
+get_models_list(): tuple
+get_selected_model(): str
+set_history(prompts, history)
+generate_text(prompt, history, system_prompt): str
+generate_text_stream(prompt, history, system_prompt, on_update, extra_args): str
+send_message(window, message): str
+send_message_stream(window, message, on_update, extra_args): str
+get_suggestions(request_prompt, amount): list[str]
+generate_chat_name(request_prompt): str | None
}
class TTSHandler{
+str key
+str schema_key
+tuple voices
-_play_lock: threading.Semaphore
-on_start: Callable
-on_stop: Callable
-play_process: multiprocessing.Process
+get_extra_settings(): list
+get_voices(): tuple
+voice_available(voice): bool
+save_audio(message, file) # Abstract
+play_audio(message)
+connect(signal, callback)
+playsound(path)
+stop()
+get_current_voice()
+set_voice(voice)
}
class STTHandler{
+str key
+str schema_key
+is_installed(): bool
+recognize_file(path): str | None # Abstract
}
class WebSearchHandler{
+str schema_key
+query(keywords: str): tuple[str, list] # Abstract
+supports_streaming_query(): bool
+query_streaming(keywords: str, add_website: Callable): tuple[str, list] # Abstract
}
class RAGIndex{
+list documents
+get_all_contexts(): list[str] # Abstract
+query(query:str): list[str] # Abstract
+insert(documents: list[str]) # Abstract
+remove(documents: list[str]) # Abstract
+get_documents(): list[str]
+get_index_size() # Abstract
+update_index(documents: list[str]) # Abstract
}
class RAGHandler{
+str key
+str schema_key
+str documents_path
+LLMHandler llm
+EmbeddingHandler embedding
+bool indexing
+set_handlers(llm: LLMHandler, embeddings: EmbeddingHandler)
+get_index_row()
+get_supported_files(): list # Abstract
+get_supported_files_reading(): list # Abstract
+load() # Abstract
+get_context(prompt:str, history: list[dict[str, str]]): list[str] # Abstract
+query_document(prompt: str, documents: list[str], chunk_size: int|None = None): list[str]
+build_index(documents: list[str], chunk_size: int|None = None): RAGIndex # Abstract
+index_exists(): bool # Abstract
+delete_index() # Abstract
+indexing_status(): float # Abstract
+create_index(button=None) # Abstract
+index_button_pressed(button=None)
}
class MemoryHandler{
+str key
+str schema_key
+int memory_size
+LLMHandler llm
+EmbeddingHandler embedding
+set_memory_size(length: int)
+set_handlers(llm: LLMHandler, embedding: EmbeddingHandler)
+get_context(prompt:str, history: list[dict[str, str]]): list[str] # Abstract
+register_response(bot_response:str, history:list[dict[str, str]]) # Abstract
+reset_memory() # Abstract
}
class EmbeddingHandler{
+str key
+str schema_key
+int dim
+load_model()
+get_embedding(text: list[str]): ndarray # Abstract
+get_embedding_size(): int
}
class NewelleExtension {
+ name
+ id
+ get_llm_handlers()
+ get_tts_handlers()
+ get_stt_handlers()
+ get_additional_prompts()
+ get_replace_codeblocks()
+ get_gtk_widget()
+ get_answer()
}
If you need extra pip libraries for the handler, you can specify them overriding the static methodget_extra_settings. The dependencies will be installed in a local pip path.
For example:
def get_extra_requirements() -> list:
return ["google-genai"] The default implementations of install() and is_installed() will install/check the dependencies specified there.
This only works if the pip dependencies have the same name of imported module.
If not, you can override the install and is_installed methods. This can be useful also in case the handler needs to install things that are not pip packages.
from .utility.pip import find_module, install_module
def is_installed(self) -> bool:
return True if find_module("whisper") is not None else False
def install(self):
print("Installing whisper...")
install_module("openai-whisper", self.pip_path)
import whisper
print("Whisper installed, installing tiny model...")
whisper.load_model("tiny")The install method will run when the button near the handler is clicked.
If is_installed returns true, the button disappears and the handler can be selected.
Every handler can have its own set of settings, dinamically built.
You can list the settings overriding the get_extra_settings() method.
For the available settings formats, you have to import the .handlers.ExtraSettings class
from .handlers import ExtraSettings
...
def get_extra_settings(self) -> list:
return [
ExtraSettings.EntrySetting("key","Title","Subtitle","default")
]All settings share the following common features:
-
key(str): The unique identifier for the setting, used for retrieval. -
title(str): The title displayed to the user for this setting. -
description(str): A detailed explanation of what the setting does, shown to the user. -
folder(str | None): If provided, a button will appear near the setting to open the specified folder. -
website(str | None): If provided, a button will appear near the setting to open the specified website. -
update_settings(bool): IfTrue, the settings will automatically update when this setting's value changes. -
refresh(Callable | None): If provided, a button will appear near the setting to execute the specified function when clicked. -
refresh_icon(str | None): The icon to be displayed on the refresh button.
In order to get the value of the settings, you can use the get_settings method.
get_setting("key") # Retrieves the settings value with key "key"
# If the value is not set, it searches it in the extra_settings
get_setting("key", False) # Retrieves the settings value with key "key"
# If the value is not set, returns None
get_setting("key", False, Object) # Retrieves the settings value with key "key"
# If the value is not set, returns NoneImportant: To avoid recursion, always use get_setting("key", False,..) in get_extra_settings or methods that are called in it.
You can set a setting with set_setting("key", value) even if the key is not listed in the get_extra_settings method.
Note: the get_extra_settings method is dynamic, you can decide to hide/show some settings. You can call self.settings_update() in any moment to update the settings UI.
ExtraSettings.EntrySetting("apikey", "API Key", "The API key to use", "", password=True),
ExtraSettings.EntrySetting("clear", "Clear property", "You can read the content in this", "default value"),EntrySetting can display an entry where it is possible to enter a line of text.
It supports the password property, which sets the field as password and allows it to be removed from exports without password.
ExtraSettings.MultilineEntrySetting("user_summary", "User Summary", "Current summary of the interactions with the assistant", ""),MultilineEntrySetting is like an EntrySetting but multiline. It is ideal for long texts. It does not support the password property.
ExtraSettings.ScaleSetting("amount", "Amount", "How much", 10, 0, 100, 0),ScaleSetting adds a slider where the user can pick a number in an interval. Important: The value is always given as a float. The full signature of the method is:
def ScaleSetting(key: str, title: str, description: str, default: float, min: float, max: float, round: int,
folder: str|None = None, website: str|None = None, update_settings: bool = False, refresh: Callable|None = None, refresh_icon: str|None = None) -> dict:Where:
-
min: the minimum value of the setting -
max: the maximum value of the setting -
round: the number of digits to round to
ExtraSettings.ComboSetting("model", "Model", "The model to use", (("Model 1", "model1"), ("Model 2", "model2")), "model2", refresh= lambda x : self.get_models()),ComboSetting adds a dropdown menu where you can select one of some entries. The values must be expressed as a tuple of tuples, containing as first element the "Display Name" of the item, and as second element the value of the item.
The setting will be set to the value of the item.
For example, if "Model 2" is selected, self.get_setting("model") will return "model2"
ExtraSettings.ToggleSetting("custom_model", "Custom Model", "Use a custom model", False),ToggleSetting displays a simple switch. The value returned is a bool.
ExtraSettings.ButtonSetting("update", "Update", "Update", lambda x: self.install(), "Update"),ButtonSetting displays a simple button that does some action when clicked. The value of the setting is not touched by the program. This is the signature of the method:
def ButtonSetting(key:str, title: str, description: str, callback: Callable, label: str|None = None, icon: str|None = None,
folder: str|None = None, website: str|None = None, update_settings: bool = False, refresh: Callable|None = None, refresh_icon: str|None = None) -> dict:Where:
-
callback: the function that will be executed when the button is clicked, takes the button as argument and runs in the UI thread -
label: if not None, the label of the button -
icon: if not None, the icon of the button, icon and label can't be set at the same time
The signature of the method is:
def DownloadSetting(key:str, title: str, description: str, is_installed: bool, callback: Callable, download_percentage: Callable, download_icon: str|None = None,
folder: str|None = None, website: str|None = None, update_settings: bool = False, refresh: Callable|None = None, refresh_icon: str|None = None) -> dict:Where:
-
is_installed: if True, the delete button will be shown -
callback: the function that will be executed when the download or delete button is clicked. Must download the file ON THE SAME THREAD -
download_percentage: the function that will be periodically executed to get the download percentage (float between 0.0 and 1.0) -
download_icon: if not None, the icon of the download button
DownloadSetting is used to allow the users to download elements (for example models) and keep track of the progress.
Important: call self.settings_update() after deleting a model.
You can find an example of DownloadSetting in the whispercpp handler
ExtraSettings.NestedSetting("nested", "Nested", "Nested", [
ExtraSettings.EntrySetting("apikey", "API Key", "The API key to use", "", password=True)
])NestedSetting is a setting that contains other settings in an expander. It can be used to group some settings. The value of the NestedSetting itself is not set by the user.
If you are building an extension, you can override the get_llm_handlers, get_tts_handlers,..., or the get_stt_handlers methods.
For example:
from .handlers import HandlerDescription
class MyCustomExtension(NewelleExtension):
...
def get_llm_handlers(self) -> list[dict]:
return [
HandlerDescription(
key="hyperbolic",
title="Hyperbolic API",
description="Use Hyperbolic.xyz API",
class=HyperbolicHandler,
website="https://hyperbolic.xyz"
)
]If you are contributing to Newelle, then you can specify the handlers in constants.py in the same way.
Some handlers might need the permission to run commands on the user PC, escaping the flatpak sandbox. In order to display a warning if the user does not have the necessary permissions, you can override the requires_sandbox_escape function:
@staticmethod
def requires_sandbox_escape() -> bool:
"""If the handler requires to run commands on the user host system"""
return TrueIn case something fails, handlers can show errors to the user.
from .handlers import ErrorSeverity
# Show a small toast at the bottom of the window
self.throw("Error text", ErrorSeverity.WARNING)
# Show an error dialog with configrmation
self.throw("Error text", ErrorSeverity.ERROR)