home-assistant-journal-asssistant

A multi-modal notebook processor for a Bullet Journal.

Overview

This is a Home Assistant custom component spanwed out the effort to explore using a Supernote e-ink tablet using a Bullet Journal plus LLM understanding to develop a personal assistant. This integration will:

• Take as input a media source path and a multi-modal conversation agent
• Process the media source with a set of dynamic prompts
• Expose a parsed RFC5545 JOURNAL as a calendar entity

See https://github.com/allenporter/supernote-llm

Development plan

• Initial dummy calendar
• Create monthly journal entity
• Create weekly journal entity
• Create daily journal entity (tasks?)
• Create other journal entries by date created
• Given a directory of notebook yaml contents:
  • Read the notebook file
  • Read through notes grouped by date
  • Check for journal entry on date
  • Create journal entry if it does not already exist
  • Add content to the journal entry
• Test out monthly review
• Test out monthly rapid log
• Test out weekly review
• Test out weekly rapid log
• Documentation
  • Update with a screenshot of current state
  • Diagram with the workflow involved
  • Explanation of LLM prompts and details on bullet journal formats
• Processing
  • conversation agent in config flow
  • Create output storage directory for yaml
  • Storage area for dynamic prompts
  • Multi-modal call that outputs yaml structured output
  • (re)process a media uri on demand
  • Structured output => Journal demo
• State management
  • Metadata store of media source uri sources
  • Compute hashes of media source content to determine changes
  • Persist journal in .ics
  • Merge existing journal contents with ingested processed media
  • etag support
• Visibility
  • Entities associated with a device service
  • Sensors exporting diagnostic information like last time processed
  • Sensor exporting # of journal entries
• Journal Entity
  • Update tests to use service calls for reads
  • Support for additions to a journal (calendar)
  • Support for writes to a journal (calendar)
  • Support for edits to a journal (calendar)
• Assistant API
  • Add journal entities to the prompt
  • Add vector search intent
  • Calendar search intent
• Vector search
  • create db at starts
  • Query database from tool end to end test
  • persistence of database
  • understand embedding functions
  • support deleting documents and removing from index
  • support more google models in config flow
  • support alternative embedding models
• support reload of config entries
• support removing config entries, cleaning up database
• make vetordb indexing asynchronous
• reload vector db after updating index
• sensor for # of raw yaml files in total
• sensor for # of pages for each notebook prefix
• sensor with processor counts of total # of processed (total_increasing?)
• monthly/quarterly reflections not showing up, check on details

Local Development

First time, prepare the environment:

$ export PYTHONPATH="${PYTHONPATH}:${PWD}"
$ hass -c /workspaces/config --script ensure_config

From then on run home assistant:

$ hass -c /workspaces/config

Running with multiple custom components

$ cd /workspaces/config
$ ln -s ../../home-assistant-supernote-cloud/custom_components/supernote_cloud custom_components/supernote_cloud
$ ln -s ../../home-assistant-journal-assistant/custom_components/journal_assistant custom_components/journal_assistant