Ideas from new user coming from logseq

[garyo]

Hi; I am super interested in this package! I've been an Emacs user since around 1983 but org, org-roam etc. never seemed to match my note taking style. I've been using logseq for the last couple of years and love it -- daily journal with easy tagging, nice formatting, search, etc. But of course I miss all the Emacs goodness: keybindings, macros, elisp. So EKG is really interesting to me!

After only a few minutes of use, here's what I've found:

• Needs full text search
• Would be great to have a transient menu (like magit) -- I can work on this if nobody else is
• How should we sync the db among multiple machines? I'll probably try syncthing. It's quite fast to update so I probably wouldn't see data lossage. (But of course I'll back up to git)
• Speaking of that, has anyone set up ekg with a git backup (exporting the db to text and committing)?
• How about auto-saving notes as I type, rather than C-c C-c which saves and closes?
• Feels like ekg creates too many windows; I'll have to look into how to reuse them.
• I like to add #inline #tags like that. Any possibility ekg can do that someday? (I know it brings up questions about whether tags are applied to the whole note or specific blocks)
• Lastly, transclusion is an amazing feature of logseq. You see the transcluded blocks just as if they were typed at the reference point, and can even edit them. That's a feature that org mode would be really hard pressed to add (overlays or something) but since ekg is backed by a db, it seems possible.

[ahyatt]

Thanks for the feedback!

For full text search, yes, I'm hoping that one day I'll get around to making this work with the triples library. However, right know ekg-embeddings does provide this, with great quality, but does require an API in OpenAI. I soon hope to have a more flexible way to get embeddings with my llm library, which I'm waiting on inclusion into GNU ELPA.

Transient menu - no one is working on this, if you want to work on it, please go ahead, I'd love for someone to help improve our user experience!

Syncing - I think syncthing is probably fine. It's a database, though, so using sqlite's dump command is probably the best way to do this.

I haven't seen anyone use git and ekg at the same time. It's maybe possible with the logseq export, and if I wanted to back up to a text repository, that seems a sensible way to do things.

For auto-saving notes, we don't have auto-save, but the develop branch does have drafts, so you can just save things manually as you go. Probably emacs has some horizontal autosave functionality that you could use for this, and it could be made to work with drafts.

Windows: yes, we could potentially have an option to re-use windows, like dired does. I haven't implemented that yet, but it seems sensible.

Inline tags: I've had the exact same thought, and maybe I'll implement it. The problem is that it's a bit weird - what if you remove the tag from the metadata, but the tag still exists in the text? I think we'd just have to ignore the text, but I'm not yet comfortable with that. I wouldn't want to use metadata + text tags, that just seems like a bad design and needlessly confusing.

We support transclusion now, please see the manual. It doesn't work like you are envisioning, mainly because unlike logseq we have a separation between a note's view for reading and view for editing. The read view has the correct tranclusion, but the edit view has the tranclusion command, since it is an inline command.

[smartmic]

For full text search, yes, I'm hoping that one day I'll get around to making this work with the triples library. However, right know ekg-embeddings does provide this, with great quality, but does require an API in OpenAI. I soon hope to have a more flexible way to get embeddings with my llm library, which I'm waiting on inclusion into GNU ELPA.

Just wondering, wouldn't SQLite3 FTS also be suitable for full text search or do you mean this by referencing the triples library?

[gour]

Syncing - I think syncthing is probably fine. It's a database, though, so using sqlite's dump command is probably the best way to do this.

I've a need to sync between my desktop and (old) netbook machine, so wonder if anyone has any experience using syncthing?

Dumping sqlite is probably more safe, but it involves dumping db regularly...

[garyo]

Thanks for the detailed response! I will work up a transient menu.

I'm not sure how to load ekg-embedding; it seems to have an additional dependency not listed in the main ekg.el file (request) and it's also not autoloaded or anything from ekg.el -- maybe it's still in development? I do have an OpenAPI key so once I get it loaded I should be able to try it out. If you have a use-package stanza to load it that would be helpful to get started.

As far as inline tags go, as a user I would expect inline tags to behave exactly the same as tags in the metadata area (until/unless you break up notes into individually-taggable blocks). I'd expect having #mytag in the text would be just like adding it to the metadata area. (Maybe you could even show all tags in the metadata area -- you could have a line for "inline tags" showing all the tags collected from the text or something...)

I'll check into the transclusion support! Sounds like it'll be very useful. (Block-level transclusion like logseq would be even cooler, but I understand ekg is note-level and blocks aren't a top-level concept.)

[ahyatt]

The dependencies are something I need to fix. However, I haven't decided how to do that yet. I could add the request library as a dependency to ekg, but most people using ekg aren't going to need it, since they won't be using ekg-embedding. Still, it might be the right thing to do.

Or, I can spin ekg-embedding out to it's own package on MELPA.

Sorry for the confusion here. I'd recommend for now just separately installing the request package however you normally install your packages.

[ahyatt]

The dependencies should be fixed with version 0.4.0, since we now depend on llm package, the llm package depends on the request package. There should be no more hidden dependencies.

[garyo]

Thanks; after a couple of tries that seems to work. I guess embedding search is fairly broad -- it returns a bunch of notes that don't contain my search terms, but it does seem to return the ones with my search terms at the top. Onward!

[ahyatt]

Glad it worked - I think the "couple of tries" is because it is doing things in the background and may need some time to do that.
Embedding search works by just comparing distances between query embeddings and note embeddings - so there's no notion of "match" or not. We always just show the top 20 notes (or was it 10?) in terms of distance.

[garyo]

That makes sense. The first time I got an error with stack trace, but I can't repro that anymore.
Maybe have an optional max distance cutoff rather than constant number? I don't know if embedding distances are in a consistent scale though. Just an idea.
Have you thought about "regular" sql full text search? (https://dev.mysql.com/doc/refman/8.0/en/fulltext-search.html) Probably a lot easier for users.

[ahyatt]

The full text search you link to is for mysql, not sqlite. Sqlite does have something as an add-on that needs to be compiled in, and I think it's built in to the sqlite that emacs uses, but I need to check. It also involves creating a new text index table with triggers to keep it updated. It's a bit complex, but it's something I'd like to look into at some point.

[garyo]

Ah, you're completely right, sorry about that.