Exporting Ekg to Denote

[jayrajput]

I have started working on exporting ekg notes to denote following the logseq export code. One challenge I am facing is that denote uses the timestamp of the note creation to create a unique ID for the notes whereas ekg database can have two notes with the same creation time if the notes were exported from org-roam. I can see duplicate creation time in my notes and looking for solution for the same.

Duplicates found:
String '20230411T164454' appears 2 times.
String '20230411T164450' appears 5 times.
String '20230411T164449' appears 4 times.
String '20230411T164451' appears 2 times.
String '20230506T114207' appears 3 times.
String '20230506T114206' appears 2 times.
String '20230506T114205' appears 14 times.
String '20230506T114204' appears 12 times.
String '20230506T114203' appears 16 times.
String '20230506T114202' appears 17 times.
String '20230506T114201' appears 18 times.
String '20230506T114200' appears 18 times.
String '20230506T114159' appears 13 times.
String '20230411T164453' appears 2 times.
String '20230411T164448' appears 5 times.
String '20230411T164447' appears 5 times.
String '20230411T164445' appears 2 times.
String '20230411T164443' appears 4 times.

Corresponding code

  (defun find-duplicates (lst)
    "Find duplicates in a list and generate a report with counts."
    (let ((count-map (make-hash-table :test 'equal))
	  (duplicates '()))
      (dolist (item lst)
	(let ((count (gethash item count-map 0)))
	  (puthash item (1+ count) count-map)))
      (maphash (lambda (key value)
		 (when (> value 1)
		   (push (format "String '%s' appears %d times." key value) duplicates)))
	       count-map)
      (when duplicates
	(message "Duplicates found:\n%s" (mapconcat 'identity duplicates "\n")))))
  
  (find-duplicates
   (mapcar (lambda (date)
	     (format-time-string denote-id-format date))
	   (mapcar #'ekg-note-creation-time
		   (mapcar #'ekg-get-note-with-id
			   (ekg-denote-notes-modified-since 0)))) )

[ahyatt]

Interesting issue! I can't think of an elegant solution, but what you could do is to find all duplicates, like you are doing now, and, with the duplicates, add +1 on the first duplicate to the timestamp and save, +2, on the second duplicate to the timestamp and save, etc. I think it'd make sense to just ensure the existing duplicates are fixed as a one-time activity, I'm guessing it'd be hard to create new duplicates.

[jayrajput]

Thanks. I will go with the +1 thing. There are few corner cases, but I shall be able to code something and then we can discuss during the code review. Mostly I want to use this discussion for the design discussion and the design idea is to make it a one time activity to fix all the duplicates. For now, I am adding checks in the ekg-denote-export to flag duplicates. Technically denote notes can be created with duplicate creation time, but I believe that will be confusing in longer run.

[ahyatt]

Sounds good. This is something you might want to bring up with the denote developers as well, they may have an interesting viewpoint.

[jayrajput]

@ahyatt . Help here to debug why the denote/last-export row is getting erased from the ekg db on try to re-run ekg-denote-connect. Code is just a copy/paste of ekg-logseq export . Here is the code

  (defun ekg-denote-connect ()
    "Connect to ekg and ensure denote schema is set up."
    (ekg-connect)
    (ekg-denote-add-schema))
  (defun ekg-denote-add-schema ()
    "Add the denote schema to the ekg database."
    (triples-add-schema ekg-db 'denote
			'(last-export :base/unique t :base/type integer)
			'(last-import :base/unique t :base/type integer)))
  (defun ekg-denote-get-last-export ()
    "Get the last export time."
    (or (plist-get
	 (triples-get-type ekg-db 'denote 'denote)
	 :last-export) 0))
  (defun ekg-denote-get-last-import ()
    "Get the last export time."
    (or (plist-get
	 (triples-get-type ekg-db 'denote 'denote)
	 :last-import) 0))
  (defun ekg-denote-set-last-export (time)
    "Set the last export time to TIME."
    (let ((plist (triples-get-type ekg-db 'denote 'denote)))
      (apply #'triples-set-type ekg-db 'denote 'denote
	     (plist-put plist :last-export (floor (float-time time))))))
  (defun ekg-denote-set-last-import (time)
    "Set the last import time to TIME."
    (let ((plist (triples-get-type ekg-db 'denote 'denote)))
      (apply #'triples-set-type ekg-db 'denote 'denote
	     (plist-put plist :last-import (floor (float-time time))))))

Here is a test run in the scratch pad showing that the ekg-denote-connect erase the denote/last-export entry

(ekg-denote-connect)
((denote base/type schema) (denote schema/property last-export) (denote/last-export base/unique t) (denote/last-export base/type integer) (denote schema/property last-import) (denote/last-import base/unique t) (denote/last-import base/type integer))

(ekg-denote-set-last-export (current-time))
((denote base/type denote) (denote denote/last-export 1686314284))

;; value can be seen as updated
(ekg-denote-get-last-export)
1686314284

(ekg-denote-connect)
((denote base/type schema) (denote schema/property last-export) (denote/last-export base/unique t) (denote/last-export base/type integer) (denote schema/property last-import) (denote/last-import base/unique t) (denote/last-import base/type integer))

;; value becomes zero
(ekg-denote-get-last-export)
0

[ahyatt]

The problem seems to be something that should affect logseq too, which is that when we add the schema, it overwrites anything anything stored in the subject. I can reproduce this via ert test in the triples library. I think this requires a fix in that library, so I'll fix this. Thanks for bringing this to my attention!

[ahyatt]

I've checked in a fix for this in the latest triples develop branch, PTAL and let me know if that fixes your problem.

[jayrajput]

Verified. Working. Thanks!!