Duplicated #+transclude line on file save

[jpt4]

Moving from: #177

Reporting a related issue: duplication of the #+transclude: line when saving a modified file.

To reproduce:

Introduce a transclusion: #+transclude:
Add the transclusion, such that the content is now visible in the file.
Modify the file somewhere else, outside of the transclusion.
Save the file.

Result:
The transclusion is removed (content disappears, transclude line remains), a duplicate #+transclude: line is placed above the original, and an error is reported to the minibuffer: Wrong type argument: number-or-marker-p, nil (This error also shows during other org-transclusion actions, like adding, but does not prevent the action from taking place.)

Software Versions:
Emacs 29.4
org-transclusion 1.4.0

This is new behavior, but I do not know from exactly when. Unfortunately, this makes org-transclusion incompatible with my workflow, though I do endorse the project overall.

Thank you in advance for your attention to this matter.

Trace:

Debugger entered--Lisp error: (wrong-type-argument number-or-marker-p nil)
  org-transclusion-keyword-remove()
  org-transclusion-add-payload((:tc-type "org-id" :src-content #("#+begin_20220607Z\nAn alternative index..." 18 202 (:parent (paragraph (:standard-properties ...) #("An alternative indexing scheme for the..." 0 185 ...))) 204 582 (:parent (paragraph (:standard-properties ...) #("Xenotation applies the implexion opera..." 0 378 ...) (superscript ... ...) #("(Z) of the Naturals will generate\nall ..." 0 133 ...))) 583 584 (:parent (superscript (:standard-properties ... :use-brackets-p nil) #("n" 0 1 ...))) 584 716 (:parent (paragraph (:standard-properties ...) #("Xenotation applies the implexion opera..." 0 378 ...) (superscript ... ...) #("(Z) of the Naturals will generate\nall ..." 0 133 ...))) 718 733 (:parent (paragraph (:standard-properties ...) #("C(x) = x/2,  x " 0 15 ...) (verbatim ...) #("1 (mod 2)\n" 0 10 ...))) 764 773 (:parent (paragraph (:standard-properties ...) #("C(x) = x/2,  x " 0 15 ...) (verbatim ...) #("1 (mod 2)\n" 0 10 ...))) 775 843 (:parent (paragraph (:standard-properties ...) #("The Collatz Conjecture states that, gi..." 0 68 ...) (superscript ... ...) #("(x), for all x, there exists a k such ..." 0 44 ...) (superscript ... ...) #("(x) = 1. Furthermore,\nfor all j > k, C" 0 38 ...) (superscript ... ...) #("(x) = {1, 4, 2}, the convergent cycle ..." 0 58 ...))) 844 845 (:parent (superscript (:standard-properties ... :use-brackets-p nil) #("k" 0 1 ...))) 845 889 (:parent (paragraph (:standard-properties ...) #("The Collatz Conjecture states that, gi..." 0 68 ...) (superscript ... ...) #("(x), for all x, there exists a k such ..." 0 44 ...) (superscript ... ...) #("(x) = 1. Furthermore,\nfor all j > k, C" 0 38 ...) (superscript ... ...) #("(x) = {1, 4, 2}, the convergent cycle ..." 0 58 ...))) 890 891 (:parent (superscript (:standard-properties ... :use-brackets-p nil) #("k" 0 1 ...))) 891 929 (:parent (paragraph (:standard-properties ...) #("The Collatz Conjecture states that, gi..." 0 68 ...) (superscript ... ...) #("(x), for all x, there exists a k such ..." 0 44 ...) (superscript ... ...) #("(x) = 1. Furthermore,\nfor all j > k, C" 0 38 ...) (superscript ... ...) #("(x) = {1, 4, 2}, the convergent cycle ..." 0 58 ...))) 930 931 (:parent (superscript (:standard-properties ... :use-brackets-p nil) #("j" 0 1 ...))) ...) :src-buf #<buffer generalized_xenotation.org> :src-beg 1 :src-end 1243) (link (:standard-properties [1 nil 44 66 68 0 nil nil nil nil nil nil nil nil #<killed buffer> nil nil (paragraph (:standard-properties ...))] :type "id" :type-explicit-p t :path "76b3dba9-7aef-4e6f-8e44-dd48c1929582" :format bracket :raw-link "id:76b3dba9-7aef-4e6f-8e44-dd48c192958..." :application nil :search-option nil)) (:link "[[id:76b3dba9-7aef-4e6f-8e44-dd48c1929..." :exclude-elements "drawer keyword" :expand-links ":expand-links" :current-indentation 0) nil)
  org-transclusion-add()
  org-transclusion-after-save-buffer()
  run-hooks(after-save-hook)
  basic-save-buffer(t)
  save-buffer(1)
  funcall-interactively(save-buffer 1)
  command-execute(save-buffer)

@akashpal-21

[akashpal-21]

From what I can understand until now -

org-transclusion-keyword-remove is responsible for removing the #+transclude: keyword after the content is yanked from the target location. This function did not undergo a change from my version to yours, but the function is rather simple -

The problem is some variable is being set to nil - but this shouldn't be the case, we do not know what variable is being set to nil or what is responsible for it. The first course of action should be to figure out what variable exactly is misbehaving like this

(defun org-transclusion-keyword-remove ()
  "Remove the keyword element at point.
Returns t if successful.  It checks if the element at point is a
keyword.  If not, returns nil."
  (let* ((elm (org-element-at-point))
         ;; Ignore and keep affiliated keywords before #+transclusion
         ;; Fix issue #115
         (beg (org-element-property :post-affiliated elm))
         (end (org-element-property :end elm))
         (post-blank (org-element-property :post-blank elm)))
    
    (print elm)
    (print beg)
    (print end)
    (print post-blank)

    (when (string= "keyword" (org-element-type elm))
      (delete-region beg (- end post-blank))
      t)))

Try this function - it would report on the value of the variables.
For me it reports the following without any error

(keyword (:key "TRANSCLUDE" :value "[[file:source]]" :begin 43 :end 75 :post-blank 2 :post-affiliated 43 :mode property-drawer :granularity element :cached t :parent (section (:begin 1 :end 116 :contents-begin 1 :contents-end 113 :robust-begin 1 :robust-end 111 :post-blank 3 :post-affiliated 1 :mode first-section :granularity element :cached t :parent (org-data (:begin 1 :contents-begin 1 :contents-end 113 :end 116 :robust-begin 3 :robust-end 111 :post-blank 3 :post-affiliated 1 :path "/home/akash/Desktop/test1/target.org" :mode org-data :CATEGORY "target" :cached t))))))

43

75

2

[jpt4]

To confirm, add the above function in e.g., my init.el, or elsewhere?

[akashpal-21]

Just evaluate it in a scratch buffer.
C-x C-e

no need to add it to your init.el and make it permanent - we are just debugging.

[jpt4]

eval-last-sexp in scratch prints org-transclusion-keyword-remove in the minibuffer, but no other values.

[akashpal-21]

@jpt4 That is correct. We have merely evaluated the new definition. When the function will run in the normal course of action it will print the debug messages we implanted.

Try to trigger the error like before after you have evaluated our debug function.

[akashpal-21]

@jpt4

There is something more simple you can do -
when your cursor is on the #+transclude: keyword - you may do (eval-expression)

M-:

    Read a single Emacs Lisp expression in the minibuffer, evaluate it, and print the value in the echo area (eval-expression). 

Then simply run

(print (org-element-at-point))

It gives me an output such as this

(keyword (:key "TRANSCLUDE" :value "[[file:source]]" :begin 43 :end 76 :post-blank 3 :post-affiliated 43 :mode property-drawer :granularity element :cached t :parent (section (:begin 1 :end 76 :contents-begin 1 :contents-end 73 :robust-begin 1 :robust-end 71 :post-blank 3 :post-affiliated 1 :mode first-section :granularity element :cached t :parent (org-data (:begin 1 :contents-begin 1 :contents-end 73 :end 76 :robust-begin 3 :robust-end 71 :post-blank 3 :post-affiliated 1 :path "/home/akash/Desktop/test1/target.org" :mode org-data :CATEGORY "target" :cached t))))))

As you can see so long as we have

:begin 43 :end 76 :post-blank 3

The code will execute properly - my hunch is one of them is returning nil for some reason.

You do not need to run any org-transclusion function to get this, this is supplied by org naturally.

[jpt4]

Thank you, attempting to reproduce the error results in:

(keyword (:standard-properties [8124 8124 nil nil 8236 0 nil nil element t nil nil nil nil #<buffer codex.org> nil nil (special-block (:standard-properties [945 945 957 174898 174909 1 nil nil element t nil nil nil nil #<buff\
er codex.org> nil nil (section (:standard-properties [1 1 1 238729 238729 0 nil first-section element t nil 1 238729 nil #<buffer codex.org> nil nil (org-data (:standard-properties [1 1 1 238729 238729 0 nil org-data nil t ni\
l 88 238729 nil #<buffer codex.org> [org-element-deferred org-element--get-global-node-properties nil t] nil nil] :path "/home/jpt4/codex/codex.org"))]))] :type "Log" :parameters nil))] :key "TRANSCLUDE" :value "[[id:5df5649f\
-23a4-4962-91bd-adb1d61d36b3][Log]] :exclude-elements \"drawer keyword\" :expand-links"))

8124

8236

0

[akashpal-21]

Well - if this is the output then it shouldn't had thrown the error about something being nil-

(:standard-properties [8124 8124 nil nil 8236 0 nil nil element

The first element is the beginning position
the fifth element is the end position
the sixth element is the post-blank (number of empty lines after the keyword)

Everything seems normal here - there is no error here.

[akashpal-21]

@nobiot can you comment on this when you have some time? Thanks.

[nobiot]

Can you let me know your org-version, please?

[nobiot]

With the following, I cannot reproduce the issue.

Introduce a transclusion: #+transclude:
Add the transclusion, such that the content is now visible in the file.
Modify the file somewhere else, outside of the transclusion.
Save the file.

This works as expected and as before. No duplicate #+transclude line.

emacs-version: 29.3
org-version: 9.6.15
org-transclusion: 1.4 with the latest commit db09159

[nobiot]

I have just tried with what I believe to be the latest stable version of Org -- 9.7.11. I cannot reproduce the issue.

[akashpal-21]

@nobiot Since the remove keyword function is so simple - it only queries the org-element and uses the information to delete some words - I believe for OP there is some problem with org-element cache, in so far, nothing in org-transclusion can cause the error as reported.

@jpt4 Try to reset the org-element-cache from your end - the function in question is extremely simple, it neither references any other external function nor is dependent on any other functionality than what is presented to us by org natively.

If you face the issue again, then the debug function I provided should output atleast one of them as nil.

[akashpal-21]

@jpt4 @nobiot

Today I had the most craziest experience - I had the same problem but in another function that uses the org-element functions. I will cut the chase and give the solution directly then explain

DELETE .emacs.d/eln-cache

Delete this folder and let emacs recompile. This problem is being caused due to stale caches!!!

I was trying to get the newest version of org pre-9.8 and everytime I tried to run a function it would give me this error!!

AND THE WEIRDEST PART IS THAT THE MOMENT I REEVALUATED THE FUNCTION THE PROBLEM WENT AWAY!
THERE IS NO WAY WE CAN DIAGNOSE THE PROBLEM IN AN APRIORI MANNER!!!

Please report back if the problem goes away when you delete this folder!!

[nobiot]

I think the root issue is as described here: org-roam/org-roam#2361 (comment)

As a fix, after upgrading Org, you'd need to recompile Org and all other libraries that use org-element (this includes Org-transclusion, Org-roam, etc.). To do this, you can run (package-recompile-all).

This should be another way of what @akashpal-21 did (deleting the eln directory).

Here is the key detail:

(org-element-property :begin link)) still works. But you may have to make sure that the inlined functions are re-compiled. After upgrading Org mode, all the users (including org-roam) of Org element API must be re-compiled.

[jpt4]

@nobiot, my package versions are as follows:
emacs-org-transclusion 1.4.0
emacs-org-roam 2.2.2-1.0b9fcbc
emacs-org 9.7.11

@akashpal-21, I have deleted .emacs.d/eln-cache, with no change in behavior. Upon saving, the #+transclude line is still duplicated. Also, I have noticed another incorrect behavior, which may have been present before: while org-transclusion-add works (with the type argument error), org-transclusion-remove has no effect, and the only way to remove transcluded content is by modifying and saving the file (which in turn still produces an additional #+transclude line).

#257 (comment)
@nobiot I am using GuixSD; (package-recompile-all) reports:

https://github.com/nobiot/org-transclusion/issues/257#issuecomment-2378939275

Thank you both for your assistance thus far.

[akashpal-21]

Did you let emacs recompile the packages after you deleted the folder before trying? You'd have to restart the emacs daemon if you were using that..
I cannot reproduce any of the errors with:
Org mode version 9.8-pre (release_9.7.11-145-g28b631

As a last resort - I can only advice to reinstall everything from scratch - just take your init file and any custom packages you may have and just install everything fresh.

There is no bugs in the function definitions, otherwise we could have recreated the errors ourselves too.

Please try to restart the emacs daemon before trying if you hadn't done it earlier.

[nobiot]

(package-recompile-all)
reports:

https://github.com/nobiot/org-transclusion/issues/257#issuecomment-2378939275

What does this mean? You run (package-recompile-all) and you get the message about GitHub? This cannot be a correct message for package-recompile-all. There must be something else going on. Can you try Org-transclusion with emacs -Q? (Does this make sense to you?).

I agree with @akashpal-21, saying this:

There is no bugs in the function definitions, otherwise we could have recreated the errors ourselves too.

What we have seen so far is there is something wrong with your environment that causes the issue. Our hypothesis is that your compiled code (org and org-transclusion) is incorrect. So you need to re-compile the code. I don't know if this is possible with GuixSD or if Guix is interfering. I have no idea how it works.

It is not the code of Org-transclusion as neither of us can reproduce the issue. So there will be no "fix" to the code.

[nobiot]

Just in case, another point.

I have deleted .emacs.d/eln-cache, with no change in behavior.

After deleting it, you need to restart Emacs, so that Emacs is not running the compiled code that you have deleted. Or if you are running a daemon (or emacs-server), you need to restart them as @akashpal-21 is saying.