Bead box indexing is very slow with no progress indication or clear documentation

[korenmiklos]

Disclaimer: Sonnet 4 wrote this based on my Asciinema session, apologies for language.

Summary: The bead box index command takes several minutes to complete even for a relatively small bead box (95 beads), with no progress indication, and unclear documentation about when indexing is required.

Steps to Reproduce:

1. Have a bead box with ~95 bead files
2. Run bead box index --box cat2
3. Observe the time taken and lack of progress feedback

Expected Behavior:

• Indexing should complete in a reasonable time for 95 files (under 30 seconds)
• Progress indication should be shown for long-running operations
• Clear documentation should explain when indexing is required and the difference between index and reindex

Actual Behavior:

• Indexing took approximately 108 seconds (from timestamp 1250.74314 to 1358.683695)
• No progress indication during the operation
• User was forced to interrupt previous indexing attempts due to the long duration
• No clear documentation about when indexing is critical vs optional

Additional Context:

• The help text shows both index and reindex commands but doesn't clearly explain the difference
• Users may not realize that input load failures could be resolved by running index
• The lack of progress feedback leads users to interrupt the process, thinking it has hung
• For a workflow-critical operation, this performance is problematic

Suggestions:

1. Add progress indicators (e.g., "Indexing... 45/95 files processed")
2. Optimize indexing performance
3. Document when indexing is required and the difference between index and reindex
4. Consider auto-indexing or better integration between input loading and indexing

Environment:

• Bead version: 0.9.1.dev1
• Platform: macOS
• Bead box size: ~95 files

[e3krisztian]

Indexing is an expensive operation, and requires reading into the archives one by one.
Nevertheless I am surprised by the time it takes to index them, as this is not that many files, so my guess is that

1. there are some archives, that takes a long time for zip to open
2. the box is on a network drive

Documentation: it is the command line help.

$ bead box -h
...
    index               Create or update box index for faster searches.
    reindex             Rebuild box index from scratch.

Thanks for the suggestions, but I can only add progress indicators (that is a good idea), the rest is either already there (index vs reindex documentation), or is unclear if it is possible at all (points 2 and 4).

[korenmiklos]

It should be clear to the user what index is doing. Do i have to invoke it manually? Why? When? When should i use reindex instead of index?

[e3krisztian]

index is an advanced feature still in development, it should speed up loading and updating beads that are in that box (edit, input load and input update operations).

index is creating the above index for all the beads in the box . This index operation can be slow, especially the first time, as it needs to process the zip files. Its intended use is to call it once, which enables it, and when the box receives/loses beads by manually adding/deleting files from it (i.e. adding not by bead save, which manages the index).

reindex is to be used, when the index becomes corrupted/unusable (may happen due to concurrent write access on a network drive).

---

The intention is: on personal machines index can be enabled.
On shared drives, the index needs some protection: it should be enabled in read-only mode (only one users should have write access), so concurrent users could not corrupt it. The owner with the write access should update the index, when some change is made to the box content.