Graceful error handling for corrupted shard files

[titusz]

Problem

If a .usearch shard file is corrupted or truncated, the usearch C++ layer may segfault rather than raising a clean Python exception. The atomic_write() pattern reduces this risk significantly, but doesn't eliminate it entirely (storage-level corruption, filesystem errors, interrupted OS-level writes).

Current behavior:

• _restore_shard() calls Index.metadata(str(path)) which returns None if metadata can't be read — this case is handled
• But if metadata reads fine and the HNSW graph structure is corrupted, Index.load() / Index.view() may hit out-of-bounds memory access in C++, causing a segfault
• _load_existing() raises RuntimeError if _restore_shard() returns None, but if C++ crashes before reaching Python error handling, the process dies silently

Proposal

Wrap shard restore/load operations in try/except at the Python level to catch C++ exceptions that make it through to Python:

def _restore_shard(self, path):
    try:
        meta = Index.metadata(str(path))
        if meta is None:
            return None
        idx = self._create_shard()
        idx.load(str(path))
        return idx
    except Exception as e:
        logger.warning(f"Failed to restore shard {path}: {e}")
        return None

Consider also:

• A dedicated CorruptedShardError exception type so consumers can catch it specifically
• Logging the corrupted shard path for diagnosis
• Optional on_corrupt="warn" / on_corrupt="raise" parameter to control behavior

Context

iscc-search uses LMDB as the source of truth and usearch indexes as derived acceleration structures. If a shard is corrupted, iscc-search can rebuild the derived index from LMDB — but only if it gets a clean Python exception instead of a segfault. Graceful error handling here enables automatic recovery rather than process death.

[titusz]

Additional finding from iscc-search integration (Session 2)

Confirmed this is a real consumer pain point. During the ShardedNphdIndex integration into iscc-search, I hit the non-segfault variant:

Scenario

A shard file exists on disk but contains garbage data (simulated file corruption). The ShardedNphdIndex constructor raises:

ValueError: End of file reached!

from usearch.index._index_dense_metadata_from_path → called during _load_existing() → called from __init__.

Impact on consumer code

Because the exception happens inside the constructor, the consumer cannot create a ShardedNphdIndex instance at all for that path — even just to rebuild it. The workaround is to shutil.rmtree() the entire shard directory before constructing a fresh instance:

def _rebuild_nphd_index(self, unit_type):
    shard_dir = self.path / unit_type
    if shard_dir.exists():
        shutil.rmtree(shard_dir)  # Must clean up before constructor
    nphd_index = ShardedNphdIndex(max_dim=self.max_dim, path=shard_dir)
    nphd_index.add(keys, vectors)
    nphd_index.save()

Suggestion

The _restore_shard() try/except proposed in this issue would fix this case too — the ValueError would be caught, the corrupt shard skipped, and the constructor would succeed with size=0. The consumer (iscc-search) would then detect the count mismatch and trigger a rebuild without needing the shutil.rmtree() workaround.