Fix snapshot automount race causing AVL tree panic #17943
Conversation
amotin
left a comment
amotin
left a comment
There was a problem hiding this comment.
It makes me wonder why parallel mounts may somehow succeed, considering identical sequential ones are failing. But if this is a state of things, then I have no objections.
amotin
dismissed
their stale review
It seems multiple successful parallel mount calls result in multiple mounts, which is not right. It needs deeper investigation and possibly different solution.
ixhamza
force-pushed
the
SEE-365-avl-panic
branch
from
f2010f8 to
e8584c7
Compare
ixhamza
force-pushed
the
SEE-365-avl-panic
branch
5 times, most recently
from
82ab4cd to
074be7b
Compare
| * The snapshot may be manually mounted as many times as desired. | ||
| * Check if snapshot is already being mounted. If found, wait for | ||
| * pending mount to complete before returning success. | ||
| */ |
There was a problem hiding this comment.
How does this handle the case of the pending mount failing
There was a problem hiding this comment.
Auto-mounting of snapshots happens transparently on first access (e.g., ls), so I believe it makes sense for parallel processes racing for first access to get the same result as the first mount attempt. Updated the PR to store the mount error in se_mount_error and return it to all waiting processes for consistency.
Multiple threads racing to automount the same snapshot can both spawn mount helper processes that successfully complete, causing both parent threads to attempt AVL tree registration and triggering a VERIFY() panic in avl_add(). This occurs because the fsconfig/fsmount API lacks the serialization provided by traditional mount() via lock_mount(). The fix adds a per-entry mutex (se_mtx) to zfs_snapentry_t that serializes mount and unmount operations on the same snapshot. The first mount thread creates a pending entry with se_spa=NULL and holds se_mtx during the helper execution. Concurrent mounts find the pending entry and return success without spawning duplicate helpers. Unmount waits on se_mtx if a mount is pending, ensuring proper serialization. This allows different snapshots to mount in parallel while preventing the AVL panic. Signed-off-by: Ameer Hamza <ahamza@ixsystems.com>
ixhamza
force-pushed
the
SEE-365-avl-panic
branch
from
074be7b to
9b44934
Compare
4 participants
Motivation and Context
This fixes a race condition that causes kernel panics when multiple processes simultaneously access a fresh snapshot. The bug triggers a
VERIFY()assertion failure in the AVL tree code when concurrent threads attempt to add identical entries during snapshot automount.Description
The race condition occurs in
zfsctl_snapshot_mount()due to a time-of-check-time-of-use (TOCTOU) bug between checking if a snapshot is mounted and adding it to the AVL tree. The sequence is:zfsctl_snapshot_ismounted()- returns FALSEcall_usermodehelper())VERIFY()assertion failsThe fix adds a pending entry mechanism with per-entry mutex synchronization. The first mount thread creates a pending AVL entry and holds se_mtx during helper execution. Concurrent mounts find the pending entry and return success without spawning duplicate helpers, preventing the AVL panic.
Kernel Stack Trace:
How Has This Been Tested?
Reproduction script:
Results:
Types of changes
Checklist:
Signed-off-by.