Enhance interactive 3D viz: overlays, server-side rendering, fast playback

[sbryngelson]

Summary

• New features: contour overlays (2D/3D), isosurface/isovolume mixing, solid color isosurfaces, opacity control, improved timestep slider
• Server-side rendering: kaleido renders Plotly figures to JPEG on Linux for fast 3D playback over SSH tunnels (bypasses JSON serialization + browser WebGL round-trip)
• Playback performance: aggressive data prefetch (3 workers, 15 steps ahead, 40-entry cache), 3D mesh prefetch (3 workers, 50K cell budget), forced Dash patch path during playback — cached frames render in ~0.002s

Dependencies

• Add kaleido for server-side Plotly image export (Linux SSH playback)
• Remove pyvista (replaced by kaleido — simpler, identical visual output)

Test plan

• ./mfc.sh precheck -j 8 passes
• 2D interactive viz (./mfc.sh viz <2d_dir> -i) works as before
• 3D isosurface/volume playback on macOS (local browser, no kaleido)
• 3D isosurface/volume playback on Linux over SSH tunnel (kaleido path)
• Contour overlays render correctly in 2D and 3D slice modes
• Overlay variable mixing (isosurface + isovolume) works in 3D

🤖 Generated with Claude Code

[github-actions]

(review resolved)

[github-actions]

(review resolved)

[github-actions]

(review resolved)

[github-actions]

Claude Code Review

Head SHA: d46ca53

Files changed (3):

• toolchain/mfc/viz/_step_cache.py
• toolchain/mfc/viz/interactive.py
• toolchain/pyproject.toml

---

Summary

• Adds contour overlays (2D/3D), solid-color/opacity isosurface controls, and isosurface/isovolume overlay mixing for 3D
• Server-side rendering via kaleido on Linux for fast 3D playback over SSH tunnels
• Aggressive 3D mesh prefetch cache (40-entry, 3-worker pool) with a coarse/fine resolution split during playback
• Timestep dropdown replaced with a slider + hidden dcc.Store that holds the actual step value
• _step_cache cache doubled (20→40) and prefetch pool tripled (1→3 workers)

---

Findings

1. kaleido as an unconditional hard dependency — toolchain/pyproject.toml line +6

"kaleido",

The code already gracefully handles missing kaleido via _kaleido_available() and a full fallback path. Making it a mandatory install forces all users (macOS, Windows, clusters without network access) to pull in kaleido, which embeds Chromium/V8 and has known platform-specific install failures. Recommend making it optional, e.g. conditional on platform or as a [project.optional-dependencies] extra:

"kaleido; sys_platform == 'linux'",

or just removing it from pyproject.toml entirely since the runtime probe already handles the absent case.

2. Wrong transform applied to overlay variable in 3D isosurface overlay — interactive.py around diff line 1246

_ov_vx, _ov_vy, _ov_vz, _ov_fi, _ov_fj, _ov_fk, _ov_int =     _compute_isomesh(
        _ov_ds, _ox, _oy, _oz,
        _tf, _ov_ilo, _ov_ihi,   # ← primary variable's log/linear transform
        ...
    )

_tf is derived from the primary variable's log-scale checkbox. The overlay variable has no independent log control, so if log mode is active and the overlay variable contains zeros or negative values, _tf will produce nan/-inf, marching cubes will fail silently (returning an empty mesh), and the overlay will disappear without any error message. The thresholds _ov_ilo/_ov_ihi are computed from _tf(_ov_3d_raw) which compounds the problem — if _tf maps half the overlay values to nan, nanmin/nanmax may produce a range that no actual values fall within. Either apply an identity transform for overlay rendering, or compute the overlay's own range before applying _tf.

3. playing-st.data as Input triggers spurious full re-renders — interactive.py around diff line 770

Input('playing-st', 'data'),
...
def _update(..., playing_st):  # pylint: disable=unused-argument

playing-st.data is listed as an Input but the parameter is explicitly marked unused-argument — the actual play state is read from the closure variable _is_playing[0]. Every play/pause toggle fires _update with the current (unchanged) step, forcing a full figure rebuild just to switch between play and pause states. This also means step-sel.data fires at the same time as playing-st.data, causing _trig3 to fail the issubset check and fall through to the slower full-render path on the very first playback frame. Consider using State instead of Input here (or removing it, since the closure already provides the value).

---

Minor / Improvement Opportunities

4. No version constraint on kaleido — kaleido had a disruptive 0.2.x series. Without a pin (kaleido>=0.1,<1.0 or similar) a future release could silently break the export interface.

5. _compute_contour_traces is not subsampled for large 2D fields — the main render path subsamples arrays > 200 K cells for the color-range computation, but contour computation uses the full-resolution data_2d. For a 1024×1024 field with 20 contour levels this calls find_contours 20 times on the full array, adding noticeable latency on each step during playback. A simple stride subsample (matching the pattern already used in _prefetch_3d_mesh) would help.

6. _mesh3_cache evicts by insertion order, not LRU — for forward playback this is functionally correct (oldest entry = already played), but if a user scrubs backward, the evicted entries are exactly the ones most likely to be needed again. A simple dict move-to-end on hit (or collections.OrderedDict) would make it a true LRU with no algorithmic cost.

[github-actions]

Claude Code Review

Head SHA: 7652c9f92da809ceb603b07e9cbe43decc132210

Files changed: 3

• toolchain/mfc/viz/_step_cache.py (+2/-2)
• toolchain/mfc/viz/interactive.py (+961/-82)
• toolchain/pyproject.toml (+1/0)

---

Summary

• Adds 2D and 3D contour overlays via skimage.find_contours, rendered as go.Scatter/go.Scatter3d traces on top of heatmaps and slices.
• Adds a second-variable isosurface/isovolume overlay in 3D modes with opacity and color-by-value controls.
• Adds solid-color isosurfaces with configurable color and opacity for the primary variable.
• Introduces a 3D mesh prefetch cache (_mesh3_cache) mirroring the existing JPEG prefetch system, with 3 workers and a 50K-cell budget for background marching-cubes.
• Adds server-side Plotly→JPEG rendering via kaleido on Linux during 3D playback, bypassing the SSH→browser WebGL round-trip. Adaptive frame-gap throttling (_min_frame_gap) prevents the dcc.Interval from outpacing the browser.
• Replaces the timestep Dropdown with a Slider + hidden dcc.Store, and removes the tooltip CSS that was no longer needed.

---

Findings

1. kaleido as a hard dependency (toolchain/pyproject.toml, line 52)

kaleido is unconditionally added to dependencies, but it is only used on Linux and the code already gracefully degrades when it is absent. This forces an unnecessary install on macOS and Windows.

Consider using a platform marker or optional extras:

"kaleido; sys_platform == 'linux'",

or move to [project.optional-dependencies]. As written, CI on non-Linux systems will pull and build kaleido for no benefit.

---

2. Kaleido disabled permanently on any render failure (interactive.py, ~line 985-986)

except Exception as _kal_exc:
    _KALEIDO_OK = False

A transient failure (OOM on a very large mesh, a one-off encoding error) permanently disables kaleido for the entire session. Only import failures (ImportError) are truly permanent. Consider resetting _KALEIDO_OK to True for non-import exceptions, or using a per-frame fallback without poisoning the flag.

---

3. _has_overlay declared twice in the same function body (interactive.py, ~lines 1027 and 1317)

# line ~1027 (inside 3D block)
_has_overlay_3d = overlay_var and overlay_var != '__none__'
...
# line ~1317 (inside 2D block, same callback)
_has_overlay = overlay_var and overlay_var != '__none__'

Two different names compute the identical expression. This is harmless but could lead to accidental divergence if conditions differ in future edits. Unifying to one variable at the top of _update would be cleaner.

---

4. _tf (primary variable transform) applied to overlay variable range (interactive.py, ~line 1229)

_ov_tf = _tf(_ov_3d_raw)
_ov_vmin = float(np.nanmin(_ov_tf))
_ov_vmax = float(np.nanmax(_ov_tf))

_tf encodes the primary variable's log-scale toggle. Applying it to the overlay variable means that enabling log scale on the primary will also log-transform the overlay's threshold computations. If the overlay variable is, say, a volume fraction (positive, log-sensible) while the primary is pressure (possibly negative, log-disabled), the thresholds computed here will be in inconsistent spaces. The overlay isosurface thresholds should be computed with a transform derived from the overlay variable's own range and the user's intent, not the primary's _tf.

---

5. overlay_nlevels reused for two different semantics (interactive.py, ~line 1248)

In 2D and 3D-slice mode, overlay_nlevels is contour-line count. In 3D isosurface-overlay mode, the same slider is passed as the isosurface count:

int(overlay_nlevels or 3),  # used as iso_n for overlay isosurface

The slider is labeled "Levels" and ranges 1–20 with default 5. Isosurfaces are typically 1–5; contour lines 5–20. This dual semantics may produce unexpected results (e.g., 20 overlay isosurfaces on a coarse mesh) and the label won't make sense for the 3D overlay case. A dedicated "Overlay iso count" slider or dynamic label update would help.

---

Improvement opportunities

• The adaptive _min_frame_gap logic updates _last_update_t[0] inside _advance_step and at the end of _update. Because _advance_step fires before _update completes, the gap measured in _advance_step (now - _last_update_t[0]) reflects the previous frame's server time, not the current one. This is the intended behavior (rate-limit based on last frame), but the comment says "measured as wall-clock time between the start of _update and the next _advance_step call" — the comment is slightly imprecise since _last_update_t is set at the end of _update.

• The 3D overlay isosurface path always uses a 500K cell budget (_get_ds3(..., 500_000)) regardless of playback state, while the primary isosurface respects _playing. During playback, this makes the overlay significantly slower to compute than the primary mesh.

[github-actions]

Claude Code Review

Head SHA: e0cb2ed

Files changed: 3

• toolchain/mfc/viz/_step_cache.py (+2/-2)
• toolchain/mfc/viz/interactive.py (+961/-82)
• toolchain/pyproject.toml (+1/-0)

---

Summary

• Adds contour overlays for 2D/3D, solid-color isosurfaces, opacity control, and an improved timestep slider (dropdown → range slider)
• Server-side JPEG rendering via kaleido for 3D playback over SSH (Linux only), bypassing WebGL serialisation bottleneck
• Aggressive prefetch: CACHE_MAX 20→40, step-cache workers 1→3, new _mesh3_cache (40 entries) for 3D mesh pre-computation up to 15 steps ahead
• Replaces PyVista dependency with kaleido

---

Findings

1. kaleido added as a hard mandatory dependency — should be optional
toolchain/pyproject.toml line 1417:

"kaleido",

The kaleido path is only exercised on Linux during playback (sys.platform == 'linux' and _is_playing[0]), yet it is now a mandatory install for every platform. The existing _kaleido_available() graceful-degradation pattern already handles the "not installed" case — making it a hard dep is inconsistent with that design and forces macOS / Windows users to carry an extra 30 MB binary. It should be an extras group (e.g. kaleido = {version = "*", optional = true}) or at least documented as install-on-Linux-only.

2. Axis transposition in _compute_contour_traces (2D case)
interactive.py lines 319–321:

px = _interp_indices(contour[:, 0], x_cc)   # contour[:,0] is ROW index
py = _interp_indices(contour[:, 1], y_cc)   # contour[:,1] is COL index

skimage.find_contours returns contours in row-major (row, col) order. Whether this mapping is correct depends on the shape convention of the 2D array passed in. If data is stored as (nx, ny) (x varies along axis-0), this is correct. But if raw comes out of the NetCDF/SILO reader as (ny, nx) or (j, i) — which is common for C-order Fortran output — the x/y coordinates will be silently swapped and contours will appear mirrored across the diagonal. This should be verified against the actual output of ad.variables[overlay_var] for a 2D case.

3. Kaleido permanently disabled after first transient failure
interactive.py line 986:

except Exception as _kal_exc:
    _KALEIDO_OK = False   # never re-enabled for the lifetime of the server

A single OOM, timeout, or transient kaleido crash permanently silences server-side rendering for the rest of the session. Users would have to restart the viz server. Consider a retry counter (e.g. disable after N consecutive failures) or simply log and skip the current frame rather than blacklisting kaleido globally.

4. Overlay isosurface always uses the primary variable's log transform
interactive.py lines 1245–1248:

_ov_vx, ... = _compute_isomesh(
    _ov_ds, _ox, _oy, _oz,
    _tf,          # ← primary variable's transform
    _ov_ilo, _ov_ihi,
    ...
)

_tf is built from the primary variable's log-scale toggle. When the user has log scale on for the primary variable, the overlay variable's isosurface levels are computed in log space too, even though the overlay variable range (_ov_vmin/_ov_vmax) was also derived via _tf. The thresholds are consistent, but the overlay variable's data is transformed by the primary variable's choice — if the overlay variable contains zeros or negatives while the primary does not, np.where(arr > 0, np.log10(...), np.nan) will silently NaN out large regions of the overlay. At minimum, a comment documenting this coupling would help.

5. _update callback signature is now 30+ inputs — Dash serialisation overhead
interactive.py around line 1407: the main _update callback now takes 30 Input/State arguments and returns 5 outputs. Every UI interaction (even moving the opacity slider) triggers serialisation and transmission of all 30 values. For large 3D datasets where each callback already takes 1–5 s, this is probably negligible, but it makes the code harder to maintain and test. Consider splitting overlay callbacks into a separate _update_overlay that only fires when overlay controls change.

---

Minor / Nits

• _GRAPH_SHOW, _GRAPH_HIDE, _SRV_SHOW, _SRV_HIDE dicts are re-constructed on every callback invocation (interactive.py lines 789–798); these could be module-level constants.
• The trailing blank line before the closing ) in the two _prefetch_3d_mesh call sites (lines ~857 and ~882) is a minor style inconsistency — ./mfc.sh format should catch it.
• _mesh3_cache (40 entries of raw mesh arrays) combined with the step-data cache (40 entries) can hold a significant amount of RAM for large 3D datasets. No upper bound on total bytes — worth documenting the expected memory footprint in the module header.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 384d9b5a-367a-4337-af11-543378b8ad2a

📥 Commits

Reviewing files that changed from the base of the PR and between 5641acd and e0cb2ed.

📒 Files selected for processing (3)

• toolchain/mfc/viz/_step_cache.py
• toolchain/mfc/viz/interactive.py
• toolchain/pyproject.toml

---

📝 Walkthrough

Walkthrough

The changes introduce a 3D mesh prefetch cache system and server-side rendering capabilities using Kaleido, along with UI enhancements for contour overlays. The step cache is expanded to retain more timesteps, the prefetch thread pool is scaled up, and new helper functions support 3D mesh caching, Kaleido availability detection, and contour computation for both 2D and 3D data. The interactive rendering flow is extended to support cached mesh reuse and server-side image rendering, with new UI controls for overlay parameters and visibility toggles. A new Kaleido dependency is added to the project configuration.

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Tip

CodeRabbit can use Trivy to scan for security misconfigurations and secrets in Infrastructure as Code files.

Add a .trivyignore file to your project to customize which findings Trivy reports.

[Copilot]

Pull request overview

This PR enhances the interactive 3D visualization tool with contour overlays (2D and 3D), solid color isosurfaces, opacity controls, server-side rendering via kaleido for fast 3D playback over SSH, and aggressive data/mesh prefetching. It replaces the dropdown timestep selector with a slider and removes the pyvista dependency in favor of kaleido.

Changes:

• Added kaleido-based server-side Plotly rendering for 3D playback on Linux, with contour overlay support (2D find_contours + 3D slice contours), isosurface/isovolume overlay mixing, solid-color isosurface option, and opacity controls.
• Replaced the timestep dropdown with a slider and improved playback performance with adaptive frame-rate limiting, 3D mesh prefetching (50K cell budget, 3 workers), and a 40-entry mesh cache.
• Increased step cache size from 20→40 entries and prefetch workers from 1→3 for faster data pipeline throughput.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File	Description
toolchain/pyproject.toml	Added kaleido dependency for server-side image export
toolchain/mfc/viz/interactive.py	Core changes: kaleido rendering, contour overlays, 3D mesh prefetch cache, new UI controls, timestep slider, adaptive frame pacing
toolchain/mfc/viz/_step_cache.py	Increased cache size to 40 and prefetch workers to 3

[github-actions]

Claude Code Review

Head SHA: e0cb2ed
Files changed: 3

• toolchain/mfc/viz/interactive.py (+961 / -82)
• toolchain/mfc/viz/_step_cache.py (+2 / -2)
• toolchain/pyproject.toml (+1 / -0)

---

Summary

• Adds server-side Kaleido JPEG rendering for fast 3D playback over SSH on Linux
• Adds 3D mesh prefetch cache (separate thread pool, 40-entry LRU, 50K-cell budget)
• Adds contour overlays (2D/3D), isosurface/isovolume mixing, solid-color surfaces, opacity control
• Bumps step cache: CACHE_MAX 20→40, prefetch workers 1→3
• Adds kaleido as a required dependency

---

Findings

Bug — Overlay isosurface uses primary variable's log transform (interactive.py, lines ~1229–1248)

_ov_tf = _tf(_ov_3d_raw)           # applies primary variable's _tf to overlay data
_ov_vmin = float(np.nanmin(_ov_tf))
_ov_vmax = float(np.nanmax(_ov_tf))
…
_compute_isomesh(_ov_ds, _ox, _oy, _oz,
                 _tf,          # <-- primary variable's transform applied again inside
                 _ov_ilo, _ov_ihi, …)

_tf is built from the primary variable's log setting. When log=True, _tf applies log10. The overlay has no independent log control, so if the primary is log-scale and the overlay variable ranges over negatives or the transform is inappropriate for the second quantity, the isosurface thresholds and mesh geometry will be wrong. The comment at line 1228 ("Compute range in transformed space so thresholds match what _compute_isomesh sees after applying _tf internally") shows this is intentional, but it silently couples the overlay's rendering to the primary's scale choice. At minimum, a UI note or separate overlay-log checkbox is needed; at most, pass an identity _tf for the overlay and let users configure it independently.

The isovolume path has the same issue at line 1294: _ov_vf = _tf(_ov_ds.ravel()).astype(np.float32).

---

Medium — kaleido is a hard dependency but only useful on Linux/SSH (pyproject.toml, line ~1417)

"kaleido",

kaleido is invoked only when sys.platform == 'linux' and _is_playing[0] and ndim == 3. All other platforms install it for nothing. If kaleido has platform-specific install issues (it has historically had issues on some Linux distros needing Chrome/Chromium), it blocks the entire toolchain install. Consider:

"kaleido; sys_platform == 'linux'",

or move it to an optional [project.optional-dependencies] extras group (e.g., ssh-viz). The runtime import guard in _kaleido_available() handles absence gracefully — the hard dep is unnecessary.

---

Medium — _KALEIDO_OK written from callback without synchronisation (interactive.py, line ~986)

_KALEIDO_OK = False   # set on kaleido render failure

_KALEIDO_OK is a module-level variable written from a Dash callback, which runs in a thread pool when threaded=True (the default). Two concurrent callbacks could both try if _KALEIKO_OK is None: _KALEIDO_OK = _kaleido_available(). In CPython the GIL makes bare boolean writes safe, but the check-then-set pattern at lines 913–914 is a data race on None → value under multiple workers. Wrapping in a threading.Lock (like the existing _jpeg_pool_lock pattern) is a one-line fix and makes the intent explicit.

---

Minor — Potential row/column axis swap in _compute_contour_traces (interactive.py, lines 319–320)

px = _interp_indices(contour[:, 0], x_cc)   # row index → x_cc
py = _interp_indices(contour[:, 1], y_cc)   # col index → y_cc

skimage.find_contours returns (row, col) = (axis-0, axis-1) indices. This is correct only if the data_2d array is stored as (nx, ny) (axis-0 = x). If the heatmap's z data uses the standard matrix convention (ny, nx) (axis-0 = y), the contour lines will be transposed relative to the heatmap. Please verify against the actual layout of ad.variables[overlay_var] for 2D fields.

---

Minor — Module-level caches not reset between s_interactive_viz() calls (interactive.py)

_mesh3_cache, _mesh3_in_flight, _KALEIDO_OK, _is_playing, _last_update_t, and _min_frame_gap are all module globals. If s_interactive_viz() is ever called twice in one process (e.g., in tests or a REPL), the state from the first call leaks into the second. Consider resetting these at the top of s_interactive_viz(), or document that the function is intended to be called exactly once per process.

---

No issues with

• Thread-pool shutdown via atexit.register — correct pattern, consistent with existing _jpeg_pool.
• _prefetch_3d_mesh lock hygiene — _mesh3_in_flight add/discard are both inside with _mesh3_lock.
• Cache eviction in _bg — double-check pattern inside lock is correct.
• Kaleido fallback path — _KALEIDO_OK = False on first failure prevents repeated kaleido attempts; graceful.
• _compute_contour_traces_3d axis assignments for all three slice_axis cases look correct given (nx, ny, nz) layout.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 45.36%. Comparing base (25a074e) to head (e0cb2ed).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1310   +/-   ##
=======================================
  Coverage   45.36%   45.36%           
=======================================
  Files          70       70           
  Lines       20515    20499   -16     
  Branches     1954     1953    -1     
=======================================
- Hits         9306     9300    -6     
+ Misses      10082    10074    -8     
+ Partials     1127     1125    -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.