End-to-end verify test killed by OOM Killer

[fafed]

Hello, while running the verification test I noticed that the python process is being terminated by the kernel due to OOM after a few minutes.

I could not find any mention of minimum system requirements in the installation instructions.
This makes it unclear whether the OOM kill I experienced is due simply because my machine does not meet the expected baseline.

Environment

• Virtual machine
• OS: Ubuntu 24.04
• RAM: 32GB (no swap)
• GPU: dedicated NVIDIA RTX A6000 48GB
• Storage 500GB (9% max used)

Steps to reproduce:

1. Install following installation instructions
2. Run: python -m pytest -s -o log_cli=true tests/test_surya.py

Test output

=================================================================== test session starts ===================================================================
platform linux -- Python 3.12.3, pytest-8.4.1, pluggy-1.6.0 -- xxxx
cachedir: .pytest_cache
rootdir: xxxx
configfile: pyproject.toml
collected 1 item

Fetching 3 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 5256.02it/s]
Fetching 23 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 2383.36it/s]

---------------------------------------------------------------------- live log call ----------------------------------------------------------------------
INFO     test_surya:test_surya.py:254 GPU detected. Running the test on device 0.
INFO     test_surya:test_surya.py:261 Surya FM: 366.19 M total parameters.
INFO     test_surya:test_surya.py:267 Loaded weights.
INFO     test_surya:test_surya.py:269 Starting inference run.
Killed
[Exit 137 (KILL)]

kernel buffer:
[Fri Oct 10 16:05:21 2025] Out of memory: Killed process 134718 (python) total-vm:45829840kB, anon-rss:7688412kB, file-rss:67584kB, shmem-rss:12599180kB, UID:1001 pgtables:42380kB oom_score_adj:0

I can provide additional logs or run further tests if needed.

[johannesschmude]

Hi @fafed,

Thanks for bringing this up. Indeed, we never tested how much system memory would be needed. And from the looks of it, this is indeed the system running out of memory. Given the logs we see here, the most likely explanation is that this happens in the data loader. Things one could try:

• Setting num_workers = 0. That reduces the number of data loaders (and thus memory by 2), it also means that this will happen in the main process.
• Set rollout_steps = 0. This will invalidate the test, but should again reduce memory usage as we are loading less (target) data.

[fafed]

Hi @johannesschmude.

I tried to use your settings without swap and with num_workers = 0 test ends in OOM. Setting rollout_steps = 0 test fails with this error:

_____________________________ test_surya_20140107 ______________________________
tests/test_surya.py:275: in test_surya_20140107
    batch_loss, data_returned = batch_step(
tests/test_surya.py:213: in batch_step
    curr_target = batch_data["forecast"][:, :, step, ...].to(device)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E   IndexError: index 1 is out of bounds for dimension 2 with size 1

I performed additional tests by adding a 20 GiB swap file. With this memory increase, the test completed successfully, but it required about one hour to finish. During this period, GPU utilization was minimal. The maximum swap usage reached 14.6 GiB (with 32GiB of RAM).

A separate test on a machine with 64 GiB of RAM and two L40S cards completed within a few minutes (results vary significantly between runs) and did not use any swap.

GPU memory usage peaked at 14.12 GiB, only one GPU was used.

Downloading the sample dataset requires approximately 1.2 TB of free disk mainly for tar rebuild.

[ep150de]

I ran into similar OOM error, but mine was: CUDA out of memory error; running system with 12gb VRAM. 4070 w/ 12GBvram 32gb dRAM, 500gb drive

Seems to be memory management and allocation related here's my error output