sdo_data_root_path=config["data"]["sdo_data_root_path"]

[Xyxiaoyan]

Hello,When I run torchrun finetune.py, I get the following error:KeyError: 'sdo_data_root_path'.It seems that my config is missing config["data"]["sdo_data_root_path"]. Could you tell me where this path should be set, or how to specify it correctly?

[take2rohit]

Hi @Xyxiaoyan ,
Thanks for raising this!

You can set sdo_data_root_path to the root directory where your SDO data resides. This issue has already been addressed in #36, which is ready to be merged into main.

Essentially, config["data"]["sdo_data_root_path"] prepends the root path to the relative paths specified in the CSV files (see #34 for reference).

For expected usage, please refer to:

Surya/downstream_examples/solar_flare_forcasting/config.yaml

Lines 1 to 6 in 4df22f4

	job_id: solar_flare_forcasting
	data:
	sdo_data_root_path: /nobackupnfs1/sroy14/processed_data/Helio/nc
	train_data_path: ./assets/train_index_surya_1_0.csv
	valid_data_path: ./assets/valid_index_surya_1_0.csv
	train_flare_data_path: ./assets/surya-bench-flare-forecasting/train.csv

Let me know if this solves your problem. Thanks!

[Xyxiaoyan]

Hi, I haven’t worked on this project for quite a while, and I noticed that the downloaded files don’t seem to contain the directories you mentioned. I checked your directory structure and modified my infer.yaml accordingly, so that my finetune.py file can run. However, I’m not sure if this modification is the correct approach. Could you please confirm?
sdo_data_root_path: ../common_data
train_data_path: ./assets/sdo_test.csv
valid_data_path: ./assets/sdo_validate.csv
flare_data_path: ./assets/surya-bench-flare-forecasting/data.csv

Here is my project directory structure. I’m not sure if I might have encountered some issues during the download process. If you could point out any problems, I would greatly appreciate it.

|-- Surya
|   |-- LICENSE
|   |-- README.md
|   |-- assets
|   |   |-- ar_seg_results.png
|   |   `-- surya_model_validation.png
|   |-- data
|   |   |-- Surya-1.0
|   |   |   |-- config.yaml
|   |   |   |-- scalers.yaml
|   |   |   `-- surya.366m.v1.pt
|   |   `-- Surya-1.0_validation_data
|   |       |-- 20140107_1500.nc
|   |       |-- ...
|   |       
|   |       `-- 20140107_1924.nc
|   |-- downstream_examples
|   |   |-- README.md
|   |   |-- ar_segmentation
|   |   |   |-- README.md
|   |   |   |-- ar_segmentation_tutorial.ipynb
|   |   |   |-- assets
|   |   |   |   |-- sdo_test.csv
|   |   |   |   `-- sdo_validate.csv
|   |   |   |-- config.yaml
|   |   |   |-- config_infer.yaml
|   |   |   |-- create_ar_csv.py
|   |   |   |-- dataset.py
|   |   |   |-- download_data.sh
|   |   |   |-- finetune.py
|   |   |   |-- infer.py
|   |   |   `-- models.py
|   |   |-- common_data
|   |   |   |-- tars
|   |   |   |   |-- test_splitted_tars
|   |   |   |   |   |-- test.tar
|   |   |   |   |   |-- test.tar.part_aa
|   |   |   |   |   |-- ...
|   |   |   |   |   
|   |   |   |   |   `-- test.tar.part_aq
|   |   |   |   `-- validate_splitted_tars
|   |   |   |       |-- val.tar.part_aa
|   |   |   |       |-- ...
|   |   |   |       
|   |   |   |       |-- val.tar.part_aq
|   |   |   |       `-- validate.tar
|   |   |   |-- test
|   |   |   |   |-- 20110103_1200.nc
|   |   |   |   |-- 20110103_1212.nc
|   |   |   |   |-- ...
|   |   |   |   
|   |   |   |   `-- 20110105_2348.nc
|   |   |   `-- validate
|   |   |       |-- 20110106_0000.nc
|   |   |       |-- 20110106_0012.nc
|   |   |       |-- ...
|   |   |       
|   |   |       `-- 20110108_1148.nc
|   |   |-- download_sample_train_data.py
|   |   |-- euv_spectra_prediction
|   |   |   |-- README.md
|   |   |   |-- __pycache__
|   |   |   |   |-- dataset.cpython-311.pyc
|   |   |   |   |-- finetune.cpython-311.pyc
|   |   |   |   |-- infer.cpython-311.pyc
|   |   |   |   `-- models.cpython-311.pyc
|   |   |   |-- assets
|   |   |   |   |-- euv-spectra
|   |   |   |   |   `-- archive
|   |   |   |   |-- euv_spectra_weights.pth
|   |   |   |   |-- infer_data
|   |   |   |   |   |-- 20110120_0100.nc
|   |   |   |   |   |-- 20110120_0200.nc
|   |   |   |   |   |-- 20110120_0300.nc
|   |   |   |   |   |-- 20120115_0048.nc
|   |   |   |   |   |-- 20120115_0100.nc
|   |   |   |   |   |-- 20120115_0148.nc
|   |   |   |   |   |-- 20120115_0200.nc
|   |   |   |   |   |-- 20120115_0248.nc
|   |   |   |   |   |-- 20120115_0300.nc
|   |   |   |   |   |-- 20130128_1812.nc
|   |   |   |   |   |-- 20130128_1912.nc
|   |   |   |   |   |-- 20130128_2012.nc
|   |   |   |   |   |-- 20190123_0200.nc
|   |   |   |   |   |-- 20190123_0300.nc
|   |   |   |   |   |-- 20190123_0400.nc
|   |   |   |   |   `-- infer_index_surya_1_0.csv
|   |   |   |   |-- scalers.yaml
|   |   |   |   |-- sdo_test.csv
|   |   |   |   |-- sdo_validate.csv
|   |   |   |   |-- train_index_surya_1_0.csv
|   |   |   |   `-- valid_index_surya_1_0.csv
|   |   |   |-- config.yaml
|   |   |   |-- config_infer.yaml
|   |   |   |-- dataset.py
|   |   |   |-- download_data.sh
|   |   |   |-- env_spectra_tutorial.py
|   |   |   |-- euv_spectra_tutorial.ipynb
|   |   |   |-- finetune.py
|   |   |   |-- infer.py
|   |   |   |-- inference_results
|   |   |   |   `-- euv_spectra_predictions.png
|   |   |   |-- logs
|   |   |   |   `-- data
|   |   |   |       |-- 20251026T053539Z_000_data_train_836770.log
|   |   |   |       |-- 20251026T053539Z_000_data_train_836818.log
|   |   |   |       |-- 20251026T053539Z_000_data_train_836866.log
|   |   |   |       |-- 20251026T053617Z_000_data_train_837392.log
|   |   |   |       `-- 20251026T054044Z_000_data_train_841013.log
|   |   |   `-- models.py
|   |   |-- solar_flare_forcasting
|   |   |   |-- README.md
|   |   |   |-- __pycache__
|   |   |   |   |-- dataset.cpython-311.pyc
|   |   |   |   |-- finetune.cpython-311.pyc
|   |   |   |   |-- infer.cpython-311.pyc
|   |   |   |   |-- metrics.cpython-311.pyc
|   |   |   |   `-- models.cpython-311.pyc
|   |   |   |-- assets
|   |   |   |   |-- infer_data
|   |   |   |   |   |-- 20110120_0100.nc
|   |   |   |   |   |-- 20110120_0200.nc
|   |   |   |   |   |-- 20110120_0300.nc
|   |   |   |   |   |-- 20120115_0048.nc
|   |   |   |   |   |-- 20120115_0100.nc
|   |   |   |   |   |-- 20120115_0148.nc
|   |   |   |   |   |-- 20120115_0200.nc
|   |   |   |   |   |-- 20120115_0248.nc
|   |   |   |   |   |-- 20120115_0300.nc
|   |   |   |   |   |-- 20130128_1812.nc
|   |   |   |   |   |-- 20130128_1912.nc
|   |   |   |   |   |-- 20130128_2012.nc
|   |   |   |   |   |-- 20190123_0200.nc
|   |   |   |   |   |-- 20190123_0300.nc
|   |   |   |   |   |-- 20190123_0400.nc
|   |   |   |   |   `-- infer_index_surya_1_0.csv
|   |   |   |   |-- scalers.yaml
|   |   |   |   |-- sdo_test.csv
|   |   |   |   |-- sdo_validate.csv
|   |   |   |   |-- solar_flare_weights.pth
|   |   |   |   |-- surya-bench-flare-forecasting
|   |   |   |   |   |-- README.md
|   |   |   |   |   |-- data.csv
|   |   |   |   |   |-- leaky_validation.csv
|   |   |   |   |   |-- test.csv
|   |   |   |   |   |-- train.csv
|   |   |   |   |   `-- validation.csv
|   |   |   |   |-- train_index_surya_1_0.csv
|   |   |   |   `-- valid_index_surya_1_0.csv
|   |   |   |-- checkpoints
|   |   |   |   `-- epoch_0.pth
|   |   |   |-- config.yaml
|   |   |   |-- config_infer.yaml
|   |   |   |-- dataset.py
|   |   |   |-- download_data.sh
|   |   |   |-- finetune.py
|   |   |   |-- infer.py
|   |   |   |-- inference_results
|   |   |   |   |-- result_0.txt
|   |   |   |   |-- result_1.txt
|   |   |   |   `-- result_2.txt
|   |   |   |-- logs
|   |   |   |   `-- data
|   |   |   |       |-- 20251025T135628Z_000_data_valid_520275.log
|   |   |   |       |-- ...
|   |   |   |       
|   |   |   |       `-- 20251027T070122Z_000_data_train_1084812.log
|   |   |   |-- metrics.py
|   |   |   |-- models.py
|   |   |   |-- solar_flare_tutorial.ipynb
|   |   |   `-- solar_flare_tutorial.py
|   |   `-- solar_wind_forcasting
|   |       |-- README.md
|   |       |-- __pycache__
|   |       |   |-- dataset.cpython-311.pyc
|   |       |   |-- finetune.cpython-311.pyc
|   |       |   |-- infer.cpython-311.pyc
|   |       |   `-- models.cpython-311.pyc
|   |       |-- assets
|   |       |   |-- Surya-bench-solarwind
|   |       |   |   |-- README.md
|   |       |   |   |-- leaky_validation.csv
|   |       |   |   |-- solar_wind_data.png
|   |       |   |   |-- test.csv
|   |       |   |   |-- train.csv
|   |       |   |   `-- validation.csv
|   |       |   |-- infer_data
|   |       |   |   |-- 20110120_0100.nc
|   |       |   |   |-- 20110120_0200.nc
|   |       |   |   |-- 20110120_0300.nc
|   |       |   |   |-- 20120115_0048.nc
|   |       |   |   |-- 20120115_0100.nc
|   |       |   |   |-- 20120115_0148.nc
|   |       |   |   |-- 20120115_0200.nc
|   |       |   |   |-- 20120115_0248.nc
|   |       |   |   |-- 20120115_0300.nc
|   |       |   |   |-- 20130128_1812.nc
|   |       |   |   |-- 20130128_1912.nc
|   |       |   |   |-- 20130128_2012.nc
|   |       |   |   |-- 20190123_0200.nc
|   |       |   |   |-- 20190123_0300.nc
|   |       |   |   |-- 20190123_0400.nc
|   |       |   |   `-- infer_index_surya_1_0.csv
|   |       |   |-- scalers.yaml
|   |       |   |-- sdo_test.csv
|   |       |   |-- sdo_validate.csv
|   |       |   |-- solar_wind_weights.pth
|   |       |   |-- train_index_surya_1_0.csv
|   |       |   `-- valid_index_surya_1_0.csv
|   |       |-- config.yaml
|   |       |-- config_infer.yaml
|   |       |-- dataset.py
|   |       |-- download_data.sh
|   |       |-- finetune.py
|   |       |-- infer.py
|   |       |-- logs
|   |       |   `-- data
|   |       |       `-- 20251026T145537Z_000_data_train_997266.log
|   |       |-- models.py
|   |       `-- solar_wind_tutorial.ipynb
|   |-- logs
|   |   `-- data
|   |       `-- 20251024T081200Z_000_data_valid_29652.log
|   |-- pyproject.toml
|   |-- surya
|   |   |-- __init__.py
|   |   |-- __pycache__
|   |   |   `-- __init__.cpython-311.pyc
|   |   |-- datasets
|   |   |   |-- __init__.py
|   |   |   |-- __pycache__
|   |   |   |   |-- __init__.cpython-311.pyc
|   |   |   |   |-- helio.cpython-311.pyc
|   |   |   |   `-- transformations.cpython-311.pyc
|   |   |   |-- helio.py
|   |   |   `-- transformations.py
|   |   |-- models
|   |   |   |-- __init__.py
|   |   |   |-- __pycache__
|   |   |   |   |-- __init__.cpython-311.pyc
|   |   |   |   |-- embedding.cpython-311.pyc
|   |   |   |   |-- flow.cpython-311.pyc
|   |   |   |   |-- helio_spectformer.cpython-311.pyc
|   |   |   |   |-- spectformer.cpython-311.pyc
|   |   |   |   `-- transformer_ls.cpython-311.pyc
|   |   |   |-- embedding.py
|   |   |   |-- flow.py
|   |   |   |-- helio_spectformer.py
|   |   |   |-- spectformer.py
|   |   |   `-- transformer_ls.py
|   |   `-- utils
|   |       |-- __init__.py
|   |       |-- __pycache__
|   |       |   |-- __init__.cpython-311.pyc
|   |       |   |-- config.cpython-311.pyc
|   |       |   |-- data.cpython-311.pyc
|   |       |   |-- distributed.cpython-311.pyc
|   |       |   |-- log.cpython-311.pyc
|   |       |   |-- misc.cpython-311.pyc
|   |       |   `-- schemas.cpython-311.pyc
|   |       |-- config.py
|   |       |-- data.py
|   |       |-- distributed.py
|   |       |-- log.py
|   |       |-- misc.py
|   |       `-- schemas.py
|   |-- surya_model_validation.png
|   |-- tests
|   |   |-- __pycache__
|   |   |   `-- test_surya.cpython-311-pytest-8.4.1.pyc
|   |   |-- test_surya.py
|   |   `-- test_surya_index.csv
|   `-- uv.lock
`-- project_tree.txt