Replies: 1 comment 1 reply
-
|
I've added to machine.json the {
"group_size": 0,
"para_deg": 6
}It gives the following error Traceback (most recent call last):
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 358, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 862, in handle_unexpected_job_state
raise RuntimeError(err_msg)
RuntimeError: job:ee0f718911fbce0faa2a41cd1504faab92fd48fb 21510 failed 3 times.
Possible remote error message: [31m==> /export/home/liluotonggpu2/dpmdtest/dpgen-FeCH/work/d70573f8b1a96d201444ee1dd94cec952ff258d7/task.000.000218/model_devi.log <==
erformance. See https://deepmd.rtfd.io/parallelism/ for more information.
DeePMD-kit: Successfully load libcudart.so.11.0
2024-07-10 09:58:46.121537: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-07-10 09:58:54.745553: F tensorflow/core/platform/statusor.cc:33] Attempting to fetch value instead of handling error INTERNAL: failed initializing StreamExecutor for CUDA device ordinal 3: INTERNAL: failed call to cuDevicePrimaryCtxRetain: CUDA_ERROR_OPERATING_SYSTEM: OS call failed or operation not supported on this OS
/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/lmp: line 11: 31934 Aborted (core dumped) /export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/_lmp "$@"
[0m
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/dpgen", line 8, in <module>
sys.exit(main())
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpgen/generator/run.py", line 5394, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpgen/generator/run.py", line 4736, in run_iter
run_model_devi(ii, jdata, mdata)
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpgen/generator/run.py", line 2081, in run_model_devi
run_md_model_devi(iter_index, jdata, mdata)
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpgen/generator/run.py", line 2075, in run_md_model_devi
submission.run_submission()
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 261, in run_submission
self.handle_unexpected_submission_state()
File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 362, in handle_unexpected_submission_state
raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/export/home/liluotonggpu2/dpmdtest/dpgen-FeCH/work/d70573f8b1a96d201444ee1dd94cec952ff258d7.
Debug information: submission_hash==d70573f8b1a96d201444ee1dd94cec952ff258d7.
Please check error messages above and in remote_root. The submission information is saved in /export/home/liluotonggpu2/.dpdispatcher/submission/d70573f8b1a96d201444ee1dd94cec952ff258d7.json.
For furthur actions, run the following command with proper flags: dpdisp submission d70573f8b1a96d201444ee1dd94cec952ff258d7 |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
what did the logs say. Is there concrete error messages pointing out which program failed？ @ZKNB |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
When I ran the task using dpgen, I found the following reported error
My param.json as follows：
{ "type_map": ["H","C","Fe"], "mass_map": [1,12,56], "init_data_prefix": "./", "init_data_sys": ["./FeCH/training_data1", "./FeCH/training_data10", "./FeCH/training_data100" ......], "sys_format":"vasp/poscar", "sys_configs_prefix": "./deep-cnt-data/", "sys_configs": [ ["C12/*/POSCAR"(contains 200 structure)] ], "_comment": " that's all ", "numb_models": 4, "default_training_param": { "model": { "type_map": ["H","C","Fe"], "descriptor": { "type": "se_atten_v2", "sel": "auto", "rcut_smth": 0.5, "rcut": 6.0, "neuron": [25,50,100], "resnet_dt": true, "axis_neuron": 12, "attn": 128, "attn_layer":0, "attn_dotr":true, "seed": 1 }, "fitting_net": { "neuron": [240,240,240], "resnet_dt": false, "seed": 1 } }, "learning_rate": { "type": "exp", "start_lr": 0.001, "stop_lr":3.5e-8, "decay_steps": 5000 }, "loss": { "start_pref_e": 0.02, "limit_pref_e": 2, "start_pref_f": 1000, "limit_pref_f": 1, "start_pref_v": 0.0, "limit_pref_v": 0.0 }, "training": { "stop_batch": 1000000, "disp_file": "lcurve.out", "disp_freq": 1000, "numb_test": 10, "save_freq": 2000, "save_ckpt": "model.ckpt", "disp_training": true, "time_training": true, "profiling": false, "profiling_file": "timeline.json", "_comment": "that's all" } }, "model_devi_dt": 0.002, "model_devi_skip": 0, "model_devi_f_trust_lo": 0.15, "model_devi_f_trust_hi": 0.45, "model_devi_e_trust_lo": 10000000000.0, "model_devi_e_trust_hi": 10000000000.0, "model_devi_clean_traj": true, "model_devi_merge_traj": true, "model_devi_jobs": [ {"sys_idx": [0],"temps": [1000.00,1300.00,1500.00],"press": [1.0],"trj_freq": 10,"nsteps": 300,"ensemble": "nvt","_idx": "00"} ], "fp_style": "vasp", "ratio_failed":0.35, "shuffle_poscar": false, "fp_task_max": 30, "fp_task_min": 5, "fp_pp_path": "./", "fp_pp_files": ["POTCAR_H","POTCAR_C","POSCAR_Fe"], "fp_incar": "./INCAR" }My machine.json as follows：
{ "api_version": "1.0", "train" :[ { "command": "dp", "machine": { "batch_type": "shell", "machine_type": "shell", "context_type": "local", "local_root" : "./", "remote_root": "/export/home/liluotonggpu2/dpmdtest/dpgen-FeCH/work" }, "resources": { "number_node": 2, "cpu_per_node": 12, "group_size": 1 } } ], "model_devi":[ { "command": "lmp", "machine": { "batch_type": "shell", "machine_type": "shell", "context_type": "local", "local_root" : "./", "remote_root": "/export/home/liluotonggpu2/dpmdtest/dpgen-FeCH/work" }, "resources": { "number_node": 2, "cpu_per_node": 12, "group_size": 1 } } ], "fp":[ { "command": "ulimit -s unlimited;mpirun -np 32 ~/bin/vasp > vasp.out", "machine": { "batch_type": "Torque", "context_type": "SSHContext", "local_root" : "./", "remote_root": "/export/home/liluotong/remote-dpgen", "remote_profile": { "hostname": "192.168.9.251", "username": "liluotong", "port": 22 } }, "resources": { "number_node": 4, "cpu_per_node": 32, "gpu_per_node": 0, "queue_name":"largemem", "group_size": 5 } } ] }The task is ready to do model_devl after the four models are trained when the error occurs, I don't know if it's due to insufficient local memory that causes the task to exit or if it's due to GPU memory issues.
Beta Was this translation helpful? Give feedback.
All reactions