About LAMMPS GPU efficiency with CPU parallelization #4614
Replies: 3 comments
-
|
Also I tried using "mpirun -np 18 lmp -in in.production.txt" with "processors 3 3 2" and I noticed a significant improvement in speed to 4.5 steps/s. I understand that perhaps using too large number of CPU cores this could have diminishing returns as more cores requires more communication with GPU. I am wondering if anyone else has experienced this phenomenon. Later I also tried decreasing processors and checking the speed. Interestingly enough, the fastest speed is "processors 2 1 1" with 6.25 steps/s. Anything above this will slow down the LAMMPS simulation. |
Beta Was this translation helpful? Give feedback.
-
|
A GPU card can only be utilized by at most a single process simultaneously if the MPS is not set up. Read MPS documentation for details. |
Beta Was this translation helpful? Give feedback.
-
|
OK, thanks for this information! With MPS, I tried running 24 processes, 12 per GPU with "processes 4 3 2" and I am getting an improved speed of 5.63 steps/s. Although I could run up to 48 processes & 24 per GPU, I am getting an out-of-memory error, likely because each process has to load the DeePMD model and perform calculations. Probably doing model compression will fix this issue. I will share my batch file here for others to use. Thanks for your time! #!/bin/bash conda activate /home/adr1/deepmd-kit-gpu_3.0.1 export OMP_NUM_THREADS=1 #On single gpu #On multiple gpu export CUDA_VISIBLE_DEVICES=0,1 |
Beta Was this translation helpful? Give feedback.
-
Hello everyone,
I am trying to run a 12000 atom system in crystalline ZrNiSn using a trained DeePMD model (v.3.0.1, se_atten_v2 model) on LAMMPS. My installation was done using off-line packages by "sh deepmd-kit-3.0.1-cuda126-Linux-x86_64.sh". Currently I am running LAMMPS on two Tesla V100-PCIE-32GB GPU with a 48 core CPU. The performance of this configuration is 3.47138 steps/s with all 48 cores, "processors 4 3 4" in LAMMPS, whereas it is 3 steps/s with only 1 core, "processors 1 1 1". The performance is not so much better when I use all 48 cores. Although I know this performance depends on many factors, especially running on a HPC cluster, but I figure this is an important discussion for anyone wanting to do large-scale simulations with DeePMD using CPU parallelization and GPU acceleration. I will share my configuration below including my SLURM batch file and LAMMPS input script, and any immediate suggestions from those more experienced to improve the performance would be greatly appreciated from other users and I. If more information is needed, please let me know in a comment and I would be happy to provide. Thank you for your time!
------SLURM SCRIPT-------
#!/bin/bash
#SBATCH --job-name=lmp_test
#SBATCH -p v100-32gb-hiprio ###change to high priority queue
#SBATCH -n 48
#SBATCH -N 1
#SBATCH --gres=gpu:2
#SBATCH --output job%j.out
#SBATCH --error job%j.err
conda activate /home/adr1/deepmd-kit-gpu_3.0.1
module load cuda/12.3
export CUDA_VISIBLE_DEVICES=0,1
export OMP_NUM_THREADS=1
export DP_INTRA_OP_PARALLELISM_THREADS=2
export DP_INTER_OP_PARALLELISM_THREADS=1
export XLA_FLAGS=--xla_gpu_cuda_data_dir=/work/apps/cuda/12.3/
export TF_XLA_FLAGS=--tf_xla_auto_jit=2
mpirun -np 48 lmp -in in.production.txt
-----LAMMPS INPUT-----
ZrNiSn
units metal
atom_style atomic
boundary p p p
processors 4 3 4
#neighbor 2.0 bin
#neigh_modify every 1 delay 0 check yes
#comm_modify cutoff 12.0
read_data structure.lammps
pair_style deepmd /work/adr1/ZrNiSn_TE/04_DeePMD_Training/1_training_folder/0_dp_model_limited_all_T/freeze_model/dp_3.0.1/model.pb
pair_coeff * * Ni Sn Zr
mass 1 58.6934
mass 2 118.71
mass 3 91.224
timestep 0.001
thermo 100
thermo_style custom step temp pe etotal lx ly lz cpu
variable T equal 1300
variable temp_total equal temp
variable volume_total equal vol
variable ke_total equal ke
variable pe_total equal pe
variable e_total equal etotal
variable lx equal lx
variable ly equal ly
variable lz equal lz
fix 101 all ave/time 100 1 100 v_temp_total v_volume_total v_e_total v_pe_total file thermo.a.dat
velocity all create ${T} 21350672 mom yes rot yes sum no dist gaussian
#Relaxation
fix 1 all nvt temp ${T} ${T} 0.1
fix 3 all momentum 1 linear 1 1 1
dump myDump all xyz 10000 movie.xyz
dump_modify myDump element Ni Sn Zr
run 100000
unfix 1
unfix 3
undump myDump
fix 1 all nve
fix 3 all momentum 1 linear 1 1 1
-----OUTPUT with "processors 4 3 4"------
(/home/adr1/deepmd-kit-gpu_3.0.1) [adr1@node366 1300K_runs_gpu2]$ mpirun -np 48 lmp -in in.production.txt
LAMMPS (29 Aug 2024)
using 1 OpenMP thread(s) per MPI task
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load DeePMD-kit: Successfully load libcudart.so.12
libcudart.so.12DeePMD-kit: Successfully load DeePMD-kit: Successfully load libcudart.so.12
libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load DeePMD-kit: Successfully load libcudart.so.12libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load DeePMD-kit: Successfully load libcudart.so.12
libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
DeePMD-kit: Successfully load libcudart.so.12
2025-02-25 12:20:38.238108: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238115: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238109: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238113: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238116: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238117: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238118: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238118: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238118: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238110: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238116: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238117: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238112: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238109: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238113: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238109: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238230: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238233: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238260: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238299: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.238305: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.239505: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.240038: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246154: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246166: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246156: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246161: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246169: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246168: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246153: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246167: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246163: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246164: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246153: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246151: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246167: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246161: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246227: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246354: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246453: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246473: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246567: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246679: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246683: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246912: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.246920: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.247390: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.248267: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable
TF_ENABLE_ONEDNN_OPTS=0.2025-02-25 12:20:38.256033: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256038: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256039: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256043: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256043: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256041: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256043: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256040: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256043: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256049: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256044: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256052: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256056: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256073: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256077: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256078: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256089: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256096: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256098: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256101: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256123: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.256903: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.257537: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264051: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264059: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264051: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264071: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264051: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264053: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264058: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264057: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264056: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264052: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264071: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264072: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264077: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264078: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264082: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264094: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264087: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264095: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264110: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264237: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264566: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264647: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.264846: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.265432: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.265958: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-02-25 12:20:38.279670: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279697: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279695: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279698: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279695: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279696: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279704: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279707: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279716: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279717: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279722: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279724: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279723: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279740: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279739: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279744: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279748: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279751: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279755: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.279798: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.280072: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.280661: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.281410: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.286695: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286699: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286699: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286702: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286708: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286712: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286715: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286719: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286722: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286728: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286728: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286731: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286739: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286745: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286750: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286754: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286756: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286756: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286762: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.286764: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.287107: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.287721: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.288148: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288150: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288159: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288149: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288149: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288165: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288166: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288171: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288172: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288173: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288172: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288195: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288195: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288204: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288209: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288209: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288212: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288215: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288233: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288310: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.288576: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.288657: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.289393: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.289431: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.289447: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.290156: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-02-25 12:20:38.295349: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295349: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295348: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295352: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295352: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295361: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295362: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295367: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295368: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295369: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295374: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295382: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295384: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295390: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295391: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295394: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295414: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295417: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295425: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295484: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.295855: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.296457: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.296758: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.296824: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-02-25 12:20:38.297588: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
Loaded 1 plugins from /home/adr1/deepmd-kit-gpu_3.0.1/lib/deepmd_lmp
Reading data file ...
triclinic box = (0 0 0) to (61.518166 61.518166 61.518166) with tilt (0 0 0)
4 by 3 by 4 MPI processor grid
reading atoms ...
12000 atoms
read_data CPU = 0.034 seconds
Summary of lammps deepmd module ...
CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE
Your simulation uses code contributions which should be cited:
The log file lists these citations in BibTeX format.
CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE
Generated 0 of 3 mixed pair_coeff terms from geometric mixing rule
Neighbor list info ...
update: every = 1 steps, delay = 0 steps, check = yes
max neighbors/atom: 2000, page size: 100000
master list distance cutoff = 9
ghost atom cutoff = 9
binsize = 4.5, bins = 14 14 14
1 neighbor lists, perpetual/occasional/extra = 1 0 0
(1) pair deepmd, perpetual
attributes: full, newton on
pair build: full/bin/atomonly
stencil: full/bin/3d
bin: standard
Setting up Verlet run ...
Unit style : metal
Current step : 0
Time step : 0.001
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.065489 45290 service.cc:146] XLA service 0x2aac4c004aa0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.065557 45290 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.065925 45308 service.cc:146] XLA service 0x2aac4c004a90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.065972 45308 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.068586 45335 service.cc:146] XLA service 0x2aac4c004ca0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.068646 45335 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.071978 45377 service.cc:146] XLA service 0x2aac4c0025f0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.072033 45377 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.072102 45260 service.cc:146] XLA service 0x2aac4c003b90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.072157 45260 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.072344 45397 service.cc:146] XLA service 0x2aac4c004010 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.072390 45397 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.073179 45147 service.cc:146] XLA service 0x2aac4c002700 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.073222 45147 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.074667 45228 service.cc:146] XLA service 0x2aac4c0049c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.074718 45228 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.075444: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.075451: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.076595: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.077456 45239 service.cc:146] XLA service 0x2aac4c005090 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.077507 45239 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.077814 45206 service.cc:146] XLA service 0x2aac4c004ca0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.077863 45206 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.078682 45384 service.cc:146] XLA service 0x2aac4c0049b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.078722 45384 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.078948 45357 service.cc:146] XLA service 0x2aac4c004780 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.078988 45357 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.079955 45393 service.cc:146] XLA service 0x2aac4c003870 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.080005 45393 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.081426 45316 service.cc:146] XLA service 0x2aac4c004810 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.081471 45316 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.082060: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.082218 45360 service.cc:146] XLA service 0x2aac4c004980 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.082256 45360 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.082397: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.082562 45372 service.cc:146] XLA service 0x2aac4c004d60 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.082598 45372 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.082628 45370 service.cc:146] XLA service 0x2aac4c004f10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.082665 45370 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.082868 45398 service.cc:146] XLA service 0x2aac4c005080 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.082913 45398 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.083004: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.083082 45150 service.cc:146] XLA service 0x2aac4c005090 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.083120 45150 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.083657: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.084327: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.084367 45251 service.cc:146] XLA service 0x2aac4c004570 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.084403 45251 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.084402 45349 service.cc:146] XLA service 0x2aac4c004920 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.084437 45349 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.084513 45378 service.cc:146] XLA service 0x2aac4c003f50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.084553 45378 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.084975 45394 service.cc:146] XLA service 0x2aac4c004e80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.085012 45394 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.085166 45215 service.cc:146] XLA service 0x2aac4c0049b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.085202 45215 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.085450 45216 service.cc:146] XLA service 0x2aac4c0049e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.085488 45216 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.085537 45263 service.cc:146] XLA service 0x2aac4c001f30 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.085573 45263 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.085980 45320 service.cc:146] XLA service 0x2aac4c0050b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.086014 45320 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.086179 45246 service.cc:146] XLA service 0x2aac4c004840 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.086216 45246 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.086322 45338 service.cc:146] XLA service 0x2aac4c004c70 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.086356 45338 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.086492 45391 service.cc:146] XLA service 0x2aac4c0047d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.086529 45391 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.087211 45195 service.cc:146] XLA service 0x2aac4c000d50 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.087268 45195 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.088190: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.088542: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.088749 45368 service.cc:146] XLA service 0x2aac4c0045e0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.088788 45368 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.088813 45311 service.cc:146] XLA service 0x2aac4c004880 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.088852 45311 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.088985 45327 service.cc:146] XLA service 0x2aac4c002b30 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.089025 45327 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.089370: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.089407 45342 service.cc:146] XLA service 0x2aac4c004ec0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.089447 45342 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.089479: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.090699 45176 service.cc:146] XLA service 0x2aac4c004a00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.090737 45176 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.090770 45179 service.cc:146] XLA service 0x2aac4c004c90 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.090808 45179 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.091216 45399 service.cc:146] XLA service 0x2aac4c002410 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.091252 45399 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.091938 45270 service.cc:146] XLA service 0x2aac4c004830 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.091974 45270 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.092231 45175 service.cc:146] XLA service 0x2aac4c004a80 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.092267 45175 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.092923 45330 service.cc:146] XLA service 0x2aac4c0049a0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.092957 45330 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.094118 45199 service.cc:146] XLA service 0x2aac4c004e10 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.094154 45199 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.094729 45353 service.cc:146] XLA service 0x2aac4c004540 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.094764 45353 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.095083 45363 service.cc:146] XLA service 0x2aac4c0049b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.095120 45363 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.095163 45196 service.cc:146] XLA service 0x2aac4c0049b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.095201 45196 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.095528: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.098040 45153 service.cc:146] XLA service 0x2aac4c004830 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.098081 45153 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.098442: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.098523 45217 service.cc:146] XLA service 0x2aac4c004ef0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.098556 45217 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1740504043.099121 45165 service.cc:146] XLA service 0x2aac4c005070 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1740504043.099154 45165 service.cc:154] StreamExecutor device (0): Tesla V100-PCIE-32GB, Compute Capability 7.0
2025-02-25 12:20:43.101174: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.103436: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.104247 45335 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.104247 45308 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.104997: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.105209 45290 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.108098: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.108606: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.108980 45260 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.110559 45397 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.110592 45377 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.111335: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.111712 45228 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.111831: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.113245: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.113639 45147 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.113794: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.113893: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.113998: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.114213: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.114726: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.114925: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.115236 45384 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.115237 45357 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.115238 45206 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.116160: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.116456: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.116715 45239 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.117233: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.119177: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.119657: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.120076: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.120252: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.121256 45393 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.121877: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.122048: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.122163: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.122200: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.122512: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.122805: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.122828: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.123250: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.123594: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.123631: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.124015: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.124179: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
2025-02-25 12:20:43.124295: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 90300
I0000 00:00:1740504043.125643 45316 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.129880 45360 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.131498 45370 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.133423 45372 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.138466 45398 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.140242 45150 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.142217 45251 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.143799 45349 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.144198 45394 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.144198 45378 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.145278 45216 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.145410 45215 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.145409 45246 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.145409 45320 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.146095 45263 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.146873 45391 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.146911 45338 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.149540 45195 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.150485 45368 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.150488 45311 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.151352 45342 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.152620 45327 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.153095 45176 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154196 45353 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154264 45179 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154444 45270 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154443 45399 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154489 45330 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154537 45199 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154642 45363 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154877 45175 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154951 45196 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.154958 45153 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.155134 45217 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
I0000 00:00:1740504043.157337 45165 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
2025-02-25 12:20:43.198958: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.201784: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.201785: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.205812: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.207968: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.208105: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.208392: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.208775: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.210315: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.211711: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.211932: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.212664: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.212720: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.216281: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.223043: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.223137: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.223911: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.229461: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.229895: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.232630: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.235277: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.238965: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.240194: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.240309: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.241083: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.241720: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.242213: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.242223: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.242307: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.242676: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.246885: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.248077: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.249295: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.251119: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.251991: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.252336: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.252597: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.254989: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.255082: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.256202: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.257332: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.258142: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.258764: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.258828: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.259102: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.261152: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.261805: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:43.262754: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var
MLIR_CRASH_REPRODUCER_DIRECTORYto enable.2025-02-25 12:20:52.453594: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.457344: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.488480: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.509892: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.536919: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.562077: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.564450: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.583105: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.586236: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.586701: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.605841: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.614049: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.614796: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.623395: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.625484: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.625801: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.632116: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.634038: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.638623: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.644994: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.646025: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.650715: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.653500: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.654306: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.655822: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.666239: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.669595: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.670174: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.670732: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_3', 12 bytes spill stores, 12 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_10', 12 bytes spill stores, 12 bytes spill loads
2025-02-25 12:20:52.682820: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.683210: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.685177: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.687250: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.691435: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.692391: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_13', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.692810: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.713832: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.732090: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.775837: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.798386: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.818684: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.823502: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.838274: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.839199: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.839697: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.852951: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.872553: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
2025-02-25 12:20:52.875827: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_15', 16 bytes spill stores, 16 bytes spill loads
ptxas warning : Registers are spilled to local memory in function 'loop_add_fusion_14', 16 bytes spill stores, 16 bytes spill loads
Per MPI rank memory allocation (min/avg/max) = 3.147 | 3.147 | 3.148 Mbytes
Step Temp PotEng TotEng Lx Ly Lz CPU
0 1300 -80572.245 -78555.955 61.518166 61.518166 61.518166 0
100 632.97661 -79437.032 -78455.29 61.518166 61.518166 61.518166 28.781316
200 740.94029 -79406.397 -78257.204 61.518166 61.518166 61.518166 57.568795
Beta Was this translation helpful? Give feedback.
All reactions