[Bug] distill wan 2.1, video is all noise

[sccao123]

Describe the bug

Hi, I noticed that you released the code for the distilled WAN model, so I tried running it. Since I’m using 7 A100 GPUs, I set nproc_per_node=7 and also reduced num_latent_t from 32 to 28 to save GPU memory. However, the output videos (those from the validation phase) were completely filled with noise. I’d like to know whether this is due to an issue in the current code or caused by my adjustments to these two parameters.

Reproduction

ash scripts/distill/distill_wan.sh. Wan2.1-T2V-1.3B-Diffusers.

Environment

Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (GCC) 8.2.0
Clang version: 3.8.0 (tags/RELEASE_380/final)
CMake version: version 3.16.0
Libc version: glibc-2.27

Python version: 3.12.11 | packaged by conda-forge | (main, Jun 4 2025, 14:45:31) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.10.0-1.0.0.28-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 525.125.06
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 160
On-line CPU(s) list: 0-159
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 4
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 3100.257
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 28160K
NUMA node0 CPU(s): 0-19,80-99
NUMA node1 CPU(s): 20-39,100-119
NUMA node2 CPU(s): 40-59,120-139
NUMA node3 CPU(s): 60-79,140-159
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku md_clear flush_l1d

Versions of relevant libraries:
[pip3] accelerate==1.0.1
[pip3] numpy==2.1.2
[pip3] nvidia-cublas-cu11==11.11.3.6
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu11==11.8.87
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu11==11.8.89
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu11==11.8.89
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu11==9.1.0.70
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu11==10.9.0.58
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu11==10.3.0.86
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu11==11.4.1.48
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu11==11.7.5.86
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.575.51
[pip3] nvidia-nccl-cu11==2.20.5
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu11==11.8.86
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] peft==0.13.2
[pip3] pyzmq==26.4.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.52.4
[pip3] triton==3.2.0
[conda] accelerate 1.0.1 pypi_0 pypi
[conda] numpy 2.1.2 pypi_0 pypi
[conda] nvidia-cublas-cu11 11.11.3.6 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu11 11.8.87 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu11 11.8.89 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu11 11.8.89 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cudnn-cu11 9.1.0.70 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu11 10.9.0.58 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
[conda] nvidia-curand-cu11 10.3.0.86 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
[conda] nvidia-cusolver-cu11 11.4.1.48 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
[conda] nvidia-cusparse-cu11 11.7.5.86 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi
[conda] nvidia-ml-py 12.575.51 pypi_0 pypi
[conda] nvidia-nccl-cu11 2.20.5 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-nvtx-cu11 11.8.86 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
[conda] peft 0.13.2 pypi_0 pypi
[conda] pyzmq 26.4.0 pypi_0 pypi
[conda] torch 2.6.0 pypi_0 pypi
[conda] torchaudio 2.6.0 pypi_0 pypi
[conda] torchvision 0.21.0 pypi_0 pypi
[conda] transformers 4.52.4 pypi_0 pypi
[conda] triton 3.2.0 pypi_0 pypi
FastVideo Version:
FastVideo Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 CPU Affinity NUMA Affinity
GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 PXB SYS SYS 0-19,80-99 0
GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 PXB SYS SYS 0-19,80-99 0
GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 SYS NODE PXB 20-39,100-119 1
GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 SYS NODE PXB 20-39,100-119 1
GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 SYS SYS SYS 40-59,120-139 2
GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 SYS SYS SYS 40-59,120-139 2
GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 SYS SYS SYS 60-79,140-159 3
GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X SYS SYS SYS 60-79,140-159 3
NIC0 PXB PXB SYS SYS SYS SYS SYS SYS X SYS SYS
NIC1 SYS SYS NODE NODE SYS SYS SYS SYS SYS X NODE
NIC2 SYS SYS PXB PXB SYS SYS SYS SYS SYS NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2

CUDNN_VERSION=8.1.1.33
LD_LIBRARY_PATH=/root/paddlejob/workspace/env_run/chenjinwen/env/cuda-11.8/lib64:/root/paddlejob/workspace/env_run/chenjinwen/env/cudnn-linux-x86_64-8.7.0.84_cuda11-archive/lib:/home/opt/gpuproxy/lib64:/usr/local/lib:/usr/local/x86_64-pc-linux-gnu/lib:/home/opt/nvidia_lib:/usr/local/cuda/lib64:/usr/lib64:/usr/local/lib:/usr/lib/x86_64-linux-gnu/
NCCL_IB_GID_INDEX=3
NCCL_IB_ADAPTIVE_ROUTING=1
NCCL_IB_DISABLE=0
NVIDIA_VISIBLE_GPUS_SLOT=2,3,4,5,6,7,0,1
NVIDIA_TOOLS=/home/opt/cuda_tools
NCCL_DEBUG_FILE=/root/paddlejob/workspace/log/nccl.%h.%p.log
NCCL_IB_CONNECT_RETRY_CNT=15
NVIDIA_VISIBLE_DEVICES=GPU-4ac5d0a2-6b9e-9b51-ce13-2c24d2fda37b,GPU-763c8a98-eb74-05fd-67a7-fdddee6aab91,GPU-cda43ad3-6a26-775e-ff0b-188ad0c656b5,GPU-d6efe4ad-4f1e-30e8-c1da-4250beac9640,GPU-e71c8db0-2c17-cea5-526d-86179e85ca51,GPU-39316292-d8d3-cebb-fd15-bbe60737578c,GPU-7b5e1f8c-3288-6bba-6c97-69284916cf6b,GPU-8145700b-60b5-db98-56fc-7f77521e735f
NCCL_IB_TIMEOUT=22
NCCL_VERSION=2.8.4
NCCL_IB_CUDA_SUPPORT=0
NVIDIA_GDRCOPY=enabled
NVIDIA_VISIBLE_GPUS_UUID=GPU-4ac5d0a2-6b9e-9b51-ce13-2c24d2fda37b,GPU-763c8a98-eb74-05fd-67a7-fdddee6aab91,GPU-cda43ad3-6a26-775e-ff0b-188ad0c656b5,GPU-d6efe4ad-4f1e-30e8-c1da-4250beac9640,GPU-e71c8db0-2c17-cea5-526d-86179e85ca51,GPU-39316292-d8d3-cebb-fd15-bbe60737578c,GPU-7b5e1f8c-3288-6bba-6c97-69284916cf6b,GPU-8145700b-60b5-db98-56fc-7f77521e735f
NVIDIA_LIB=/usr/local/nvidia/lib64
NCCL_P2P_DISABLE=0
CUDA_VERSION=11.2.1
NCCL_IB_QPS_PER_CONNECTION=2
NCCL_ERROR_FILE=/root/paddlejob/workspace/log/err.%h.%p.log
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_REQUIRE_CUDA=cuda>=11.2 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441 driver>=450,driver<451
NCCL_DEBUG_SUBSYS=INIT,ENV,GRAPH
NCCL_DEBUG=INFO
NCCL_SOCKET_IFNAME=xgbe1
CUDA_MODULE_LOADING=LAZY
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root

[sccao123]

Besides, I changed another place. At first I got error:[rank3]: Traceback (most recent call last):
[rank3]: File "/root/paddlejob/workspace/env_run/sc_code/FastVideo/fastvideo/distill.py", line 815, in
[rank3]: main(args)
[rank3]: File "/root/paddlejob/workspace/env_run/sc_code/FastVideo/fastvideo/distill.py", line 493, in main
[rank3]: loss, grad_norm, pred_norm = distill_one_step(
[rank3]: ^^^^^^^^^^^^^^^^^
[rank3]: File "/root/paddlejob/workspace/env_run/sc_code/FastVideo/fastvideo/distill.py", line 146, in distill_one_step
[rank3]: model_pred = transformer(**teacher_kwargs)[0]
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/fastvideo/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/fastvideo/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/fastvideo/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 864, in forward
[rank3]: output = self._fsdp_wrapped_module(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/fastvideo/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/fastvideo/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/paddlejob/workspace/env_run/sc_code/FastVideo/fastvideo/models/wan_hf/modeling_wan.py", line 442, in forward
[rank3]: temb, timestep_proj, encoder_hidden_states, encoder_hidden_states_image = self.condition_embedder(
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/fastvideo/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
[rank3]: return self._call_impl(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/miniconda3/envs/fastvideo/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
[rank3]: return forward_call(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/root/paddlejob/workspace/env_run/sc_code/FastVideo/fastvideo/models/wan_hf/modeling_wan.py", line 169, in forward
[rank3]: time_embedder_dtype = next(iter(self.time_embedder.parameters())).dtype
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: StopIteration. Then I commented fastvideo/models/wan_hf/modeling_wan.py, line 169-171. Then there is no error.

[sccao123]

maybe I was using wrong dataset? I just use the dataset for distilling hunyuan, but its resolution is not suitable for wan? If so, could you please share the dataset to distill wan? Thank you again.

[sccao123]

should I use /scripts/preprocess/preprocess_wan_data.sh to get the dataset? I will try it and update the result later.

[sccao123]

For this time, I use the correct dataset but I got NCCL connection timeout. Why is that? part of output: [rank4]:[E611 18:54:35.586722984 ProcessGroupNCCL.cpp:629] [Rank 4] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=93, OpType=ALLREDUCE, NumelIn=16, NumelOut=16, Timeout(ms)=600000) ran for 600021 milliseconds before timing out.
[rank4]:[E611 18:54:35.586898265 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 4] failure detected by watchdog at work sequence id: 93 PG status: last enqueued work: 95, last completed work: 92
[rank4]:[E611 18:54:35.586913431 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank1]:[E611 18:54:36.468933402 ProcessGroupNCCL.cpp:629] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=93, OpType=ALLREDUCE, NumelIn=336, NumelOut=336, Timeout(ms)=600000) ran for 600014 milliseconds before timing out.
[rank5]:[E611 18:54:36.469029993 ProcessGroupNCCL.cpp:629] [Rank 5] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=93, OpType=ALLREDUCE, NumelIn=336, NumelOut=336, Timeout(ms)=600000) ran for 600064 milliseconds before timing out. ......

[sccao123]

Then I copy the content of the dataset to scale the size of dataset. Then I observed a very strange phenomenon - the NCCL communication timeout occurs later than before. As the dataset size increases, the step at which the NCCL timeout occurs becomes progressively later. I don't understand why this is happening. I'm currently copying the original dataset to 600 times its original size and running the code. Initially, I mistakenly used the HD-Mixkit-Finetune-Hunyuan dataset for distilling the Wan model, and at that time, no NCCL communication issues occurred. Now that I'm using the correct dataset, this problem has emerged. Do you have any insights into why this might be happening?

[sccao123]

But now I feel something is wrong with the distilling process. This time I used all the default parameters from distill_wan.sh without any modifications. Then I noticed that during each validation phase, the 8 generated videos are identical in content - I mean, while it produces 8 different videos each time, the videos with the same sequence number are identical across validations. As shown in the image (though not just noise), they contain no substantive content. Is this due to a code issue or because I used too much duplicated data for training? Could it be that Image-Vid-Finetune-Src is fundamentally unsuitable for training, and using it would inherently cause this problem?

[SolitaryThinker]

Hi, thank you for the efforts to try the wan distillation code. We are actively working on adding and verifying distillation in V1 as well as using the scripts (v0) you tried as a reference. I will update here once it is ready. You can see our ongoing efforts for an updated distillation in this PR: #471

[sccao123]

OK, thank you for your effort. I will also try to figure out whether there is some bug in v0 script.

[sccao123]

At first, I think there must be something wrong with sample_validation_video function in ./fastvideo/utils/validation.py. Using the distilled checkpoint and v1_inference_wan.sh, I could generate valid video. However, the distilled model demonstrates inferior stability and output quality compared to the original model when generating videos with 16 or 8 sampling steps. The prompts I'm using are included in the training set of FastVideo/Image-Vid-Finetune-Src. Is this degradation caused by dataset issues or inherent code problems?

[SolitaryThinker]

Our experience is that distillation with PCM still causes quite a bit of degradation, so the lower quality you observe is expected. We are also exploring other distillation recipes in V1 actively. Do you mind sharing some of your generated video using the distilled checkpoint? Could you also share the settings you used? Thank you

[sccao123]

Er.Lang.Shen.wearing.an.elaborate.golden.headpiece.and.a.flowing.white.robe.stands.prominently.in.mp4

Er.Lang.Shen.wearing.an.elaborate.golden.headpiece.and.a.flowing.white.robe.stands.prominently.in.1.mp4

Er.Lang.Shen.wearing.an.elaborate.golden.headpiece.and.a.flowing.white.robe.stands.prominently.in.3.mp4

Er.Lang.Shen.wearing.an.elaborate.golden.headpiece.and.a.flowing.white.robe.stands.prominently.in.2.mp4

First, thank you for your prompt response. What I want to emphasize is that the distilled checkpoint performs worse even with identical step counts. The two videos above were both generated at seed 42 with num-inference-steps=16 - the first using the original model, the second using the distilled version. The last two videos were both generated at seed 42 with num-inference-steps=8 - the first using the original model, the second using the distilled version. It's evident that the distilled model performs significantly worse than the original model in low-step inference scenarios. The complete command was: fastvideo generate
--model-path $MODEL_BASE
--sp-size $num_gpus
--tp-size $num_gpus
--num-gpus $num_gpus
--height 480
--width 832
--num-frames 81
--num-inference-steps 16
--fps 16
--guidance-scale 3.0
--prompt "Er Lang Shen, wearing an elaborate golden headpiece and a flowing white robe, stands prominently in the foreground against a misty backdrop. Their expression is calm yet assertive, as they wield a gracefully carved staff, pointing it forward with purpose. Behind Er Lang Shen, Ju Ling Shen, a colossal figure looms, partially obscured by the fog, with intricate armor that gleams faintly in the dim light. This giant seems almost frozen in time, casting an imposing shadow that stretches across the scene. Er Lang Shen subtly tilts their head, a small smile playing on their lips, as if confident in the presence of their formidable ally. The wind softly sways their long, dark hair, adding a dynamic softness to their poised stance. As Er Lang Shen steps sideways, they maintain a steady gaze, their eyes full of resolve, which contrasts with the serene yet powerful aura of the setting. The intricate patterns on their robe reflect their status and reinforce the mysterious, mystical atmosphere that envelops the moment."
--negative-prompt "Bright tones, overexposed, static, blurred details, subtitles, style, works, paintings, images, static, overall gray, worst quality, low quality, JPEG compression residue, ugly, incomplete, extra fingers, poorly drawn hands, poorly drawn faces, deformed, disfigured, misshapen limbs, fused fingers, still picture, messy background, three legs, many people in the background, walking backwards"
--seed 42
--output-path outputs_video/. I'd like to know which dataset was used during your code testing phase. Would it be possible to share the distilled checkpoint? Thank you again for your assistance. The environment setting is: Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (GCC) 8.2.0
Clang version: 3.8.0 (tags/RELEASE_380/final)
CMake version: version 3.16.0
Libc version: glibc-2.27

Python version: 3.12.11 | packaged by conda-forge | (main, Jun 4 2025, 14:45:31) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.10.0-1.0.0.28-x86_64-with-glibc2.27
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-40GB
GPU 1: NVIDIA A100-SXM4-40GB
GPU 2: NVIDIA A100-SXM4-40GB
GPU 3: NVIDIA A100-SXM4-40GB
GPU 4: NVIDIA A100-SXM4-40GB
GPU 5: NVIDIA A100-SXM4-40GB
GPU 6: NVIDIA A100-SXM4-40GB
GPU 7: NVIDIA A100-SXM4-40GB

Nvidia driver version: 525.125.06
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.1
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 160
On-line CPU(s) list: 0-159
Thread(s) per core: 2
Core(s) per socket: 20
Socket(s): 4
NUMA node(s): 4
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
Stepping: 4
CPU MHz: 3100.262
CPU max MHz: 3700.0000
CPU min MHz: 1000.0000
BogoMIPS: 4800.00
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 28160K
NUMA node0 CPU(s): 0-19,80-99
NUMA node1 CPU(s): 20-39,100-119
NUMA node2 CPU(s): 40-59,120-139
NUMA node3 CPU(s): 60-79,140-159
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku md_clear flush_l1d

Versions of relevant libraries:
[pip3] accelerate==1.0.1
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-ml-py==12.575.51
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] peft==0.13.2
[pip3] torch==2.6.0
[pip3] torchdata==0.11.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.52.4
[pip3] triton==3.2.0
[conda] accelerate 1.0.1 pypi_0 pypi
[conda] numpy 2.2.6 pypi_0 pypi
[conda] nvidia-cublas-cu12 12.4.5.8 pypi_0 pypi
[conda] nvidia-cuda-cupti-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-nvrtc-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cuda-runtime-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-cudnn-cu12 9.1.0.70 pypi_0 pypi
[conda] nvidia-cufft-cu12 11.2.1.3 pypi_0 pypi
[conda] nvidia-curand-cu12 10.3.5.147 pypi_0 pypi
[conda] nvidia-cusolver-cu12 11.6.1.9 pypi_0 pypi
[conda] nvidia-cusparse-cu12 12.3.1.170 pypi_0 pypi
[conda] nvidia-cusparselt-cu12 0.6.2 pypi_0 pypi
[conda] nvidia-ml-py 12.575.51 pypi_0 pypi
[conda] nvidia-nccl-cu12 2.21.5 pypi_0 pypi
[conda] nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
[conda] nvidia-nvtx-cu12 12.4.127 pypi_0 pypi
[conda] peft 0.13.2 pypi_0 pypi
[conda] torch 2.6.0 pypi_0 pypi
[conda] torchdata 0.11.0 pypi_0 pypi
[conda] torchvision 0.21.0 pypi_0 pypi
[conda] transformers 4.52.4 pypi_0 pypi
[conda] triton 3.2.0 pypi_0 pypi
FastVideo Version:
FastVideo Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 CPU Affinity NUMA Affinity
GPU0 X NV12 NV12 NV12 NV12 NV12 NV12 NV12 PXB SYS SYS 0-19,80-99 0
GPU1 NV12 X NV12 NV12 NV12 NV12 NV12 NV12 PXB SYS SYS 0-19,80-99 0
GPU2 NV12 NV12 X NV12 NV12 NV12 NV12 NV12 SYS NODE PXB 20-39,100-119 1
GPU3 NV12 NV12 NV12 X NV12 NV12 NV12 NV12 SYS NODE PXB 20-39,100-119 1
GPU4 NV12 NV12 NV12 NV12 X NV12 NV12 NV12 SYS SYS SYS 40-59,120-139 2
GPU5 NV12 NV12 NV12 NV12 NV12 X NV12 NV12 SYS SYS SYS 40-59,120-139 2
GPU6 NV12 NV12 NV12 NV12 NV12 NV12 X NV12 SYS SYS SYS 60-79,140-159 3
GPU7 NV12 NV12 NV12 NV12 NV12 NV12 NV12 X SYS SYS SYS 60-79,140-159 3
NIC0 PXB PXB SYS SYS SYS SYS SYS SYS X SYS SYS
NIC1 SYS SYS NODE NODE SYS SYS SYS SYS SYS X NODE
NIC2 SYS SYS PXB PXB SYS SYS SYS SYS SYS NODE X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2

CUDNN_VERSION=8.1.1.33
LD_LIBRARY_PATH=/root/paddlejob/workspace/env_run/chenjinwen/env/cuda-11.8/lib64:/root/paddlejob/workspace/env_run/chenjinwen/env/cudnn-linux-x86_64-8.7.0.84_cuda11-archive/lib:/home/opt/gpuproxy/lib64:/usr/local/lib:/usr/local/x86_64-pc-linux-gnu/lib:/home/opt/nvidia_lib:/usr/local/cuda/lib64:/usr/lib64:/usr/local/lib:/usr/lib/x86_64-linux-gnu/
NCCL_IB_GID_INDEX=3
NCCL_IB_ADAPTIVE_ROUTING=1
NCCL_IB_DISABLE=0
NVIDIA_VISIBLE_GPUS_SLOT=2,3,4,5,6,7,0,1
NVIDIA_TOOLS=/home/opt/cuda_tools
NCCL_CUMULATIVE_ITERATIONS=4
NCCL_DEBUG_FILE=/root/paddlejob/workspace/log/nccl.%h.%p.log
NCCL_IB_CONNECT_RETRY_CNT=15
NVIDIA_VISIBLE_DEVICES=GPU-4ac5d0a2-6b9e-9b51-ce13-2c24d2fda37b,GPU-763c8a98-eb74-05fd-67a7-fdddee6aab91,GPU-cda43ad3-6a26-775e-ff0b-188ad0c656b5,GPU-d6efe4ad-4f1e-30e8-c1da-4250beac9640,GPU-e71c8db0-2c17-cea5-526d-86179e85ca51,GPU-39316292-d8d3-cebb-fd15-bbe60737578c,GPU-7b5e1f8c-3288-6bba-6c97-69284916cf6b,GPU-8145700b-60b5-db98-56fc-7f77521e735f
NCCL_IB_TIMEOUT=22
NCCL_ASYNC_ERROR_HANDLING=1
NCCL_VERSION=2.8.4
NCCL_IB_CUDA_SUPPORT=0
NVIDIA_GDRCOPY=enabled
NVIDIA_VISIBLE_GPUS_UUID=GPU-4ac5d0a2-6b9e-9b51-ce13-2c24d2fda37b,GPU-763c8a98-eb74-05fd-67a7-fdddee6aab91,GPU-cda43ad3-6a26-775e-ff0b-188ad0c656b5,GPU-d6efe4ad-4f1e-30e8-c1da-4250beac9640,GPU-e71c8db0-2c17-cea5-526d-86179e85ca51,GPU-39316292-d8d3-cebb-fd15-bbe60737578c,GPU-7b5e1f8c-3288-6bba-6c97-69284916cf6b,GPU-8145700b-60b5-db98-56fc-7f77521e735f
NVIDIA_LIB=/usr/local/nvidia/lib64
NCCL_MIN_NCHANNELS=4
NCCL_P2P_DISABLE=0
CUDA_VERSION=11.2.1
NCCL_IB_QPS_PER_CONNECTION=2
NCCL_ERROR_FILE=/root/paddlejob/workspace/log/err.%h.%p.log
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_REQUIRE_CUDA=cuda>=11.2 brand=tesla,driver>=418,driver<419 brand=tesla,driver>=440,driver<441 driver>=450,driver<451
NCCL_DEBUG_SUBSYS=INIT,ENV,GRAPH
NCCL_DEBUG=TRACE
NCCL_SOCKET_IFNAME=xgbe1
CUDA_MODULE_LOADING=LAZY
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root.

[sccao123]

To clarify further: I kept all parameters in distill_wan.sh unchanged, but replicated the dataset 800 times to avoid NCCL communication timeouts. I suspect the NCCL timeouts may be related to post-epoch processing – this would explain why increasing the dataset size prevents the error. Additionally, I installed the fastvideo library from source to prevent the use of outdated code.

[SolitaryThinker]

Our V1's wan2.1 implementation is slightly different than HF's (used by v0 distillation). But I don't think it should cause this big of a quality drop. cc @jzhang38 @Eigensystem

[sccao123]

one question about num_latent_t, in #383 , [BrianChen1129] said that frames_num should be 4*(num_latent_t-1) + 1. If so, for WAN, num_latent_t should be 21. However, in distill_wan.sh, it is: --num_latent_t 32. Should we change this to 21?

[sccao123]

I have another question regarding distilling the WAN model using high-resolution videos. Would it work if I use 1080p videos for distillation, or could I directly apply the HD-Mixkit-Finetune-Hunyuan dataset? Do I need to adjust the num_height and num_width parameters in distill_wan.sh for this? Additionally, in previous discussions you mentioned avoiding Image-Vid-Finetune-Src for distillation - was this due to insufficient data volume or dataset incompatibility issues?