[Preprocess] [feat] Support HunyuanVideo Model #754

JiayiZhangA · 2025-08-22T10:42:03Z

No description provided.

gemini-code-assist

Summary of Changes

Hello @JiayiZhangA, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces support for the HunyuanVideo model within the FastVideo framework, primarily by adding new preprocessing pipelines and a utility script. It enables the preparation of datasets for both text-to-video and image-to-video tasks using the Hunyuan model, alongside minor improvements to data validation and dataset loading mechanisms.

Highlights

Hunyuan Preprocessing Script: A new bash script (preprocess_hunyuan_data_t2v.sh) is added to facilitate the preprocessing of Hunyuan text-to-video datasets, defining model paths, dataset locations, and various video parameters.
Dedicated Hunyuan Pipelines: New Python classes (PreprocessPipelineI2V and PreprocessPipelineT2V) are introduced to define specific preprocessing workflows for Hunyuan models, handling stages like text transformation, prompt encoding, and video encoding for both image-to-video and text-to-video tasks.
Robust Data Validation: The data validation logic in preprocessing components has been enhanced to correctly handle cases where the 'fps' key might be absent from the input batch, improving the robustness of the data pipeline.
Dataset Loading Enhancement: The dataset loading mechanism has been refined to explicitly specify the 'data' field when loading training datasets, ensuring accurate data retrieval for the preprocessing workflow.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for Hunyuan, including preprocessing pipelines for text-to-video and image-to-video tasks. The changes include a new preprocessing script, new pipeline definitions, and updates to the data validation and loading logic.

My review focuses on improving script robustness, reducing code duplication in the new pipelines for better maintainability, and removing leftover debugging code. Overall, the changes look good and add valuable functionality.

gemini-code-assist · 2025-08-22T10:43:44Z

examples/training/finetune/wan_t2v_1.3B/crush_smol/preprocess_hunyuan_data_t2v.sh

+#!/bin/bash
+
+GPU_NUM=1 # 2,4,8
+MODEL_PATH="hunyuanvideo-community/HunyuanVideo"
+DATASET_PATH="/FastVideo/data/mini_i2v_dataset/crush-smol_raw"
+OUTPUT_DIR="/FastVideo/data/mini_i2v_dataset/crush-smol_processed_t2v_hunyuan/"
+
+torchrun --nproc_per_node=$GPU_NUM \
+    -m fastvideo.pipelines.preprocess.v1_preprocessing_new \
+    --model_path $MODEL_PATH \
+    --mode preprocess \
+    --workload_type t2v \
+    --preprocess.dataset_path $DATASET_PATH \
+    --preprocess.dataset_output_dir $OUTPUT_DIR \
+    --preprocess.preprocess_video_batch_size 2 \
+    --preprocess.dataloader_num_workers 0 \
+    --preprocess.max_height 480 \
+    --preprocess.max_width 832 \
+    --preprocess.num_frames 77 \
+    --preprocess.train_fps 16 \
+    --preprocess.samples_per_file 8 \
+    --preprocess.flush_frequency 8 \
+    --preprocess.video_length_tolerance_range 5


For improved robustness and to prevent potential issues with paths containing spaces or special characters, it's recommended to:

Add set -euo pipefail at the beginning of the script. This will make the script exit on any error, treat unset variables as an error, and fail if any command in a pipeline fails.

Double-quote all variable expansions to prevent word splitting and globbing.

These changes will make the script more reliable and safer to run.

#!/bin/bash set -euo pipefail GPU_NUM=1 # 2,4,8 MODEL_PATH="hunyuanvideo-community/HunyuanVideo" DATASET_PATH="/FastVideo/data/mini_i2v_dataset/crush-smol_raw" OUTPUT_DIR="/FastVideo/data/mini_i2v_dataset/crush-smol_processed_t2v_hunyuan/" torchrun --nproc_per_node="$GPU_NUM" \ -m fastvideo.pipelines.preprocess.v1_preprocessing_new \ --model_path "$MODEL_PATH" \ --mode preprocess \ --workload_type t2v \ --preprocess.dataset_path "$DATASET_PATH" \ --preprocess.dataset_output_dir "$OUTPUT_DIR" \ --preprocess.preprocess_video_batch_size 2 \ --preprocess.dataloader_num_workers 0 \ --preprocess.max_height 480 \ --preprocess.max_width 832 \ --preprocess.num_frames 77 \ --preprocess.train_fps 16 \ --preprocess.samples_per_file 8 \ --preprocess.flush_frequency 8 \ --preprocess.video_length_tolerance_range 5

gemini-code-assist · 2025-08-22T10:43:44Z

fastvideo/workflow/preprocess/components.py

+        print("-------------------------------")
+        print(batch)


These print statements appear to be for debugging purposes. They should be removed before merging to avoid polluting the logs and potential performance overhead.

gemini-code-assist · 2025-08-22T10:43:44Z

fastvideo/pipelines/preprocess/hunyuan/hunyuan_preprocess_pipelines.py

+class PreprocessPipelineI2V(ComposedPipelineBase):
+    _required_config_modules = [
+        "image_encoder", "image_processor", "text_encoder", "tokenizer", "vae"
+    ]
+
+    def create_pipeline_stages(self, fastvideo_args: FastVideoArgs):
+        assert fastvideo_args.preprocess_config is not None
+        self.add_stage(stage_name="text_transform_stage",
+                       stage=TextTransformStage(
+                           cfg_uncondition_drop_rate=fastvideo_args.
+                           preprocess_config.training_cfg_rate,
+                           seed=fastvideo_args.preprocess_config.seed,
+                       ))
+        self.add_stage(stage_name="prompt_encoding_stage",
+                       stage=TextEncodingStage(
+                           text_encoders=[self.get_module("text_encoder")],
+                           tokenizers=[self.get_module("tokenizer")],
+                       ))
+        self.add_stage(
+            stage_name="video_transform_stage",
+            stage=VideoTransformStage(
+                train_fps=fastvideo_args.preprocess_config.train_fps,
+                num_frames=fastvideo_args.preprocess_config.num_frames,
+                max_height=fastvideo_args.preprocess_config.max_height,
+                max_width=fastvideo_args.preprocess_config.max_width,
+                do_temporal_sample=fastvideo_args.preprocess_config.
+                do_temporal_sample,
+            ))
+        if (self.get_module("image_encoder") is not None
+                and self.get_module("image_processor") is not None):
+            self.add_stage(
+                stage_name="image_encoding_stage",
+                stage=ImageEncodingStage(
+                    image_encoder=self.get_module("image_encoder"),
+                    image_processor=self.get_module("image_processor"),
+                ))
+        self.add_stage(stage_name="image_vae_encoding_stage",
+                       stage=ImageVAEEncodingStage(
+                           vae=self.get_module("vae"), ))
+        self.add_stage(stage_name="video_encoding_stage",
+                       stage=EncodingStage(vae=self.get_module("vae"), ))
+
+
+class PreprocessPipelineT2V(ComposedPipelineBase):
+    _required_config_modules = ["text_encoder", "tokenizer", "vae"]
+
+    def create_pipeline_stages(self, fastvideo_args: FastVideoArgs):
+        assert fastvideo_args.preprocess_config is not None
+        self.add_stage(stage_name="text_transform_stage",
+                       stage=TextTransformStage(
+                           cfg_uncondition_drop_rate=fastvideo_args.
+                           preprocess_config.training_cfg_rate,
+                           seed=fastvideo_args.preprocess_config.seed,
+                       ))
+        self.add_stage(stage_name="prompt_encoding_stage",
+                       stage=TextEncodingStage(
+                           text_encoders=[self.get_module("text_encoder")],
+                           tokenizers=[self.get_module("tokenizer")],
+                       ))
+        self.add_stage(
+            stage_name="video_transform_stage",
+            stage=VideoTransformStage(
+                train_fps=fastvideo_args.preprocess_config.train_fps,
+                num_frames=fastvideo_args.preprocess_config.num_frames,
+                max_height=fastvideo_args.preprocess_config.max_height,
+                max_width=fastvideo_args.preprocess_config.max_width,
+                do_temporal_sample=fastvideo_args.preprocess_config.
+                do_temporal_sample,
+            ))
+        self.add_stage(stage_name="video_encoding_stage",
+                       stage=EncodingStage(vae=self.get_module("vae"), ))


The classes PreprocessPipelineI2V and PreprocessPipelineT2V share a significant amount of duplicated code. To improve maintainability and reduce redundancy, you can refactor the common logic into a helper function or a base class.

Here's a suggested refactoring using a helper function to build the common pipeline stages.

def _add_base_stages(pipeline: ComposedPipelineBase, fastvideo_args: FastVideoArgs): """Helper to add common preprocessing stages.""" assert fastvideo_args.preprocess_config is not None pipeline.add_stage(stage_name="text_transform_stage", stage=TextTransformStage( cfg_uncondition_drop_rate=fastvideo_args. preprocess_config.training_cfg_rate, seed=fastvideo_args.preprocess_config.seed, )) pipeline.add_stage(stage_name="prompt_encoding_stage", stage=TextEncodingStage( text_encoders=[pipeline.get_module("text_encoder")], tokenizers=[pipeline.get_module("tokenizer")], )) pipeline.add_stage( stage_name="video_transform_stage", stage=VideoTransformStage( train_fps=fastvideo_args.preprocess_config.train_fps, num_frames=fastvideo_args.preprocess_config.num_frames, max_height=fastvideo_args.preprocess_config.max_height, max_width=fastvideo_args.preprocess_config.max_width, do_temporal_sample=fastvideo_args.preprocess_config. do_temporal_sample, )) class PreprocessPipelineI2V(ComposedPipelineBase): _required_config_modules = [ "image_encoder", "image_processor", "text_encoder", "tokenizer", "vae" ] def create_pipeline_stages(self, fastvideo_args: FastVideoArgs): _add_base_stages(self, fastvideo_args) if (self.get_module("image_encoder") is not None and self.get_module("image_processor") is not None): self.add_stage( stage_name="image_encoding_stage", stage=ImageEncodingStage( image_encoder=self.get_module("image_encoder"), image_processor=self.get_module("image_processor"), )) self.add_stage(stage_name="image_vae_encoding_stage", stage=ImageVAEEncodingStage( vae=self.get_module("vae"), )) self.add_stage(stage_name="video_encoding_stage", stage=EncodingStage(vae=self.get_module("vae"), )) class PreprocessPipelineT2V(ComposedPipelineBase): _required_config_modules = ["text_encoder", "tokenizer", "vae"] def create_pipeline_stages(self, fastvideo_args: FastVideoArgs): _add_base_stages(self, fastvideo_args) self.add_stage(stage_name="video_encoding_stage", stage=EncodingStage(vae=self.get_module("vae"), ))

Edenzzzz · 2025-08-23T03:18:32Z

fastvideo/workflow/preprocess/components.py


    def _validate_data_type(self, batch: dict[str, Any]) -> bool:
        """Validate basic validity of data items"""
+        print("-------------------------------")


Eigensystem · 2025-08-23T03:50:13Z

please rebase

Eigensystem · 2025-08-23T03:56:21Z

fastvideo/workflow/preprocess/preprocess_workflow.py

        # training dataset
        training_dataset = load_dataset(preprocess_config.dataset_path,
-                                        split="train")
+                                        split="train",field="data")


Please change field to a member of PreprocessConfig to specify it in cli rather than hardcoding it

Edenzzzz

Looks like this isn't rebased correctly, with many fake changes to irrelevant files. Please rebase or merge main

fastvideo/models/vaes/hunyuanvae.py

fastvideo/workflow/preprocess/components.py

Eigensystem · 2025-09-10T09:47:19Z

scripts/dataset_preparation/prepare_json_file.py


    # Save to JSON file
    output_file = folder_path / output_name
+    print(folder_path,output_file,output_name)


remove this

Eigensystem · 2025-09-10T09:57:15Z

fastvideo/workflow/preprocess/components.py

 from datasets import Dataset, Video, load_dataset

 from fastvideo.configs.configs import DatasetType, PreprocessConfig
-from fastvideo.distributed.parallel_state import get_world_rank, get_world_size


There are some issues with your rebase. You should align your code with our implementation(accept both) instead of only use your code(accept current). Please revert this.

Eigensystem · 2025-09-10T09:57:26Z

fastvideo/workflow/preprocess/components.py

        return written_count


-def build_dataset(preprocess_config: PreprocessConfig, split: str,


revert this

Eigensystem · 2025-09-10T09:57:45Z

fastvideo/workflow/preprocess/preprocess_workflow.py

        self.add_component("raw_data_validator", raw_data_validator)

        # training dataset
-        training_dataset = build_dataset(preprocess_config,


revert this

gemini-code-assist bot reviewed Aug 22, 2025

View reviewed changes

SolitaryThinker changed the title ~~Hunyuan~~ [Preprocess] [feat] Support HunyuanVideo Model Aug 22, 2025

SolitaryThinker assigned Eigensystem and SolitaryThinker Aug 22, 2025

Edenzzzz reviewed Aug 23, 2025

View reviewed changes

Eigensystem reviewed Aug 23, 2025

View reviewed changes

nguyenhiep559505-glitch approved these changes Aug 23, 2025

View reviewed changes

Edenzzzz requested changes Aug 25, 2025

View reviewed changes

JiayiZhangA and others added 4 commits August 27, 2025 19:47

hunyuan video data preprocess

93e8e2b

clean up

766b931

[Feat][Preprocess] support merged dataset (hao-ai-lab#752)

d30879a

hunyuan video data preprocess

3b39366

JiayiZhangA force-pushed the hunyuan branch from 46159f0 to 2ed4d01 Compare August 27, 2025 20:02

SolitaryThinker added the go Trigger Buildkite CI label Aug 27, 2025

SolitaryThinker requested review from Edenzzzz, Eigensystem and SolitaryThinker August 27, 2025 23:13

Edenzzzz reviewed Aug 28, 2025

View reviewed changes

fastvideo/models/vaes/hunyuanvae.py Outdated Show resolved Hide resolved

Edenzzzz requested changes Aug 28, 2025

View reviewed changes

fastvideo/workflow/preprocess/components.py Outdated Show resolved Hide resolved

fastvideo/workflow/preprocess/components.py Outdated Show resolved Hide resolved

fastvideo/workflow/preprocess/components.py Outdated Show resolved Hide resolved

JiayiZhangA force-pushed the hunyuan branch from c8ec12f to f877e53 Compare August 31, 2025 00:03

Update hunyuan video t2v preprocess

94b8767

JiayiZhangA force-pushed the hunyuan branch from f877e53 to 94b8767 Compare August 31, 2025 00:07

Eigensystem requested changes Sep 10, 2025

View reviewed changes

		return written_count


		def build_dataset(preprocess_config: PreprocessConfig, split: str,

[Preprocess] [feat] Support HunyuanVideo Model #754

Are you sure you want to change the base?

[Preprocess] [feat] Support HunyuanVideo Model #754

Uh oh!

Conversation

JiayiZhangA commented Aug 22, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

Edenzzzz Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

Eigensystem commented Aug 23, 2025

Uh oh!

Eigensystem Aug 23, 2025

Choose a reason for hiding this comment

Uh oh!

Edenzzzz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Eigensystem Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Eigensystem Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Eigensystem Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Eigensystem Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants