components diffusers_text_to_image_finetune

Text to Image Diffusers Model Finetune

diffusers_text_to_image_finetune

Overview

Component to finetune stable diffusion models using diffusers for text to image.

Version: 0.0.7

View in Studio: https://ml.azure.com/registries/azureml/components/diffusers_text_to_image_finetune/version/0.0.7

Inputs

component input: model path

Name	Description	Type	Default	Optional	Enum
model_path	Output folder of model selector containing model metadata like config, checkpoints, tokenizer config.	uri_folder		False

component input: Instance data dir

Name	Description	Type	Default	Optional	Enum
instance_data_dir	A folder containing the training data of instance images.	uri_folder		False

optional component input: Class data dir

Name	Description	Type	Default	Optional	Enum
class_data_dir	A folder containing the training data of class images.	uri_folder		True
task_name	Which task the model is solving.	string			['stable-diffusion-text-to-image']

Instance prompt

Name	Description	Type	Default	Optional	Enum
instance_prompt	The prompt with identifier specifying the instance.	string		True
resolution	The image resolution for training.	integer	512	True

Lora parameters LoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights. This vastly reduces the storage requirement for large language models adapted to specific tasks and enables efficient task-switching during deployment all without introducing inference latency. LoRA also outperforms several other adaptation methods including adapter, prefix-tuning, and fine-tuning. Currently, LoRA is supported for gpt2, bert, roberta, deberta, distilbert, t5, bart, mbart and camembert model families

Name	Description	Type	Default	Optional	Enum
apply_lora	If "true" enables lora.	boolean	True	False
lora_alpha	alpha attention parameter for lora.	integer	128	True
lora_r	lora dimension	integer	8	True
lora_dropout	lora dropout value	number	0.0	True
tokenizer_max_length	The maximum length of the tokenizer. If not set, will default to the tokenizer's max length.	integer		True

Text Encoder

Name	Description	Type	Default	Optional	Enum
text_encoder_type	Text encoder to be used.	string		True	['CLIPTextModel', 'T5EncoderModel']
text_encoder_name	Huggingface id of text encoder. This model should of type specified in text_encoder_type. If not specified the default from the model will be used.	string		True
train_text_encoder	Whether to train the text encoder. If set, the text encoder should be float32 precision.	boolean	False	True
pre_compute_text_embeddings	Whether or not to pre-compute text embeddings. If text embeddings are pre-computed, the text encoder will not be kept in memory during training and will leave more GPU memory available for training the rest of the model. This is not compatible with --train_text_encoder.	boolean	True	True
text_encoder_use_attention_mask	Whether to use attention mask for the text encoder	boolean	False	True

UNET related

Name	Description	Type	Default	Optional	Enum
class_labels_conditioning	The optional class_label conditioning to pass to the unet, available values are timesteps.	string		True

Noise Scheduler

Name	Description	Type	Default	Optional	Enum
noise_scheduler_name	Noise scheduler to be used.	string		True	['DPMSolverMultistepScheduler', 'DDPMScheduler', 'PNDMScheduler']
noise_scheduler_num_train_timesteps	The number of diffusion steps to train the model.	integer		True
noise_scheduler_variance_type	Clip the variance when adding noise to the denoised sample.	string		True	['fixed_small', 'fixed_small_log', 'fixed_large', 'fixed_large_log', 'learned', 'learned_range']
noise_scheduler_prediction_type	Prediction type of the scheduler function; can be epsilon (predicts the noise of the diffusion process), sample (directly predicts the noisy sample) or v_prediction` (see section 2.4 of Imagen Video paper).	string		True	['epsilon', 'sample', 'v_prediction']
noise_scheduler_timestep_spacing	The way the timesteps should be scaled. Refer to Table 2 of the Common Diffusion Noise Schedules and Sample Steps are Flawed for more information.	string		True
noise_scheduler_steps_offset	An offset added to the inference steps. You can use a combination of offset=1 and set_alpha_to_one=False to make the last step use step 0 for the previous alpha product like in Stable Diffusion.	integer		True
extra_noise_scheduler_args	Optional additional arguments that are supplied to noise scheduler. The arguments should be semi-colon separated key value pairs and should be enclosed in double quotes. For example, "clip_sample_range=1.0; clip_sample=True" for DDPMScheduler.	string		True

Offset Noise

Name	Description	Type	Default	Optional	Enum
offset_noise	Fine-tuning against a modified noise. See https://www.crosslabs.org//blog/diffusion-with-offset-noise for more information.	boolean		True

Prior preservation loss

Name	Description	Type	Default	Optional	Enum
with_prior_preservation	Flag to add prior preservation loss.	boolean	True
class_prompt	The prompt to specify images in the same class as provided instance images.	string		True
num_class_images	Minimal class images for prior preservation loss. If there are not enough images already present in class_data_dir, additional images will be sampled with class_prompt.	integer	100	True
prior_generation_precision	Choose prior generation precision between fp32, fp16 and bf16 (bfloat16). Bf16 requires PyTorch >= 1.10.and an Nvidia Ampere GPU. Default to fp16 if a GPU is available else fp32.	string	fp32	True	['fp32', 'fp16', 'bf16']
prior_loss_weight	The weight of prior preservation loss.	number	1.0	True
sample_batch_size	Batch size (per device) for sampling class images when training with_prior_preservation set to True.	integer	4	True
num_validation_images	Specify number of images to generate using instance_prompt. Images are stored in the output/checkpoint-* directories. Please note that this will increase the training time. If you select num_validation_images = 0, then run will generate 5 images in last checkpoint.	integer	0
number_of_workers	Number of subprocesses to use for data loading (PyTorch only). 0 means that the data will be loaded in the main process.	integer	6	True

Training parameters

Name	Description	Type	Default	Optional	Enum
number_of_epochs	Number of training epochs. If left empty, will be chosen automatically based on the task type and model selected.	integer		True
max_steps	If set to a positive number, the total number of training steps to perform. Overrides 'number_of_epochs'. In case of using a finite iterable dataset the training may stop before reaching the set number of steps when all data is exhausted. If left empty, will be chosen automatically based on the task type and model selected.	integer		True
training_batch_size	Train batch size. If left empty, will be chosen automatically based on the task type and model selected.	integer	1	True
auto_find_batch_size	Flag to enable auto finding of batch size. If the provided 'per_device_train_batch_size' goes into Out Of Memory (OOM) enabling auto_find_batch_size will find the correct batch size by iteratively reducing 'per_device_train_batch_size' by a factor of 2 till the OOM is fixed.	boolean	False	True

learning rate and learning rate scheduler

Name	Description	Type	Default	Optional	Enum
learning_rate	Start learning rate. Defaults to linear scheduler. If left empty, will be chosen automatically based on the task type and model selected.	number		True
learning_rate_scheduler	The scheduler type to use. If left empty, will be chosen automatically based on the task type and model selected.	string		True	['warmup_linear', 'warmup_cosine', 'warmup_cosine_with_restarts', 'warmup_polynomial', 'constant', 'warmup_constant']
warmup_steps	Number of steps used for a linear warmup from 0 to learning_rate. If left empty, will be chosen automatically based on the task type and model selected.	integer	0	True

optimizer

Name	Description	Type	Default	Optional	Enum
optimizer	optimizer to be used while training. 'adamw_ort_fused' optimizer is only supported for ORT training. If left empty, will be chosen automatically based on the task type and model selected.	string		True	['adamw_hf', 'adamw', 'sgd', 'adafactor', 'adagrad', 'adamw_ort_fused']
weight_decay	The weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in AdamW and sgd optimizer. If left empty, will be chosen automatically based on the task type and model selected.	number	0	True
extra_optim_args	Optional additional arguments that are supplied to SGD Optimizer. The arguments should be semi-colon separated key value pairs and should be enclosed in double quotes. For example, "momentum=0.5; nesterov=True" for sgd. Please make sure to use a valid parameter names for the chosen optimizer. For exact parameter names, please refer https://pytorch.org/docs/1.13/generated/torch.optim.SGD.html#torch.optim.SGD for SGD. Parameters supplied in extra_optim_args will take precedence over the parameter supplied via other arguments such as weight_decay. If weight_decay is provided via "weight_decay" parameter and via extra_optim_args both, values specified in extra_optim_args will be used.	string		True

gradient accumulation

Name	Description	Type	Default	Optional	Enum
gradient_accumulation_step	Number of update steps to accumulate the gradients for, before performing a backward/update pass. If left empty, will be chosen automatically based on the task type and model selected.	integer		True

mixed precision training

Name	Description	Type	Default	Optional	Enum
precision	Apply mixed precision training. This can reduce memory footprint by performing operations in half-precision.	string	32	True	['32', '16']

random seed

Name	Description	Type	Default	Optional	Enum
random_seed	Random seed that will be set at the beginning of training.	integer	42	True

logging strategy parameters

Name	Description	Type	Default	Optional	Enum
logging_strategy	The logging strategy to adopt during training.	string	epoch	True	['epoch', 'steps']
logging_steps	Number of update steps between two logs if logging_strategy='steps'.	integer	500	True

model checkpointing limit

Name	Description	Type	Default	Optional	Enum
save_total_limit	If a value is passed, will limit the total amount of checkpoints. Deletes the older checkpoints in output_dir. If the value is -1 saves all checkpoints".	integer	5	True

Grad Norm

Name	Description	Type	Default	Optional	Enum
max_grad_norm	Maximum gradient norm (for gradient clipping). If left empty, will be chosen automatically based on the task type and model selected.	number		True

save mlflow model

Name	Description	Type	Default	Optional	Enum
save_as_mlflow_model	Save as mlflow model with pyfunc as flavour.	boolean	True	True

Outputs

Name	Description	Type
mlflow_model_folder	Output dir to save the finetune model as mlflow model.	mlflow_model
pytorch_model_folder	Output dir to save the finetune model as torch model.	custom_model

Environment

azureml://registries/azureml/environments/acft-transformers-image-gpu/versions/37