Dependencies not specified

[rubenvillarreal]

Your environment yaml is not complete. You dont specify accelerate version which causes dependency issues and there is no installation for huggingface.

Is this project still ongoing? (also, the sd15 link from runwayml is no longer in huggingface)

[rubenvillarreal]

Here is an environment that actually works.

name: motionclone
channels:

• pytorch
• nvidia
  dependencies:
• python=3.11.3
• pytorch=2.0.1
• torchvision=0.15.2
• pytorch-cuda=11.8 # Matches PyTorch 2.0.1 CUDA version
• pip
• pip:
  • diffusers==0.16.0 # updated diffusers version (supports PyTorch 2.0)
  • huggingface_hub==0.14.1 # include Hub with bugfix
  • accelerate==0.18.0 # pin Accelerate to satisfy diffusers offload
  • transformers==4.28.1 # same Transformers version (for Stable Diffusion)
  • xformers==0.0.20 # xFormers build compatible with torch 2.0.1
  • imageio[ffmpeg]
  • decord==0.6.0
  • gdown
  • einops
  • omegaconf
  • safetensors
  • gradio
  • wandb
  • triton
  • opencv-python
  • fsspec # required by newer huggingface_hub
  • pyyaml # required by newer huggingface_hub

[archernsy]

Hello! I followed the environment YAML configuration you provided, but I still encounter errors when running the inference code. Have you encountered this issue?

python i2v_video_sample.py --inference_config "configs/i2v_rgb.yaml" --examples "configs/i2v_rgb.jsonl"
/usr/bin/ld: skipping incompatible /usr/lib/i386-linux-gnu/libcuda.so when searching for -lcuda
/usr/bin/ld: cannot find -lcuda
collect2: error: ld returned 1 exit status
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /media/data4/liuyj/xjy/video_diffusion_rept/MotionClone/i2v_video_sample.py:4 in │
│ │
│ 1 import argparse │
│ 2 from omegaconf import OmegaConf │
│ 3 import torch │
│ ❱ 4 from diffusers import AutoencoderKL, DDIMScheduler │
│ 5 from transformers import CLIPTextModel, CLIPTokenizer │
│ 6 from motionclone.models.unet import UNet3DConditionModel │
│ 7 from motionclone.models.sparse_controlnet import SparseControlNetModel │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/diffusers/init. │
│ py:36 in │
│ │
│ 33 except OptionalDependencyNotAvailable: │
│ 34 │ from .utils.dummy_pt_objects import * # noqa F403 │
│ 35 else: │
│ ❱ 36 │ from .models import ( │
│ 37 │ │ AutoencoderKL, │
│ 38 │ │ ControlNetModel, │
│ 39 │ │ ModelMixin, │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/diffusers/models/__ │
│ init__.py:19 in │
│ │
│ 16 │
│ 17 │
│ 18 if is_torch_available(): │
│ ❱ 19 │ from .autoencoder_kl import AutoencoderKL │
│ 20 │ from .controlnet import ControlNetModel │
│ 21 │ from .dual_transformer_2d import DualTransformer2DModel │
│ 22 │ from .modeling_utils import ModelMixin │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/diffusers/models/au │
│ toencoder_kl.py:23 in │
│ │
│ 20 from ..configuration_utils import ConfigMixin, register_to_config │
│ 21 from ..utils import BaseOutput, apply_forward_hook │
│ 22 from .modeling_utils import ModelMixin │
│ ❱ 23 from .vae import Decoder, DecoderOutput, DiagonalGaussianDistribution, Encoder │
│ 24 │
│ 25 │
│ 26 @DataClass │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/diffusers/models/va │
│ e.py:22 in │
│ │
│ 19 import torch.nn as nn │
│ 20 │
│ 21 from ..utils import BaseOutput, randn_tensor │
│ ❱ 22 from .unet_2d_blocks import UNetMidBlock2D, get_down_block, get_up_block │
│ 23 │
│ 24 │
│ 25 @DataClass │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/diffusers/models/un │
│ et_2d_blocks.py:21 in │
│ │
│ 18 import torch.nn.functional as F │
│ 19 from torch import nn │
│ 20 │
│ ❱ 21 from .attention import AdaGroupNorm, AttentionBlock │
│ 22 from .attention_processor import Attention, AttnAddedKVProcessor, AttnAddedKVProcessor2_ │
│ 23 from .dual_transformer_2d import DualTransformer2DModel │
│ 24 from .resnet import Downsample2D, FirDownsample2D, FirUpsample2D, KDownsample2D, KUpsamp │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/diffusers/models/at │
│ tention.py:22 in │
│ │
│ 19 from torch import nn │
│ 20 │
│ 21 from ..utils.import_utils import is_xformers_available │
│ ❱ 22 from .attention_processor import Attention │
│ 23 from .embeddings import CombinedTimestepLabelEmbeddings │
│ 24 │
│ 25 │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/diffusers/models/at │
│ tention_processor.py:29 in │
│ │
│ 26 │
│ 27 if is_xformers_available(): │
│ 28 │ import xformers │
│ ❱ 29 │ import xformers.ops │
│ 30 else: │
│ 31 │ xformers = None │
│ 32 │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/xformers/ops/__init │
│ .py:8 in │
│ │
│ 5 │
│ 6 import torch │
│ 7 │
│ ❱ 8 from .fmha import ( │
│ 9 │ AttentionBias, │
│ 10 │ AttentionOp, │
│ 11 │ AttentionOpBase, │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/xformers/ops/fmha/ │
│ init.py:10 in │
│ │
│ 7 │
│ 8 import torch │
│ 9 │
│ ❱ 10 from . import cutlass, flash, small_k, triton │
│ 11 from .attn_bias import AttentionBias, BlockDiagonalMask, LowerTriangularMask │
│ 12 from .common import ( │
│ 13 │ AttentionBwOpBase, │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/xformers/ops/fmha/t │
│ riton.py:15 in │
│ │
│ 12 from ... import _is_triton_available │
│ 13 from ..common import register_operator │
│ 14 │
│ ❱ 15 if TYPE_CHECKING or _is_triton_available(): │
│ 16 │ from ..._flash_attn.flash_attn_triton import ( │
│ 17 │ │ _flash_attn_backward, │
│ 18 │ │ _flash_attn_forward, │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/xformers/init.p │
│ y:34 in func_wrapper │
│ │
│ 31 │ def func_wrapper(): │
│ 32 │ │ nonlocal value │
│ 33 │ │ if value is None: │
│ ❱ 34 │ │ │ value = func() │
│ 35 │ │ return value │
│ 36 │ │
│ 37 │ return func_wrapper │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/xformers/init.p │
│ y:47 in _is_triton_available │
│ │
│ 44 │ if os.environ.get("XFORMERS_FORCE_DISABLE_TRITON", "0") == "1": │
│ 45 │ │ return False │
│ 46 │ try: │
│ ❱ 47 │ │ from xformers.triton.softmax import softmax as triton_softmax # noqa │
│ 48 │ │ │
│ 49 │ │ return True │
│ 50 │ except (ImportError, AttributeError) as e: │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/xformers/triton/sof │
│ tmax.py:14 in │
│ │
│ 11 import triton │
│ 12 from torch.cuda.amp import custom_bwd, custom_fwd │
│ 13 │
│ ❱ 14 from xformers.triton.k_softmax import _softmax, _softmax_backward │
│ 15 │
│ 16 # CREDITS: This is adapted from the vanilla Triton example. See https://openai.com/blog/ │
│ 17 # and https://triton-lang.org/getting-started/tutorials/02-fused-softmax.html │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/xformers/triton/k_s │
│ oftmax.py:21 in │
│ │
│ 18 # autotune: Triton will test out these configurations, and automatically pick the fastes │
│ 19 # heuristic: add arguments to the kernel call automatically given some heuristics. These │
│ 20 # fmt: off │
│ ❱ 21 @triton.autotune( │
│ 22 │ configs=[ │
│ 23 │ │ triton.Config({}, num_warps=1), │
│ 24 │ │ triton.Config({}, num_warps=2), │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/runtime/auto │
│ tuner.py:368 in decorator │
│ │
│ 365 │ """ │
│ 366 │ │
│ 367 │ def decorator(fn): │
│ ❱ 368 │ │ return Autotuner(fn, fn.arg_names, configs, key, reset_to_zero, restore_value, p │
│ 369 │ │ │ │ │ │ post_hook=post_hook, prune_configs_by=prune_configs_by, warmup= │
│ 370 │ │ │ │ │ │ use_cuda_graph=use_cuda_graph) │
│ 371 │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/runtime/auto │
│ tuner.py:130 in init │
│ │
│ 127 │ │ │ return │
│ 128 │ │ │
│ 129 │ │ if do_bench is None: │
│ ❱ 130 │ │ │ self.do_bench = driver.active.get_benchmarker() │
│ 131 │ │ else: │
│ 132 │ │ │ self.do_bench = do_bench │
│ 133 │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/runtime/driv │
│ er.py:23 in getattr │
│ │
│ 20 │ │ │ self._obj = self._init_fn() │
│ 21 │ │
│ 22 │ def getattr(self, name): │
│ ❱ 23 │ │ self._initialize_obj() │
│ 24 │ │ return getattr(self._obj, name) │
│ 25 │ │
│ 26 │ def setattr(self, name, value): │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/runtime/driv │
│ er.py:20 in _initialize_obj │
│ │
│ 17 │ │
│ 18 │ def _initialize_obj(self): │
│ 19 │ │ if self._obj is None: │
│ ❱ 20 │ │ │ self._obj = self._init_fn() │
│ 21 │ │
│ 22 │ def getattr(self, name): │
│ 23 │ │ self._initialize_obj() │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/runtime/driv │
│ er.py:9 in _create_driver │
│ │
│ 6 │ actives = [x.driver for x in backends.values() if x.driver.is_active()] │
│ 7 │ if len(actives) != 1: │
│ 8 │ │ raise RuntimeError(f"{len(actives)} active drivers ({actives}). There should onl │
│ ❱ 9 │ return actives0 │
│ 10 │
│ 11 │
│ 12 class LazyProxy: │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/backends/nvi │
│ dia/driver.py:450 in init │
│ │
│ 447 class CudaDriver(GPUDriver): │
│ 448 │ │
│ 449 │ def init(self): │
│ ❱ 450 │ │ self.utils = CudaUtils() # TODO: make static │
│ 451 │ │ self.launcher_cls = CudaLauncher │
│ 452 │ │ super().init() │
│ 453 │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/backends/nvi │
│ dia/driver.py:80 in init │
│ │
│ 77 │ │ return cls.instance │
│ 78 │ │
│ 79 │ def init(self): │
│ ❱ 80 │ │ mod = compile_module_from_src(Path(os.path.join(dirname, "driver.c")).read_text( │
│ 81 │ │ self.load_binary = mod.load_binary │
│ 82 │ │ self.get_device_properties = mod.get_device_properties │
│ 83 │ │ self.cuOccupancyMaxActiveClusters = mod.cuOccupancyMaxActiveClusters │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/backends/nvi │
│ dia/driver.py:57 in compile_module_from_src │
│ │
│ 54 │ │ │ src_path = os.path.join(tmpdir, "main.c") │
│ 55 │ │ │ with open(src_path, "w") as f: │
│ 56 │ │ │ │ f.write(src) │
│ ❱ 57 │ │ │ so = _build(name, src_path, tmpdir, library_dirs(), include_dir, libraries) │
│ 58 │ │ │ with open(so, "rb") as f: │
│ 59 │ │ │ │ cache_path = cache.put(f.read(), f"{name}.so", binary=True) │
│ 60 │ import importlib.util │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/runtime/buil │
│ d.py:50 in _build │
│ │
│ 47 │ cc_cmd += [f'-l{lib}' for lib in libraries] │
│ 48 │ cc_cmd += [f"-L{dir}" for dir in library_dirs] │
│ 49 │ cc_cmd += [f"-I{dir}" for dir in include_dirs if dir is not None] │
│ ❱ 50 │ ret = subprocess.check_call(cc_cmd) │
│ 51 │ if ret == 0: │
│ 52 │ │ return so │
│ 53 │ # fallback on setuptools │
│ │
│ /media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/subprocess.py:413 in check_call │
│ │
│ 410 │ │ cmd = kwargs.get("args") │
│ 411 │ │ if cmd is None: │
│ 412 │ │ │ cmd = popenargs[0] │
│ ❱ 413 │ │ raise CalledProcessError(retcode, cmd) │
│ 414 │ return 0 │
│ 415 │
│ 416 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['/usr/bin/gcc-8', '/tmp/tmpa734pck8/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmpa734pck8/cuda_utils.cpython-311-x86_64-linux-gnu.so', '-lcuda',
'-L/media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/backends/nvidia/lib', '-L/usr/lib/x86_64-linux-gnu', '-L/usr/lib/i386-linux-gnu',
'-I/media/data4/liuyj/anaconda3/envs/motionclone03/lib/python3.11/site-packages/triton/backends/nvidia/include', '-I/tmp/tmpa734pck8', '-I/media/data4/liuyj/anaconda3/envs/motionclone03/include/python3.11']'
returned non-zero exit status 1.

[rubenvillarreal]

are you on windows? This line at the start means tells me there might be issues with cuda since .so files are for linux and not windows. i have not tried getting this to run on windows sorry

/usr/bin/ld: skipping incompatible /usr/lib/i386-linux-gnu/libcuda.so when searching for -lcuda

[archernsy]

Hello! Thank you very much for your reply! I found the problem and it was caused by the overversion of triton. It is currently running successfully.

[ljaiverson]

Hi @rubenvillarreal
Thanks for the environment that works. Did you figure out the SD1.5 link issue? I am currently using a SD1.5 from sd-legacy on hugging face. But it always gives me a SafetensorError: Error while deserializing header: HeaderTooLarge. Not sure whether the issue is the source of the SD1.5.

[ljaiverson]

NVM, my git-lfs was not installed successfully...