Skip to content

A minimal PyTorch implementation of text-to-3D models, e.g., DreamFusion and Magic3D, based on NerfAcc

License

Notifications You must be signed in to change notification settings

chinhsuanwu/dreamfusionacc

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

DreamFusionAcc

Overview

This is a minimal PyTorch implementation of DreamFusion and its variant Magic3D, where we utilize Instant-NGP as the neural renderer and Stable Diffusion/DeepFloyd IF as the guidance.

It takes ~30min to train with Stable Diffusion and ~40min with DeepFloyd IF on a single 3090.

⚠️ This repo has not yet been well-optimized, e.g., memory usage and lack of refinement stage. However, it is still a good reference for its easy-to-follow and lightweight implementation. Please use stable-dreamfusion or threestudio for better quality 3D generation and mesh export.

Installation

git clone https://github.com/chinhsuanwu/dreamfusionacc.git
cd dreamfusionacc
pip install -r requirements.txt
Dependencies (click to expand)

Dependencies

  • torch
  • tinycudann
  • nerfacc
  • numpy
  • imageio
  • einops
  • diffusers
  • trainsformers

You may install the pre-built wheels for NerfAcc. Please check out here. This repo is built upon torch 1.13.0 + cu117.

Howtos

To train

python train.py --config config/peacock.yaml

You can find all controllable settings in the yaml file. After the training is done, run

python test.py --config config/peacock.yaml

to render 360˚ visualizations.

Citation

@article{poole2022dreamfusion,
  author = {Poole, Ben and Jain, Ajay and Barron, Jonathan T. and Mildenhall, Ben},
  title = {DreamFusion: Text-to-3D using 2D Diffusion},
  journal = {arXiv},
  year = {2022},
}

@inproceedings{lin2023magic3d,
  title={Magic3D: High-Resolution Text-to-3D Content Creation},
  author={Lin, Chen-Hsuan and Gao, Jun and Tang, Luming and Takikawa, Towaki and Zeng, Xiaohui and Huang, Xun and Kreis, Karsten and Fidler, Sanja and Liu, Ming-Yu and Lin, Tsung-Yi},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition ({CVPR})},
  year={2023}
}

Credits

This implementation is heavily based on NerfAcc and stable-dreamfusion. Kudos to the authors for their amazing work!

tyro logo

About

A minimal PyTorch implementation of text-to-3D models, e.g., DreamFusion and Magic3D, based on NerfAcc

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages