This fork of Relightable3DGaussian adds the abitlity to rotate the environment light to the realtime GUI, as seen on their project page:
-
Install the repository as described in the original readme.
-
Download the trained models from here. Copy the folders
datasets
,output
andpoint
into the root of the repository.- If this link isn't valid anymore, my university account likely expired. You can also train the models yourself, as described in the original readme.
-
Activate the conda environment with
conda activate r3dg
. -
Run gui.py with
-t neilf_composite
. For example:python gui.py -m output/NeRF_Syn/lego/neilf -t neilf_composite --sample_num=24
- change the
lego
part in the command toficus
,ship
or any other folder name inoutput/NeRF_Syn
to load a different model. - reduce
sample_num
to reduce the raytracing samples and speed up rendering. For reference: The original paper states 33 FPS with 24 samples on a NVIDIA GeForce RTX 3090 GPU for the composite scene seen above (but this animation was actually rendered offline with 384 samples in 0,018 FPS). My GTX 1080 Ti achieves 25-28 FPS with mere 6 samples on the single ficus model, while my GTX 1050 mobile achieves 3-8 FPS with the same model and samplesποΈ.
- change the
-
Use the left and right arrow keys to change the rotation of the environment light.
Jian Gao1*, Chun Gu2*, Youtian Lin1, Hao Zhu1, Xun Cao1, Li Zhang2 , Yao Yao1
1Nanjing University 2Fudan University
*denotes Equally contributed.
This is official implement of Relightable 3D Gaussian for the paper Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing.
git clone https://github.com/NJU-3DV/Relightable3DGaussian.git
# install environment
conda env create --file environment.yml
conda activate r3dg
# install pytorch=1.12.1
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
# install torch_scatter==2.1.1
pip install torch_scatter==2.1.1
# install kornia==0.6.12
pip install kornia==0.6.12
# install nvdiffrast=0.3.1
git clone https://github.com/NVlabs/nvdiffrast
pip install ./nvdiffrast
We recommend that users compile the extension with CUDA 11.8 to avoid the potential problems mentioned in 3D Guassian Splatting.
# install knn-cuda
pip install ./submodules/simple-knn
# install bvh
pip install ./bvh
# install relightable 3D Gaussian
pip install ./r3dg-rasterization
Download the NeRF synthetic dataset from LINK provided by NeRF.
For real-world DTU data, we follow the Vis-MVSNet to get the depth maps and then filter the depth map through photometric and geometric check. We then convert the depth map to normal through kornia. And we get perfect masks from IDR. The pre-processed DTU data can be downloaded here.
For real-world Tanks and Temples data, we also use Vis-MVSNet and kornia to get the filtered MVS depth maps and normal maps. The masks are from NSVF. The pre-processed Tanks and Temples data can be downloaded here.
We organize the datasets like this:
Relightable3DGaussian
βββ datasets
βββ nerf_synthetic
| βββ chair
| βββ ...
βββ neilfpp
| βββ data_dtu
β β βββ DTU_scan24
| | | |ββ inputs
| | | | |ββ depths
| | | | |ββ images
| | | | |ββ model
| | | | |ββ normals
| | | | |ββ pmasks
| | | | |ββ sfm_scene.json
β β βββ ...
| βββ data_tnt
β β βββ Barn
| | | |ββ inputs
| | | | |ββ depths
| | | | |ββ images
| | | | |ββ model
| | | | |ββ normals
| | | | |ββ pmasks
| | | | |ββ sfm_scene.json
β β βββ ...
For multi-object composition, we manually generate a ground plane with relightable 3D Gaussian representation, which can be downloaded here. We put the ground.ply in the folder ./point.
We run the code in a single NVIDIA GeForce RTX 3090 GPU (24G). To reproduce the results in the paper, please run the following code. NeRF Synthetic dataset:
sh script/run_nerf.sh
DTU data:
sh script/run_dtu.sh
Tanks and Temples data:
sh script/run_tnt.sh
Explicit point cloud representation facilitates composition. We recommend that users explore cloud compare to implement point cloud transformations such as scaling, translation, and rotation to composite a new scene. For multi-object composition and relighting result demonstrated in the paper, you could run:
python relighting.py \
-co configs/teaser \
-e "env_map/teaser.hdr" \
--output "output/relighting/teaser_trace" \
--sample 384
For multi-object composition and relighting video illustrated in the project page, you could run:
python relighting.py \
-e env_map/composition.hdr \
-co configs/nerf_syn \
--output "output/relighting/nerf_syn" \
--sample_num 384 \
--video
python relighting.py \
-co configs/nerf_syn_light \
-e "env_map/composition.hdr" \
--output "output/relighting/nerf_syn_light" \
--sample_num 384 \
--video
For multi-scene composition video, you could run:
python relighting.py \
-co configs/tnt \
-e "env_map/ocean_from_horn.jpg" \
--output "output/relighting/tnt" \
--sample_num 384 \
--video
We also provide a GUI for visualization. You can utilize this GUI to supervise the training process by just adding the --gui option to the command for training. You can also visualize the model after training by running:
# for 3D Gaussian
python gui.py -m output/NeRF_Syn/lego/3dgs -t render
# for relightable 3D Gaussian
python gui.py -m output/NeRF_Syn/lego/neilf -t neilf
We provide here a modified version of Vis-MVSNet to get the MVS cues for geometry enhancement.
cd vismvsnet
(1) Run using shell You could change the <Path_to_data>, <img_width>, <img_height> and <src_num> according to your own data in the run_pre.sh. Note that the images and masks should be stored in <Path_to_data>/input and <Path_to_data>/masks. (Mask == 0: background, Mask==255: Object) And get the filtered MVS depth by:
sh run_pre.sh
(2) Or you colud run step by step Step1. Use COLMAP to convert a collection of un-posed images to posed ones.
python convert.py -s <Path_to_data>
Step2. Convert COLMAP data to MVSNet data.
python colmap2mvsnet.py --dense_folder <Path_to_data> --max_d 256
Step3. Run VisMVSNet. The <img_width>, <img_height> and <src_num> should be determined based on the dataset.
python test.py --data_root <Path_to_data> --resize "<img_width>,<img_height>" --crop "<img_width>,<img_heigh>" --num_src <num_src>
Step4. Run depth map filtering.
python filter.py --data <Path_to_data>/vis_mvsnet --pair <Path_to_data>/pair.txt --view 5 --vthresh 2 --pthresh '.6,.6,.6' --out_dir <Path_to_data>/filtered
Then, run training:
# 3DGS
python train.py \
-s <Path_to_data> \
-m <Path_to_output>/3dgs \
--lambda_mask_entropy 0.1 \
--lambda_normal_render_depth 0.01 \
--lambda_normal_mvs_depth 0.01 \
--lambda_depth 1 \
--densification_interval 500 \
--save_training_vis
# R3DG
python train.py \
-s <Path_to_data> \
-m <Path_to_output>/neilf \
-c <Path_to_output>/3dgs/chkpnt30000.pth \
-t neilf \
--lambda_mask_entropy 0.1 \
--lambda_normal_render_depth 0.01 \
--use_global_shs \
--finetune_visibility \
--iterations 40000 \
--test_interval 1000 \
--checkpoint_interval 2500 \
--lambda_light 0.01 \
--lambda_base_color 0.005 \
--lambda_base_color_smooth 0.006 \
--lambda_metallic_smooth 0.002 \
--lambda_roughness_smooth 0.002 \
--lambda_visibility 0.1 \
--save_training_vis
If you find our work useful in your research, please be so kind to cite:
@article{R3DG2023,
author = {Gao, Jian and Gu, Chun and Lin, Youtian and Zhu, Hao and Cao, Xun and Zhang, Li and Yao, Yao},
title = {Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing},
journal = {arXiv:2311.16043},
year = {2023},
}