Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing (ECCV2024)
Jian Gao1*, Chun Gu2*, Youtian Lin1, Hao Zhu1, Xun Cao1, Li Zhang2 , Yao Yao1
1Nanjing University 2Fudan University
*denotes Equally contributed.
This is official implement of Relightable 3D Gaussian for the paper Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing.
git clone https://github.com/NJU-3DV/Relightable3DGaussian.git
# install environment
conda env create --file environment.yml
conda activate r3dg
# install pytorch=1.12.1
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge
# install torch_scatter==2.1.1
pip install torch_scatter==2.1.1
# install kornia==0.6.12
pip install kornia==0.6.12
# install nvdiffrast=0.3.1
git clone https://github.com/NVlabs/nvdiffrast
pip install ./nvdiffrast
We recommend that users compile the extension with CUDA 11.8 to avoid the potential problems mentioned in 3D Guassian Splatting.
# install knn-cuda
pip install ./submodules/simple-knn
# install bvh
pip install ./bvh
# install relightable 3D Gaussian
pip install ./r3dg-rasterization
Download the NeRF synthetic dataset from LINK provided by NeRF.
For real-world DTU data, we follow the Vis-MVSNet to get the depth maps and then filter the depth map through photometric and geometric check. We then convert the depth map to normal through kornia. And we get perfect masks from IDR. The pre-processed DTU data can be downloaded here.
For real-world Tanks and Temples data, we also use Vis-MVSNet and kornia to get the filtered MVS depth maps and normal maps. The masks are from NSVF. The pre-processed Tanks and Temples data can be downloaded here.
Download the NeRF synthetic dataset from LINK provided by InvRender.
We organize the datasets like this:
Relightable3DGaussian
βββ datasets
βββ nerf_synthetic
| βββ chair
| βββ ...
βββ neilfpp
| βββ data_dtu
β β βββ DTU_scan24
| | | |ββ inputs
| | | | |ββ depths
| | | | |ββ images
| | | | |ββ model
| | | | |ββ normals
| | | | |ββ pmasks
| | | | |ββ sfm_scene.json
β β βββ ...
| βββ data_tnt
β β βββ Barn
| | | |ββ inputs
| | | | |ββ depths
| | | | |ββ images
| | | | |ββ model
| | | | |ββ normals
| | | | |ββ pmasks
| | | | |ββ sfm_scene.json
β β βββ ...
βββ Synthetic4Relight
| βββ air_baloons
| βββ ...
For multi-object composition, we manually generate a ground plane with relightable 3D Gaussian representation, which can be downloaded here. We put the ground.ply in the folder ./point.
We run the code in a single NVIDIA GeForce RTX 3090 GPU (24G). To reproduce the results in the paper, please run the following code. NeRF Synthetic dataset:
sh script/run_nerf.sh
DTU data:
sh script/run_dtu.sh
Tanks and Temples data:
sh script/run_tnt.sh
Synthetic4Relight data:
sh script/run_syn.sh
Run the following command to evaluate Novel View Synthesis:
# e.g. DTU dataset
# stage 1
python eval_nvs.py --eval \
-m output/dtu/${i}/3dgs \
-c output/dtu/${i}/3dgs/chkpnt30000.pth
# stage 2
python eval_nvs.py --eval \
-m output/dtu/${i}/neilf \
-c output/dtu/${i}/neilf/chkpnt50000.pth \
-t neilf
Run the following command to evaluate Relighting (for Synthetic4Relight only):
# e.g.
python eval_relighting_syn4.py \
-m output/Syn4Relight/hotdog/neilf \
-c output/Syn4Relight/hotdog/neilf/chkpnt50000.pth \
--sample_num 384
Explicit point cloud representation facilitates composition. We recommend that users explore cloud compare to implement point cloud transformations such as scaling, translation, and rotation to composite a new scene. For multi-object composition and relighting result demonstrated in the paper, you could run:
python relighting.py \
-co configs/teaser \
-e "env_map/teaser.hdr" \
--output "output/relighting/teaser_trace" \
--sample 384
For multi-object composition and relighting video illustrated in the project page, you could run:
python relighting.py \
-e env_map/composition.hdr \
-co configs/nerf_syn \
--output "output/relighting/nerf_syn" \
--sample_num 384 \
--video
python relighting.py \
-co configs/nerf_syn_light \
-e "env_map/composition.hdr" \
--output "output/relighting/nerf_syn_light" \
--sample_num 384 \
--video
For multi-scene composition video, you could run:
python relighting.py \
-co configs/tnt \
-e "env_map/ocean_from_horn.jpg" \
--output "output/relighting/tnt" \
--sample_num 384 \
--video
We also provide a GUI for visualization. You can utilize this GUI to supervise the training process by just adding the --gui option to the command for training. You can also visualize the model after training by running:
# for 3D Gaussian
python gui.py -m output/NeRF_Syn/lego/3dgs -t render
# for relightable 3D Gaussian
python gui.py -m output/NeRF_Syn/lego/neilf -t neilf
We recommend that users reorganize their own data as neilfpp-like dataset and then optimize. Modified VisMVSNet and auxiliary scripts to prepare your own data will come soon.
If you find our work useful in your research, please be so kind to cite:
@article{R3DG2023,
author = {Gao, Jian and Gu, Chun and Lin, Youtian and Zhu, Hao and Cao, Xun and Zhang, Li and Yao, Yao},
title = {Relightable 3D Gaussian: Real-time Point Cloud Relighting with BRDF Decomposition and Ray Tracing},
journal = {arXiv:2311.16043},
year = {2023},
}