Multi-view PIFuHD Using the Single-view Pretrained Model

PIFuHD currently offers a pretrained model designed for single-view (frontal) image 3D reconstruction. This repository extends PIFuHD's capabilities, enabling 3D reconstruction from multi-view (frontal and backside) images using the existing single-view pretrained model. Achieving this 'zeroshot' multi-view 3D reconstruction is made possible by altering the image processing pipelines and implementing channel-wise Adaptive Instance Normalization (AdaIN). This modification notably improves the detailing on the backside of the 3D mesh."

The sample images are in 'sample_images' folder. The rendering and mesh source.

Setup

Please refer to the original PIFuHD repository for setup instructions, which include package installation and downloading the pretrained model.

Preprocessing

For 3D reconstruction, a {frontal_image_name}_rect.txt file is required. If the input images are center-aligned, a .txt file with the contents '0 0 {image_height} {image_width}' should suffice. If the input images are not centered-aligned, refer to the original Testing section to obtain {frontal_image_name}_rect.txt.

Testing

Run the following command for testing:

python -m apps.simple_test -i {folder_containing_images_and_rect} -b {path_of_backside_image} --use_rect 

Modification

Major code modifications:

    .
    ├── apps
    |   ├── simple_test.py         # command line argument for a backside image 
    |   └── recon.py               # feed a backside image to PIFuHD 
    └── lib                        
        ├── model                  
        |   └── HGPIFuNetwNML.py   # obtain a backside normal map and apply AdaIN.
        └── data                   
            └── EvalDataset.py     # image preprocessing for a backside image
    

License

CC-BY-NC 4.0. See the LICENSE file.