Smartphone-based-RTI

Geometric and 3D Computer Vision 2021

See G3DCV2021_FinalProject.pdf for more details.

1. Introduction

Reflectance Transformation Imaging (RTI) is a technique to capture the reflectance of an object under different lighting conditions. It enables the interactive relighting of the subject from any direction and the mathematical enhancement of subject’s surface shape and color attributes. Typically, RTI uses a static camera and an array of light sources at known (or knowable) positions.

2. Assignment

In this project, you’ll explore RTI using a cheaper setup composed by two consumer smartphones equipped with a camera and LED flashlight. The idea is to use one of the two smartphones as a movable light source while the other captures the object from a frontal position. We exploit the camera of the movable light source to recover the light direction vector at each point and provide the same data obtainable with a light dome.

2.1. Requirements

The project must contain 3 different programs:

1. A camera_calibrator that loads one of the two provided calibration videos and computes the intrinsic parameters of the camera (the intrinsic matrix K and the lens distortion vector assuming a 5-parameters model) without any user intervention.
2. The analysis program to process a video sequence and compute a texture map of the object (ie a function, for each object pixel, mapping light direction to pixel intensity). The whole analysis must be performed without any user intervention. Tunable parameters should not depend on the specific sequence processed.
3. An interactive_relighting program to render a previously processed object according to a user-definable light source direction

2.2. Assets

Download the assets folder from here.
Create the assets/ folder and extract the G3DCV2021_data archive on the root folder.
The G3DCV2021_data folder is also stored on the repository into the assets release.

3. Setup Project

Here a small description of the developed project done, and how it works.
First download the assets as described before.

3.1 Libraries

This project use python 3.7.
Download the required project libraries.
Here the commands to import required libraries with conda
conda install numpy matplotlib
conda install -c conda-forge opencv Note: must be 4.2.0 or greater
conda install -c anaconda scipy
conda install ffmpeg-python
conda install -c conda-forge pysoundfile
conda install -c conda-forge moviepy
conda install -c conda-forge svglib
conda install -c anaconda reportlab
conda install scikit-image
conda install -c conda-forge python-dotenv

GPU Libraries (optional)

Note: this is required to run on GPU
conda install numba
conda install cudatoolkit

3.2 Project Structure

The project contains the 3 different programs required from the assignment.

Main programs

1. camera_calibrator.py: basic program that read the two chessboard images and save camera intrinsics into assets folder.
2. analysis.py: core program that perform different tasks, it can require some time to finish (1/2 hours):
   1. sync_videos(): extract the audio offset between the static and moving video (required for sync)
   2. generate_video_default_frame(): extract and save a default frame from static video (a frame with no light)
   3. extract_video_frames(): extract the video frames with the class FeatureMatcher This class will compute the corner detection, intensity extraction and get the camera pose of given frames (static and moving).
      Results are saved on storage and can be reused.
   4. compute_intensities(): compute light vectors intensities for each frame pixel.
   5. interpolate_intensities(): compute the interpolation of pixel intensities, two interpolation function are supported:
      1. Polynomial Texture Maps (PTM)
      2. Linear Radial Basis Function (RBF)
   6. prepare_images_data(): prepare images for each camera position (normalized) with interpolated values. This process is required in order to speed up relighting program.
   7. Save interpolation results on storage (can require some GB)
3. interactive_relighting.py: the final program, require the analysis results in order to execute. It reads the interpolation results and render the relighted image dynamically based on the light direction specified with the cursor.

Utils

1. audio_utils.py: contains utility functions regarding audio management.
2. email_utils.py: contains utility functions regarding email management.
3. image_utils.py: contains utility functions regarding image and video management.
4. utilities.py: contains misc utility functions.

3.3. Parameters

Each program has some tunable parameters on compute() function:

1. camera_calibrator.py: no tunable parameters.
2. analysis.py:
   1. video_name: specify the name of the video to analyze.
   2. from_storage: specify whether to read a previous extract_video_frames() results, in this case analysis will start from compute_intensities() phase.
   3. storage_filepath: specify a different path whether to read extract_video_frames() results
   4. interpolate_PTM: specify the interpolation function to use
   5. notification_email: specify if send a notification email when the computation is finish.
      Note in order to use this parameter you should rename .env.example to .env and fill it with valid google account credentials. The env file is used in order to keep credentials off-repo.
   6. debug: specify whether to compute analysis on debug mode (analyze first pixel only on interpolation)
3. interactive_relighting.py:
   1. video_name: specify the name of the video to relight. storage_filepath: specify a different path whether to read analysis() results.

3.4. Run and Test

Each program have a compute() function.
In order to test each program individually there is also a main() function, that will simply call the compute() function with some pre-set parameters.

• To run all the 3 programs sequentially from skratch simply use $ python main.py.
• To run them separately use $ python {program_name}.py, where program_name is one of the 3 main programs.