Neural-DP-Simulator

[ICCP'23]: Learning to Synthesize Photorealistic Dual-pixel Images from RGBD Frames

Authors: Feiran Li*, Heng Guo*, Hiroaki Santo, Fumio Okura, Yasuyuki Matsushita (*equal contribution)

TODO:

1. We have only released the processed (i.e., RGB) data currently. The RAW images (~700GB) are too large and we are working on the release.

Dataset

• Download the processed RGB dataset: 16-bit version from DropBox or 8-bit version from Kaggle, 16-bit version from OneDrive. Note that the bit depth difference only applies to the RGB image and depth maps are all in 16 bit.

• Each timestamp-named folder refers to one scenario. Within it:

  • meta_data.h5 records various info such as focal length, focus distance, and calibration data
  • cam0 and cam1 contain the RGB DP images and depth maps of the left-right cameras, respectively. All RGB images are saved in 16-bit format. The depth maps are also saved in 16-bit format in mm unit.

• data_processing/ contains the code we used for processing the data acquired from the camera.

Our simulator

our_simulator/ contains the code for training & testing & inferring using our DP-Simulator