This repository contains the calibration code (Calibration) and the deep learning code (Learning Tasks) for the paper Beyond the Pixel: a Photometrically Calibrated HDR Dataset for Luminance and Color Prediction
pip install pytorch_lightning matplotlib skylibs glob2 opencv-contrib-python collections omegaconf natsort configargparseAdditionally, MATLAB is required for the HDR-VDP-3 metric.
To run the calibration process for your own camera setup as it is done in the paper, follow the README in the Calibration folder.
The pipeline used for learning depends on the desired task.
Per-pixel luminance outputs an HDR image and the scale needed to bring it to absolute luminance.
Per-pixel color predicts the temperature map directly.
Planar illuminance generate only the illuminance scalar.
If working with the full dataset (available at http://hdrdb.com/indoor-hdr-photometric/), it needs to be split in train/test/val , inpainted and rescaled to manageable size. The following script automate this to the setup used in the paper.
python prepare_dataset.py [path_to_full_dataset]
For convenience, 3 config files are provided in configs/, one for each task.
python train.py --config [config_file]
When testing, it is best to link to the config file generated by the training script (by default in checkpoints/[name]/lightning_logs/version_[x]/config.txt)
The test.py script generate the inference predictions.
python test.py --config [config_file]
The metrics.py script computes the metrics and generate visualisations from the inference predictions.
python metrics.py --config [config_file]
When fine-tuning, the config file must be modified to increase the n_epoch property.
When fine-tuning, it is best to link to the config file generated by the training script (by default in checkpoints/[name]/lightning_logs/version_[x]/config.txt)
python fine-tune.py --config [config_file]
If you wish to test or fine-tune the weights used in the paper, you can download them and place them in the checkpoints/ folder.
https://hdrdb-public.s3.valeria.science/indoor_photometric/[Experiment_name].zip
| Experiment_name | Mode | In paper | Link |
|---|---|---|---|
| Luminance_linear | Luminance | Table 1 | link |
| Luminance_gamma | Luminance | Table 1 | link |
| Luminance_noise | Luminance | Table 1 | link |
| Luminance_quantized | Luminance | Table 1 | link |
| Luminance_LDR | Luminance | Table 1 | link |
| Luminance_LDR_fine_tune | Luminance | Table 3 | link |
| Temperature_WB_rand_augment | Temperature | Figure 7 | link |
| Temperature_WB_augment_theta_fine_tune | Temperature | Table 3 | link |
| illum_hemi_HDR | Illuminance | Table 2 | link |
| illum_hemi_LDR_scale | Illuminance | Table 2 | link |
| illum_hemi_LDR | Illuminance | Table 2 | link |
| illum_120_HDR | Illuminance | Table 2 | link |
| illum_120_LDR_scale | Illuminance | Table 2 | link |
| illum_120_LDR | Illuminance | Table 2 | link |
| illum_60_HDR | Illuminance | Table 2 | link |
| illum_60_LDR_scale | Illuminance | Table 2 | link |
| illum_60_LDR | Illuminance | Table 2 | link |
| illum_rand_HDR | Illuminance | Table 2 | link |
| illum_rand_LDR_scale | Illuminance | Table 2 | link |
| illum_rand_LDR | Illuminance | Table 2 | link |
| illum_120_LDR_theta | Illuminance | Table 3 | link |
Beyond the Pixel: a Photometrically Calibrated HDR Dataset for Luminance and Color Prediction
Christophe Bolduc, Justine Giroux, Marc Hébert, Claude Demers, Jean-François Lalonde
International Conference on Computer Vision (ICCV), 2023
Project page / Paper / Dataset / BibTeX