Deep Light Field Acquisition Using Learned CodedMask Distributions for Color Filter Array Sensors
 

G. Le Guludec, E. Miandji, C. Guillemot

Deep Light Field Acquisition Using Learned CodedMask Distributions for Color Filter Array Sensors,
IEEE Trans. on computational imaging, vol. 7, pp. 475-488, 2021, May 2021.


Abstract

Compressive light field photography enables light field acquisition using a single sensor by utilizing a color coded mask. This approach is very cost effective since consumer-level digital cameras can be turned into a light field camera by simply placing a coded mask between the sensor and the aperture plane and solving an inverse problem to obtain an estimate of the original light field. This paper describes a deep learning architecture for compressive light field acquisition using a color coded mask and a sensor with Color Filter Array (CFA). Unlike previous methods where a fixed mask pattern is used, our deep network learns the optimal distribution of the color coded mask pixels. The proposed solution enables end-to-end learning of the color-coded mask distribution, the CFA, and the reconstruction network. Consequently, the resulting network can efficiently perform joint demosaicing and light field reconstruction of images acquired with color-coded mask and a CFA sensor. Compared to previous methods based on deep learning with monochrome sensors, as well as traditional compressive sensing approaches using CFA sensors, we obtain superior color reconstruction of the light fields.

Overview of the method


Results on the Kalantari and Stanford dataset

Light fields from the dataset of Nima Khademi Kalantari and Ravi Ramamoorthi. Deep High Dynamic Range Imaging of Dynamic Scenes, ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017). and the Dataset of Lytro Illum images by Abhilash Sunder Raj, Michael Lowney, Raj Shah, and Gordon Wetzstein

Buttercup




Ground truth Ours (PSNR = 32.37 dB) Nabati (PSNR = 29.98 dB)

Cars




Ground truth Ours (PSNR = 31.23 dB) Nabati (PSNR = 29.88 dB)

Orchids




Ground truth Ours (PSNR = 32.94 dB) Nabati (PSNR = 30.99 dB)

Rock




Ground truth Ours (PSNR = 32.20 dB) Nabati (PSNR = 30.11 dB)

Seahorse




Ground truth Ours (PSNR = 33.40 dB) Nabati (PSNR = 32.36 dB)

Tulips




Ground truth Ours (PSNR = 42.85) Nabati (PSNR = 38.26 dB)

White Rose




Ground truth Ours (PSNR = 32.95 dB) Nabati (PSNR = 31.84 dB)

Additional results on the HCI dataset

Light fields from Wanner, S. Meister, and B. Goldluecke, Datasets and benchmarks for densely sampled 4D light fields, Conf. on Vision, Modeling &Visualization (VMV), 2013, pp. 225226. Note that these light fields come from a very different distribution than the one used for training.

Monas Room (from the old HCI dataset)



Ground truth Ours (PSNR = 34.72 dB)

Boxes



Ground truth Ours (PSNR = 31.00 dB)

Cotton



Ground truth Ours (PSNR = 41.04 dB)

Dino



Ground truth Ours (PSNR = 33.78 dB)

Sideboard



Ground truth Ours (PSNR = 25.28 dB)