High Dynamic Range Light Fields via Weighted Low Rank Approximation

Inpainting principle
M. Le Pendu, A. Smolic, C. Guillemot,

"High Dynamic Range Light Fields via Weighted Low Rank Approximation", IEEE Internation Conf. on Image Processing, submitted, 2018.(pdf)

Collaboration with Trinity College Dublin


In this paper, we propose a method for capturing High Dynamic Range (HDR) light fields with dense viewpoint sampling. Analogously to the traditional HDR acquisition process, several light fields are captured at varying exposures with a plenoptic camera. The RAW data is de-multiplexed to retrieve all light field viewpoints for each exposure and perform a soft detection of saturated pixels. Considering a matrix which concatenates all the vectorized views, we formulate the problem of recovering saturated areas as a Weighted Low Rank Approximation (WLRA) where the weights are defined from the soft saturation detection. We show that our algorithm successfully recovers the parallax in the over-exposed areas while the Truncated Nuclear Norm (TNN) minimization, traditionally used for single view HDR imaging, does not generalize to light fields. Advantages of our weighted approach as well as the simultaneous processing of all the viewpoints are also demonstrated in our experiments.

Soft Saturation Detection

The plenoptic RAW data is first de-multiplexed into light field views using the toolbox of [1] with the enhancements of [2]. Soft saturation detection is performed directly on sensor data (i.e. before the devignetting and demosaicing steps). For a pixel of normalized value x, we compute its saturation \begin{equation} \begin{aligned} s(x) = \min((x+(1-\tau))^{12}, 1). \end{aligned} \end{equation}
In practice, even if a pixel is not fully saturated (x<1), neighboring pixels associated to the other color components on the bayer pattern may saturate. Hence, the level of full saturation τ is set to 0.9 to account for unreliable colors of high pixel values. Similarly to the color data, the sensor saturation image is de-multiplexed using [1] (without devignetting and demosaicing) to obtain a saturation map of each light field view.

Weighted Low Rank Approximation

In order to cope with possible movements of the camera when capturing the different exposures, all the views of the low exposure light fields are aligned to the high exposure one using HLRA [3]. For fast computations, homography parameters are determined only from the central views of the high and low exposure light fields. Prior to WLRA, we convert RGB data to luminance Y and CIE chromaticity components u'v'. The saturation s is converted into weights w as: \begin{equation} \begin{aligned} w=\frac{1-s}{s+1/w_{max}}. \end{aligned} \end{equation} The non-saturated pixels (s = 0) are thus associated to the maximum weight indicating a high confidence, while the fully saturated pixels (s = 1) have a null weight. In our experiments, we use wmax = 100. Finally, weights and image data are arranged into respective matrices, each column contains a vectorized light field view.

Our WLRA method generalizes the matrix completion algorithm in [4] to the case of non-binary weights. Given the matrix M to approximate, a global noise tolerance parameter ε, and the element-wise weight matrix W, the problem formulation is: \begin{equation} \begin{aligned} \min_{X} &\quad \mathrm{rank}(X)\\[-2pt] \text{s.t.} &\quad X=Z\\[-3pt] &\quad ||{W\circ(Z-M)}||_F^2\leq\epsilon, \end{aligned} \label{eq:WLRAeq} \end{equation} where the operator "o" is the element-wise multiplication. The introduction of the matrix Z and the constraint X=Z makes it possible to use the Alternating Direction Method of Multipliers. For more details on the way the problem is solved, please refer to our paper.

Experimental Results

Evaluation of the Rank Minimization compared with the Truncated Nuclear Norm Minimization

In single view HDR imaging, existing low rank based methods minimize the Truncated Nuclear Norm (TNN) with r = 1 as the matrix rank can be assumed to be 1. However, this assumption does not hold for light fields with disparities between views, and produces blur in the completed regions as shown in Figures (c),(g) below. Using higher values of r keeps the details but not the low spatial frequencies (Figures (a),(b),(e),(f)). Our rank minimization successfully recovers both low and high frequencies (Figures (d),(h)).

The figure below shows: (a)-(d) show the central view in false color for each method of a 13x13 HDR Light field reconstructed from 3 exposures. (e),(f),(g),(h) are the epipolar images corresponding to the blue segment in (a),(b),(c),(d) respectively (vertical axis shows vertical viewpoint change). The epipolar image in (g) shows that the structures completed using TNN minimization with r=1 are blurred and identical in all the views (parallax is not preserved).

Recovery of saturated areas from two exposures

One view of a HDR light field captured from two significantly different exposures (images are gamma encoded for display purpose). Some over-exposed areas in the high exposure capture are under-exposed in the low exposure one. Independent processing of the viewpoints (a) highlights the presence of noise due to underexposition. Using all the viewpoints in the WLRA (b) naturally exploits redundancies to reduce the noise.

Video of (left) original light field, (right) HDR reconstructed light field represented in false colors to show the coherency between the views.

HDR light field reconstructed
To play the video click on the image

Video showing the sub-apertures with a progressive change of exposure.

HDR light field reconstructed
To play the video click on the image


  1. D. G. Dansereau, O. Pizarro, and S. B. Williams, Decoding, calibration and rectification for lenselet-based plenoptic cameras,in Proc. CVPR, 2013, pp. 1027–1034.
  2. P. David, M. Le Pendu, and C. Guillemot, White lenslet image guided demosaicing for plenoptic cameras, in IEEE 19th International Workshop on Multimedia Signal Processing (MMSP), Oct. 2017, pp.1-6.
  3. X. Jiang, M. Le Pendu, R. A. Farrugia, and C. Guillemot, Light field compression with homography-based low-rank approximation, IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 7, pp. 1132-1145, Oct. 2017.
  4. M. Le Pendu, X. Jiang, and C. Guillemot, Light field inpainting propagation via low rank matrix completion, IEEE Transactions on Image Processing, vol. 27, no. 4, pp.1981-1993, Apr. 2018