Lenslet white image guided demosaicing for plenoptic cameras

P. David, M. Le Pendu, C. Guillemot,
"Lenslet white image guided demosaicing for plenoptic cameras",
IEEE Multimedia Signal Processing (MMSP) workshop, Oct. 2017.(pdf)


Most modern cameras use a color filter array on their sensor in order to capture color images. This array is composed of red, green and blue filters and so, each pixel on the sensor lacks two color channels which can be retrieved by a process called demosaicing. In this paper, we propose a new demosaicing method for plenoptic cameras. This type of cameras has become a growing trend and their captured raw images have a particular lenslet structure which must be taken into account to retrieve the sub-aperture images which compose the light field. First, we analyze and describe the flaws of the state-of-the-art light field decoding pipeline. To better identify the different sources of artifacts, our analysis is performed by generating ideal lenslet images from synthetic light fields and use them as input of the decoding pipeline. Then, we detail a new method of demosaicing based on the provided white lenslet images serving as guide. Furthermore, we show that this kind of guided interpolation can be useful on other steps of the decoding pipeline. Finally, the quality of the resulting sub-aperture images is assessed for both synthetic and real light fields using visual comparisons as well as objective metrics.

Light field decoding pipeline analysis with synthetic data

Light field decoding pipeline

The light field decoding pipeline comprises several steps:
  • Devignetting the lenslet image
  • Demosaicing the lenslet image
  • Aligning the pixels with the lenslets
  • Demultiplexing the views
  • Resampling the views

Synthetic lenslet image generation

We developed a method to make a synthetic lenslet image from a synthetic light field. We first remove some peripheral views (in the corners), in order to have circular angular patches (as the captured plenoptic light field are). Then we put the different angular patches near one another along a hexagonal grid. This operation is the exact contrary of the slicing or demultiplexing step in the decoding pipeline. We also add vignetting to the lenslets and generate a white image that holds the exact vignetting profile of the lenslet image (dividing the lenslet image by the white image will give us perfect devignetting).

White image guided demosaicing algorithm

We propose to discard the pixels that are out of the lenslets from the demosaicing step. To do this, we adapted the gradient corrected interpolation method by weighting the bilinear interpolation and gradient correction:
  • First with a mask (b): we do not want to interpolate data from different lenslets as this creates crosstalk artifacts. Knowing the lenslet grid parameters, we know the position of the lenslets centers and we can identify the pixels which belong to the same lenslets.
  • Then with a white image (c): we have full confidence in the pixels that have the maximum values on the white image whereas we have less confidence in the pixels that are darker on the white image, as they are noisier.

Results with synthetic and real light fields

Evaluation of the proposed method with synthetic data

Left: Ground truth light field. Center: Light field decoded with Dansereau's methods. Right: Light field decoded with our methods.
Butterfly Still Life
Buddha Mona

Evaluation of the proposed method with real light fields

Lytro 1

Beers Tape Measure


Bee 1 Rose

Download the Matlab functions replacing the function LFDecodeLensletImageSimple in the D. Dansereau's Light Field Toolbox