Several types of devices exist for capturing real light fields, which go from arrays of cameras capturing the scene from slightly different viewpoints, to single cameras mounted on moving gantries and plenoptic cameras, using arrays of microlenses in front of the photosensor to obtain angular information about the captured scene.

In this project we consider both light fields captured by arrays of cameras (sparse light fields with large baselines) and by micro-lenses based cameras (plenoptic cameras) leading to dense light fields with narrow baselines, the two types of light fields presenting different characteristics. The problems addressed are in particular:

  • Compressive acquisition, low rank and sparse approximation of light fields
  • Graph-based representation and compression of light fields
  • Learning dictionaries, subspaces, manifolds for light fields
  • Scene analysis from light fields: depth estimation, scene flow analysis
  • Compression and restoration
  • Light field editing: segmentation, edit propagation, inpainting, super-resolution, restoration
  • Light fields acquisition

    Decoding Raw Light Fields Captured by Plenoptic Cameras

    The raw light fields data captured by plenotic cameras is a lenslet image from which sub-aperture images (or views) can be extracted. A matlab implementation of this decoding pipeline is available in the matlab light field toolbox. The decoding process which extracts the sub-aperture images from the lenslet image, includes several steps: de-vignetting, color de-mosaicking, conversion of the hexagonal to a rectangular sampling grid, and colour correction. We have first analyzed the different steps of this decoding pipeline to identify the issues which lead to various artefacts in the extracted views. This analysis led us to propose a method for white image guided color demosaicing of the lenslet image. Similarly, we have proposed an interpolation guided by the white image for aligning the micro-lens array and the sensor. [More here ...]

    High Dynamic Range Light Fields Recovery from Multiple Exposures

    Building upon the homography-based low rank approximation (HLRA) method we proposed (below) for light field compact representation and compression, we have, in collaboration with Trinity College Dublin, developed a method for capturing High Dynamic Range (HDR) light fields with dense viewpoint sampling from multiple exposures. The problem of recovering saturated areas is formulated as a Weighted Low Rank Approximation (WLRA) where the weights are defined from the soft saturation detection. The proposed WLRA method generalizes the matrix completion algorithm developed in the project (see below) for light fields inpainting.[More here ...]

    Compressive Acquisition of Light Fields

    In collaboration with the Univ. of Linkoping, we have developed a compressive sensing framework for reconstructing the full light field of a scene captured using a single-sensor consumer camera. To achieve this, we use a color coded mask in front of the camera sensor. To further enhance the reconstruction quality, we utilize multiple shots by moving the mask or the sensor randomly. The compressed sensing framework relies on a training based dictionary over a light field data set. Numerical simulations show significant improvements in reconstruction quality over a similar coded aperture system for light field capture. [More here ...]

    Scene analysis from light fields

    Sparse to dense estimation of scene flow from light fields

    We have addressed the problem of scene flow estimation from sparsely sampled video light fields. The scene flow estimation method is based on an affine model in the 4D ray space that allows us to estimate a dense flow from sparse estimates in 4D clusters. A dataset of synthetic video light fields created for assessing scene flow estimation techniques is also described. Experiments show that the proposed method gives error rates on the optical flow components that are comparable to those obtained with state of the art optical flow estimation methods, while computing a more accurate disparity variation when compared with prior scene flow estimation techniques. The dataset can be accessed [ here.]

    Depth estimation with occlusion handling from a sparse set of light field views

    We have addressed the problem of depth estimation for every viewpoint of a dense light field, exploiting information from only a sparse set of views. This problem is particularly relevant for applications such as light field reconstruction from a subset of views, for view synthesis and for compression. Unlike most existing methods for scene depth estimation from light fields, the proposed algorithm computes disparity (or equivalently depth) for every viewpoint taking into account occlusions. In addition, it preserves the continuity of the depth space and does not require prior knowledge on the depth range. The experiments show that, both for synthetic and real light fields, our algorithm achieves competitive performance to state-of-the-art algorithms which exploit the entire light field and usually yield to the depth map for the center viewpoint only. [More here ...]

    A Learning Based Depth Estimation Framework for 4D Densely and Sparsely Sampled Light Fields

    We have developed a learning based solution to disparity (depth) estimation for either densely or sparsely sampled light fields. Disparity between stereo pairs among a sparse subset of anchor views is first estimated by a fine-tuned FlowNet 2.0 network adapted to disparity prediction task. These coarse estimates are fused by exploiting the photo-consistency warping error, and refined by a Multi-view Stereo Refinement Network (MSRNet). The propagation of disparity from anchor viewpoints towards other viewpoints is performed by an occlusion-aware soft 3D reconstruction method. The experiments show that, both for dense and sparse light fields, our algorithm outperforms significantly the state-of-the-art algorithms, especially for subpixel accuracy. [More here ...]

    Framework for Learning Scene Depth from a Flexible Subset of Dense and Sparse Light Field Views

    we propose a learning based depth estimation framework suitable for both densely and sparsely sampled light fields. The proposed framework consists of three processing steps: initial depth estimation, fusion with occlusion handling, and refinement. The estimation can be performed from a flexible subset of input views. The fusion of initial disparity estimates, relying on two warping error measures, allows us to have an accurate estimation in occluded regions and along the contours. In contrast with methods relying on the computation of cost volumes, the proposed approach does not need any prior information on the disparity range. Experimental results show that the proposed method outperforms state-of-the-art light fields depth estimation methods, including prior methods based on deep neural architectures. [More here ...]

    Denoising of 3D Point Clouds Constructed From Light Fields

    The availability of multiple views enables scene depth estimation, that can be used to generate 3D point clouds. The constructed 3D point clouds, however, generally contain distortions and artefacts primarily caused by inaccuracies in the depth maps. This paper describes a method for noise removal in 3D point clouds constructed from light fields. While existing methods discard outliers, the proposed approach instead attempts to correct the positions of points, and thus reduce noise without removing any points, by exploiting the consistency among views in a light-field. The proposed 3D point cloud construction and denoising method exploits uncertainty measures on depth values. We also investigate the possible use of the corrected point cloud to improve the quality of the depth maps estimated from the light field. [More here ...]

    View synthesis

    A Lightweight Neural Network for Monocular View Generation with Occlusion Handling

    We have developed a very lightweight neural network architecture, trained on stereo data pairs, which performs view synthesis from one single image. With the growing success of multi-view formats, this problem is increasingly relevant. The network returns a prediction built from disparity estimation, which fills in wrongly predicted regions using a occlusion handling technique. To do so, during training, the network learns to estimate the left-right consistency structural constraint on the pair of stereo input images, to be able to replicate it at test time from one single image. At test time, the approach can generate a left-side and a right-side view from one input image, as well as a depth map and a pixelwise confidence measure in the prediction. The work outperforms visually and metric-wise state-of-the-art approaches on the challenging KITTI dataset, all while reducing by a very significant order of magnitude the required number of parameters (6.5 M). [More here ...]

    Representations of Light Fields: Low rank, sparse and graph representations

    A Fourier Disparity Layer representation for Light Fields

    In collaboration with Trinity College Dubli, we have developed a new Light Field representation for efficient Light Field processing and rendering called Fourier Disparity Layers (FDL). The proposed FDL representation samples the Light Field in the depth (or equivalently the disparity) dimension by decomposing the scene as a discrete sum of layers. The layers can be constructed from various types of Light Field inputs including a set of sub-aperture images, a focal stack, or even a combination of both. From our derivations in the Fourier domain, the layers are simply obtained by a regularized least square regression performed independently at each spatial frequency, which is efficiently parallelized in a GPU implementation. Our model is also used to derive a gradient descent based calibration step that estimates the input view positions and an optimal set of disparity values required for the layer construction. Once the layers are known, they can be shifted and filtered to produce different viewpoints of the scene while controlling the focus and simulating a camera aperture of arbitrary shape and size, and used for real time Light Field rendering, view interpolation, extrapolation, and denoising.[More here ...]

    Homography-based Low Rank Approximation for Light Fields Compression

    We study the problem of low rank approximation of light fields for compression. A homography-based approximation method has been developed which jointly searches for homographies to align the different views of the light field together with the low rank approximation matrices. A coding algorithm relying on this homography-based low rank approximation has then been designed. The two key parameters of the coding algorithm (rank and quantization parameter) are, for a given target bit-rate, predicted with a model learned from input light fields texture and disparity features, using radom trees. The results show the benefit of the joint optimization of the homographies together with the low-rank approximation as well as PSNR-rate performance gains compared with those obtained by directly applying HEVC on the light field views re-structured as a video sequence.[More here ...]

    Geometry-Aware Graph Transforms for Light Field Compact Representation

    We have addressed the problem of energy compaction of dense 4D light fields by designing geometry-aware local graph-based transforms. Local graphs are constructed on super-rays that can be seen as a grouping of spatially and geometry-dependent angularly correlated pixels. Both non separable and separable transforms are considered. Despite the local support of limited size defined by the super-rays, the Laplacian matrix of the non separable graph remains of highdimension and its diagonalization to compute the transform eigen vectors remains computa-tionally expensive. To solve this problem, we then perform the local spatio-angular transform in a separable manner. We show that when the shape of corresponding super-pixels in the different views is not isometric, the basis functions of the spatial transforms are not coherent, resulting in decreased correlation between spatial transform coefficients. We hence propose a novel transform optimization method that aims at preserving angular correlation even when the shapes of the super-pixels are not isometric. Experimental results show the benefit of theapproach in terms of energy compaction. A coding scheme is also described to assess the rate-distortion perfomances of the proposed transforms and is compared to state of the art encodersnamely HEVC and JPEG Pleno VM 1.1 [More here ...]

    Prediction and Sampling with Local Graph Transforms forQuasi-Lossless Light Field Compression

    Graph-based transforms have been shown to be powerful tools in terms of image energy compaction. However, when the support increases to best capture signal dependencies, the computation of the basis functions becomes rapidly untractable. This problem is in particular compelling for high dimensional imaging data such as light fields. The use of local transforms with limited supports is a way to cope with this computational difficulty. Unfortunately, the locality of the support may not allow us to fully exploit long term signal dependencies present in both the spatial and angular dimensions in the case of light fields. This paper describes sampling and prediction schemes with local graph-based transforms enabling to efficiently compact thesignal energy and exploit dependencies beyond the local graph support. The proposed approach is investigated and is shown to be very efficient in the context of spatio-angular transforms for quasi-lossless compression of light fields.[More here ...]

    Graph-based Representation of Light Fields

    We have explored the use of graph-based representations for light fields. The graph connections are derived from the disparity and hold just enough information to synthesize other sub-aperture images from one reference image of the light field. Based on the concept of epipolar segment, the graph connections are sparsified (less important segments are removed) by a rate-distortion optimization. The graph vertices and connections are compressed using HEVC. The graph connections capturing the inter-view dependencies are used as the support of a Graph Fourier Transform used to encode disoccluded pixels. [More here ...]

    Graph-based Transforms for Predictive Light Field Compression based on Super-Pixels

    We have explored the use of graph-based transforms to capture correlation in light fields. We consider a scheme in which view synthesis is used as a first step to exploit inter-view correlation. Local graph-based transforms (GT) are then considered for energy compaction of the residue signals. The structure of the local graphs is derived from a coherent super-pixel over-segmentation of the different views. The GT is computed and applied in a separable manner with a first spatial unweighted transform followed by an inter-view GT. For the inter-view GT, both unweighted and weighted GT have been considered. The use of separable instead of non separable transforms allows us to limit the complexity inherent to the computation of the basis functions. A dedicated simple coding scheme is then described for the proposed GT based light field decomposition. Experimental results show a significant improvement with our method compared to the CNN view synthesis method and to the HEVC direct coding of the light field views.[More here ...]

    Rate-Distortion Optimized Super-Ray Merging for Light Field Compression

    We describe a method for constructing super-rays to be used as the support of a 4D shape-adaptive transforms. Super-rays are used to capture inter-view and spatial redundancy in light fields. Here, we consider a scheme in which a first step of view synthesis based on CNN is used to remove inter-view correlation. The super-ray based transforms are then applied on prediction residues. To ensure that the super-ray segmentation is highly correlated with the residues to be encoded, the super-rays are computed on synthesized residues and optimized in a rate-distortion sense. Experimental results show that the proposed coding scheme outperforms HEVC-based schemes at low bitrate.[More here ...]

    Light Fields editing

    Light fields Inpainting via low rank matrix completion

    We have developed an novel method for propagating the inpainting of the central view of a light field to all the other views in order to inpaint all views in a consistent manner. After generating a set of warped versions of the inpainted central view with random homographies, both the original light field views and the warped ones are vectorized and concatenated into a matrix. Because of the redundancy between the views, the matrix satisfies a low rank assumption enabling us to fill the region to inpaint with low rank matrix completion. A new matrix completion algorithm, better suited to the inpainting application than existing methods, has also been developed. Unlike most existing light field inpainting algorithms, our method does not require any depth prior. Another interesting feature of the low rank approach is its ability to cope with color and illumination variations between the input views of the light field. .[More here ...]

    Light fields Inpainting via PDE-based diffusion in epipolar plane images

    This paper presents a novel approach for light field editing. The problem of propagating an edit from a single view to the remaining light field is solved by a structure tensor driven diffusion on the epipolar plane images. The proposed method is shown to be useful for two applications: light field inpainting and recolorization. While the light field recolorization is obtained with a straightforward diffusion, the inpainting application is particularly challenging, as the structure tensors accounting for disparities are unknown under the occluding mask. We address this issue with a disparity inpainting by means of an interpolation constrained by superpixel boundaries. Results on synthetic and real light field images demonstrate the effectiveness of the proposed method. .[More here ...]

    Fast light field inpainting using angular warping with a color-guided disparity interpolation

    This paper describes a method for fast and efficient inpainting of light fields. We first revisit disparity estimation based on smoothed structure tensors and analyze typical artefacts with their impact for the inpainting problem. We then propose an approach which is computationally fast while giving more coherent disparity in the masked region. This disparity is then used for propagating, by angular warping, the inpainted texture of one view to the entire light field. Performed experiments show the ability of our approach to yield appealing results while running considerably faster.[More here ...]

    Light Fields restoration and super-resolution

    4D Anisotropic Diffusion Framework with PDEs for Light Field Regularization and Inverse Problems

    We have considered the problem of vector valued regularization of light fields based on PDEs. We propose a regularization method operating in the 4D ray space that does not require prior estimation of disparity maps. The method performs a PDE-based anisotropic diffusion along directions defined by local structures in the 4D ray space. We analyze light field regularization in the 4D ray space using the proposed 4D anisotropic diffusion framework, and illustrate its effect for several light field processing applications: denoising, angular and spatial interpolation, regularization for enhancing disparity estimation as well as inpainting. [More here ...]

    Light Fields Denoising using 4D Anisotropic Diffusion

    We have developed a novel light field denoising algorithm using a vector-valued regularization operating in the 4D ray space. The method performs a PDE-based anisotropic diffusion along directions defined by local structures in the 4D ray space. It does not require prior estimation of disparity maps. The local structures in the 4D light field are extracted using a 4D tensor structure. Experimental results show that the proposed denoising algorithm performs well compared to state of the art methods while keeping tractable complexity.[More here ...]

    Light fields Super-Resolution based on projections between subspaces of patch volumes

    We have developed an example-based super-resolution algorithm for light fields, which allows the increase of the spatial resolution of the different views in a consistent manner across all sub-aperture images of the light field. The algorithm learns linear projections between subspaces of reduced dimension in which reside patch-volumes extracted from the light field. The method is extended to cope with angular super-resolution, where 2D patches of intermediate sub-aperture images are approximated from neighbouring subaperture images using multivariate ridge regression. Experimental results show significant quality improvement when compared to state-of-the-art single-image super-resolution methods applied on each view separately, as well as when compared to a recent light field super-resolution technique based on deep learning.[More here ...]

    Light fields Super-Resolution based on deep convolutional networks with low rank priors

    This paper describes a learning-based spatial light field super-resolution method that allows the restoration of the entire light field with consistency across all sub-aperture images. The algorithm first uses optical flow to align the light field and then reduces its angular dimension using low-rank approximation. We then consider the linearly independent columns of the resulting low-rank model as an embedding, which is restored using a deep convolutional neural network (DCNN). The super-resolved embedding is then used to reconstruct the remaining sub-aperture images. The original disparities are restored using inverse warping where missing pixels are approximated using a novel light field inpainting algorithm. Experimental results show that the proposed method outperforms existing light field super-resolution algorithms, including using convolutional networks.[More here ...]