4D anisotropic diffusion framework with PDEs for light field regularization and inverse problems
Pierre Allain, Laurent Guillo, Christine Guillemot, 4D Anisotropic Diffusion Framework with PDEs
for Light Field Regularization and Inverse Problems, IEEE Trans. on Computational Imaging, vol. 6, pp. 109-124, June 2019.
[preprint].
Abstract
In this paper, we consider the problem of vector-valued regularization of light fields based on PDEs.
We propose a regularization method operating in the 4D ray space that does not require prior estimation of disparity maps.
The method performs a PDE-based anisotropic diffusion along directions defined by local structures in the 4D ray space.
We analyze light field regularization in the 4D ray space using the proposed 4D anisotropic diffusion framework by first considering a light field toy example, i.e., a tesseract.
This simple light field example allows an in-depth analysis of how each eigenvector influences the diffusion process.
We then illustrate the diffusion effect for several light field processing applications: denoising, angular and spatial interpolation, regularization for enhancing disparity estimation as well as inpainting.
Denoising
Light Fields Denoising using 4D Anisotropic Diffusion
We have developed a novel light field denoising algorithm using a vector-valued regularization operating in the 4D ray space.
The method performs a PDE-based anisotropic diffusion along directions defined by local structures in the 4D ray space.
It does not require prior estimation of disparity maps.
The local structures in the 4D light field are extracted using a 4D tensor structure.
Experimental results show that the proposed denoising algorithm performs well compared to state of the art methods while keeping tractable complexity.
[More here ...]
Miscellaneous denoising
Dithering quantization
Light field quantized on 3 bits using Floyd-Steinberg dithering method. PSNR 8.42 dB.
Diffusion over 50 iterations at center view.
Denoised, global PSNR 27.27 dB.
Uniform quantization
Light field uniformely quantized on 9 bits. PSNR 27.50 dB.
Diffusion over 10 iterations at center view.
Denoised, global PSNR 30.45 dB.
Uniform quantization
Light field uniformely quantized on 12 bits. PSNR 34.17 dB.
Diffusion over 5 iterations at center view.
Denoised, global PSNR 36.62 dB.
Poisson noise
Poisson noise, standard deviation 0.1. PSNR 15.57 dB.
Diffusion over 30 iterations at center view.
Denoised, global PSNR 29.13 dB.
Angular interpolation
Desktop
Sparse light field to interpolate.
Diffusion over 200 iterations on all views.
Diffusion over 200 iterations at center view.
Interpolated light field.
Papillon
Sparse light field to interpolate.
Diffusion over 150 iterations on all views.
Diffusion over 150 iterations at view [3,3].
Interpolated light field.
Spatial interpolation
Bee 2
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 8 iterations.
BenchInParis
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 11 iterations.
Bikes
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 11 iterations.
Buddha
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 4 iterations.
Conehead
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 5 iterations.
Duck
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 9 iterations.
Friends
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 25 iterations.
Fruits
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 7 iterations.
Mini
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 19 iterations.
MonasRoom
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 3 iterations.
Papillon
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 12 iterations.
Rose
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 9 iterations.
Sphynx
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 14 iterations.
Watch
Low resolution light field resized by factor 3 using nearest neighbour interpolation.
Diffusion over 100 iterations, center view.
Interpolated light field after 6 iterations.
Comparison with super-resolution methods
Super-resolution results using diffusion. Magnification factor set to 3.
Super-resolution methods. Magnification factor set to 3. Original light field is spatially smoothed using a Gaussian kernel of standard deviation σ = 1.6.
The color and brightness difference between (c,d) columns and the others is due to different inputs of the methods.
An histogram matching is thus applied for better comparison.
Results of these columns courtesy of [41].
[40] Y. Yoon, H. Jeon, D. Yoo, J. Lee, and I. S. Kweon, “Light-field image super-resolution using convolutional neural network,” IEEE Signal Processing Letters, vol. 24, no. 6, pp. 848–852, June 2017.
[41] R. Farrugia and C. Guillemot, “Light field super-resolution using a lowrank prior and deep convolutional neural networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1–1, 2019.
Inpainting
Leaves
Light field with missing parts.
Diffusion over 440 iterations.
Inpainted light field.
EPI at upper text line. Diffusion over 440 iterations.
EPI at middle text line. Diffusion over 440 iterations.
EPI at bottom text line. Diffusion over 440 iterations.
Succulents
Light field with missing parts.
Diffusion over 400 iterations.
Inpainted light field.
Upper EPI. Diffusion over 400 iterations.
Middle EPI. Diffusion over 400 iterations.
Bottom EPI. Diffusion over 400 iterations.
Regularization
BouquetFlower2
Original light field.
Regularization of light field over 80 iterations, center view.
Regularized light field after 50 iterations.
Regularization over 80 iterations. EPI at upper location of views.
Regularization over 80 iterations. EPI at upper location of views.
Evolution of spatio-angular confidence over regularization iterations.
Evolution of disparity estimate over regularization iterations.
Alpha channel as spatio-angular confidence (above), black background.
Tesseract
Additional videos are proposed to show the behaviour of diffusion along eigenvectors. The tesseract toy example is presented in the paper.
Animations can look choppy because of the automatic adaptation of the integration timestep (Runge-Kutta-Fehlberg method).
Single eigenvector diffusion
Along eigenvector ν1
Along eigenvector ν2
Along eigenvector ν3
Along eigenvector ν4
Multiple eigenvectors diffusion
Along eigenvectors ν1, ν2, ν3 ν4 (isotropic)
Along eigenvector ν1, ν2, ν3
Along eigenvector ν2, ν3
Along eigenvector ν3, ν4
Denoising. Gaussian noise σ = 40/255
Denoising using 2D anisotropic tensor diffusion (100 iterations)
Denoising using 4D anisotropic tensor diffusion (150 iterations)
