Open Source Light Field Toolbox for Super-Resolution

Overview

BasicLFSR

BasicLFSR is an open-source and easy-to-use Light Field (LF) image Super-Ressolution (SR) toolbox based on PyTorch, including a collection of papers on LF image SR and a benchmark to comprehensively evaluate the performance of existing methods. We also provided simple pipelines to train/valid/test state-of-the-art methods to get started quickly, and you can transform your methods into the benchmark.

Note: This repository will be updated on a regular basis, and the pretrained models of existing methods will be open-sourced one after another. So stay tuned!

Methods

Methods Paper Repository
LFSSR Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution. TIP2018 spatialsr/
DeepLightFieldSSR
resLF Residual Networks for Light Field Image Super-Resolution. CVPR2019 shuozh/resLF
HDDRNet High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction. TPAMI2019 monaen/
LightFieldReconstruction
LF-InterNet Spatial-Angular Interaction for Light Field Image Super-Resolution. ECCV2019 YingqianWang/
LF-InterNet
LFSSR-ATO Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. CVPR2020 jingjin25/
LFSSR-ATO
LF-DFnet Light field image super-resolution using deformable convolution. TIP2020 YingqianWang/
LF-DFnet
MEG-Net End-to-End Light Field Spatial Super-Resolution Network using Multiple Epipolar Geometry. TIP2021 shuozh/MEG-Net

Datasets

We used the EPFL, HCInew, HCIold, INRIA and STFgantry datasets for both training and test. Please first download our datasets via Baidu Drive (key:7nzy) or OneDrive, and place the 5 datasets to the folder ./datasets/.

  • After downloading, you should find following structure:

    ├──./datasets/
    │    ├── EPFL
    │    │    ├── training
    │    │    │    ├── Bench_in_Paris.mat
    │    │    │    ├── Billboards.mat
    │    │    │    ├── ...
    │    │    ├── test
    │    │    │    ├── Bikes.mat
    │    │    │    ├── Books__Decoded.mat
    │    │    │    ├── ...
    │    ├── HCI_new
    │    ├── ...
    
  • Run Generate_Data_for_Training.m to generate training data. The generated data will be saved in ./data_for_train/ (SR_5x5_2x, SR_5x5_4x).

  • Run Generate_Data_for_Test.m to generate test data. The generated data will be saved in ./data_for_test/ (SR_5x5_2x, SR_5x5_4x).

Benchmark

We benchmark several methods on above datasets, and PSNR and SSIM metrics are used for quantitative evaluation.

PSNR and SSIM values achieved by different methods for 2xSR:

Method Scale #Params. EPFL HCInew HCIold INRIA STFgantry Average
Bilinear x2 -- 28.479949/0.918006 30.717944/0.919248 36.243278/0.970928 30.133901/0.945545 29.577468/0.931030 31.030508/0.936951
Bicubic x2 -- 29.739509/0.937581 31.887011/0.935637 37.685776/0.978536 31.331483/0.957731 31.062631/0.949769 32.341282/0.951851
VDSR x2
EDSR x2 33.088922/0.962924 34.828374/0.959156 41.013989/0.987400 34.984982/0.976397 36.295865/0.981809
RCSN x2
resLF x2
LFSSR x2 33.670594/0.974351 36.801555/0.974910 43.811050/0.993773 35.279443/0.983202 37.943969/0.989818
LF-ATO x2 34.271635/0.975711 37.243620/0.976684 44.205264/0.994202 36.169943/0.984241 39.636445/0.992862
LF-InterNet x2
LF-DFnet x2
MEG-Net x2
LFT x2

PSNR and SSIM values achieved by different methods for 4xSR:

Method Scale #Params. EPFL HCInew HCIold INRIA STFgantry Average
Bilinear x4 -- 24.567490/0.815793 27.084949/0.839677 31.688225/0.925630 26.226265/0.875682 25.203262/0.826105 26.954038/0.856577
Bicubic x4 -- 25.264206/0.832389 27.714905/0.851661 32.576315/0.934428 26.951718/0.886740 26.087451/0.845230 27.718919/0.870090
VDSR x4
EDSR x4
RCSN x4
resLF x4
LFSSR x4
LF-ATO x4
LF-InterNet x4
LF-DFnet x4
MEG-Net x4
LFT x4

Train

  • Run train.py to perform network training. Example for training [model_name] on 5x5 angular resolution for 2x/4x SR:
    $ python train.py --model_name [model_name] --angRes 5 --scale_factor 2 --batch_size 8
    $ python train.py --model_name [model_name] --angRes 5 --scale_factor 4 --batch_size 4
    
  • Checkpoints and Logs will be saved to ./log/, and the ./log/ has following structure:
    ├──./log/
    │    ├── SR_5x5_2x
    │    │    ├── [dataset_name]
    │    │         ├── [model_name]
    │    │         │    ├── [model_name]_log.txt
    │    │         │    ├── checkpoints
    │    │         │    │    ├── [model_name]_5x5_2x_epoch_01_model.pth
    │    │         │    │    ├── [model_name]_5x5_2x_epoch_02_model.pth
    │    │         │    │    ├── ...
    │    │         │    ├── results
    │    │         │    │    ├── VAL_epoch_01
    │    │         │    │    ├── VAL_epoch_02
    │    │         │    │    ├── ...
    │    │         ├── [other_model_name]
    │    │         ├── ...
    │    ├── SR_5x5_4x
    

Test

  • Run test.py to perform network inference. Example for test [model_name] on 5x5 angular resolution for 2x/4xSR:

    $ python test.py --model_name [model_name] --angRes 5 --scale_factor 2  
    $ python test.py --model_name [model_name] --angRes 5 --scale_factor 4 
    
  • The PSNR and SSIM values of each dataset will be saved to ./log/, and the ./log/ is following structure:

    ├──./log/
    │    ├── SR_5x5_2x
    │    │    ├── [dataset_name]
    │    │        ├── [model_name]
    │    │        │    ├── [model_name]_log.txt
    │    │        │    ├── checkpoints
    │    │        │    │   ├── ...
    │    │        │    ├── results
    │    │        │    │    ├── Test
    │    │        │    │    │    ├── evaluation.xls
    │    │        │    │    │    ├── [dataset_1_name]
    │    │        │    │    │    │    ├── [scene_1_name]
    │    │        │    │    │    │    │    ├── [scene_1_name]_CenterView.bmp
    │    │        │    │    │    │    │    ├── [scene_1_name]_SAI.bmp
    │    │        │    │    │    │    │    ├── views
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_0.bmp
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_1.bmp
    │    │        │    │    │    │    │    │    ├── ...
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_4_4.bmp
    │    │        │    │    │    │    ├── [scene_2_name]
    │    │        │    │    │    │    ├── ...
    │    │        │    │    │    ├── [dataset_2_name]
    │    │        │    │    │    ├── ...
    │    │        │    │    ├── VAL_epoch_01
    │    │        │    │    ├── ...
    │    │        ├── [other_model_name]
    │    │        ├── ...
    │    ├── SR_5x5_4x
    

Recources

We provide some original super-resolved images and useful resources to facilitate researchers to reproduce the above results.

Other Recources

Contact

Any question regarding this work can be addressed to [email protected].

Owner
Squidward
Squidward
A multi-mode modulator for multi-domain few-shot classification (ICCV)

A multi-mode modulator for multi-domain few-shot classification (ICCV)

Yanbin Liu 8 Apr 28, 2022
This is the code for CVPR 2021 oral paper: Jigsaw Clustering for Unsupervised Visual Representation Learning

JigsawClustering Jigsaw Clustering for Unsupervised Visual Representation Learning Pengguang Chen, Shu Liu, Jiaya Jia Introduction This project provid

DV Lab 73 Sep 18, 2022
PyTorch implementation for View-Guided Point Cloud Completion

PyTorch implementation for View-Guided Point Cloud Completion

22 Jan 04, 2023
GrailQA: Strongly Generalizable Question Answering

GrailQA is a new large-scale, high-quality KBQA dataset with 64,331 questions annotated with both answers and corresponding logical forms in different syntax (i.e., SPARQL, S-expression, etc.). It ca

OSU DKI Lab 76 Dec 21, 2022
Official implementation of CATs: Cost Aggregation Transformers for Visual Correspondence NeurIPS'21

CATs: Cost Aggregation Transformers for Visual Correspondence NeurIPS'21 For more information, check out the paper on [arXiv]. Training with different

Sunghwan Hong 120 Jan 04, 2023
Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

Robust Video Matting in PyTorch, TensorFlow, TensorFlow.js, ONNX, CoreML!

Peter Lin 6.5k Jan 04, 2023
This repo generates the training data and the model for Morpheus-Deblend

Morpheus-Deblend This repo generates the training data and the model for Morpheus-Deblend. This is the active development repo for the project and as

Ryan Hausen 2 Apr 18, 2022
Self Governing Neural Networks (SGNN): the Projection Layer

Self Governing Neural Networks (SGNN): the Projection Layer A SGNN's word projections preprocessing pipeline in scikit-learn In this notebook, we'll u

Guillaume Chevalier 22 Nov 06, 2022
StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion

StarGANv2-VC: A Diverse, Unsupervised, Non-parallel Framework for Natural-Sounding Voice Conversion Yinghao Aaron Li, Ali Zare, Nima Mesgarani We pres

Aaron (Yinghao) Li 282 Jan 01, 2023
Data augmentation for NLP, accepted at EMNLP 2021 Findings

AEDA: An Easier Data Augmentation Technique for Text Classification This is the code for the EMNLP 2021 paper AEDA: An Easier Data Augmentation Techni

Akbar Karimi 81 Dec 09, 2022
Code for the TASLP paper "PSLA: Improving Audio Tagging With Pretraining, Sampling, Labeling, and Aggregation".

PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation Introduction Getting Started FSD50K Recipe AudioSet Recipe Label E

Yuan Gong 84 Dec 27, 2022
Creating Multi Task Models With Keras

Creating Multi Task Models With Keras About The Project! I used the keras and Tensorflow Library, To build a Deep Learning Neural Network to Creating

Srajan Chourasia 4 Nov 28, 2022
Generate vibrant and detailed images using only text.

CLIP Guided Diffusion From RiversHaveWings. Generate vibrant and detailed images using only text. See captions and more generations in the Gallery See

Clay M. 401 Dec 28, 2022
PyTorch implementation of Memory-based semantic segmentation for off-road unstructured natural environments.

MemSeg: Memory-based semantic segmentation for off-road unstructured natural environments Introduction This repository is a PyTorch implementation of

11 Nov 28, 2022
PaddleViT: State-of-the-art Visual Transformer and MLP Models for PaddlePaddle 2.0+

PaddlePaddle Vision Transformers State-of-the-art Visual Transformer and MLP Models for PaddlePaddle 🤖 PaddlePaddle Visual Transformers (PaddleViT or

1k Dec 28, 2022
Clean Machine Learning, a Coding Kata

Kata: Clean Machine Learning From Dirty Code First, open the Kata in Google Colab (or else download it) You can clone this project and launch jupyter-

Neuraxio 13 Nov 03, 2022
Sound Source Localization for AI Grand Challenge 2021

Sound-Source-Localization Sound Source Localization study for AI Grand Challenge 2021 (sponsored by NC Soft Vision Lab) Preparation 1. Place the data-

sanghoon 19 Mar 29, 2022
Machine learning library for fast and efficient Gaussian mixture models

This repository contains code which implements the Stochastic Gaussian Mixture Model (S-GMM) for event-based datasets Dependencies CMake Premake4 Blaz

Omar Oubari 1 Dec 19, 2022
Neural network-based build time estimation for additive manufacturing

Neural network-based build time estimation for additive manufacturing Oh, Y., Sharp, M., Sprock, T., & Kwon, S. (2021). Neural network-based build tim

Yosep 1 Nov 15, 2021
CTF Challenge for CSAW Finals 2021

Terminal Velocity Misc CTF Challenge for CSAW Finals 2021 This is a challenge I've had in mind for almost 15 years and never got around to building un

Jordan 6 Jul 30, 2022