Open Source Light Field Toolbox for Super-Resolution

Overview

BasicLFSR

BasicLFSR is an open-source and easy-to-use Light Field (LF) image Super-Ressolution (SR) toolbox based on PyTorch, including a collection of papers on LF image SR and a benchmark to comprehensively evaluate the performance of existing methods. We also provided simple pipelines to train/valid/test state-of-the-art methods to get started quickly, and you can transform your methods into the benchmark.

Note: This repository will be updated on a regular basis, and the pretrained models of existing methods will be open-sourced one after another. So stay tuned!

Methods

Methods Paper Repository
LFSSR Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution. TIP2018 spatialsr/
DeepLightFieldSSR
resLF Residual Networks for Light Field Image Super-Resolution. CVPR2019 shuozh/resLF
HDDRNet High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction. TPAMI2019 monaen/
LightFieldReconstruction
LF-InterNet Spatial-Angular Interaction for Light Field Image Super-Resolution. ECCV2019 YingqianWang/
LF-InterNet
LFSSR-ATO Light field spatial super-resolution via deep combinatorial geometry embedding and structural consistency regularization. CVPR2020 jingjin25/
LFSSR-ATO
LF-DFnet Light field image super-resolution using deformable convolution. TIP2020 YingqianWang/
LF-DFnet
MEG-Net End-to-End Light Field Spatial Super-Resolution Network using Multiple Epipolar Geometry. TIP2021 shuozh/MEG-Net

Datasets

We used the EPFL, HCInew, HCIold, INRIA and STFgantry datasets for both training and test. Please first download our datasets via Baidu Drive (key:7nzy) or OneDrive, and place the 5 datasets to the folder ./datasets/.

  • After downloading, you should find following structure:

    ├──./datasets/
    │    ├── EPFL
    │    │    ├── training
    │    │    │    ├── Bench_in_Paris.mat
    │    │    │    ├── Billboards.mat
    │    │    │    ├── ...
    │    │    ├── test
    │    │    │    ├── Bikes.mat
    │    │    │    ├── Books__Decoded.mat
    │    │    │    ├── ...
    │    ├── HCI_new
    │    ├── ...
    
  • Run Generate_Data_for_Training.m to generate training data. The generated data will be saved in ./data_for_train/ (SR_5x5_2x, SR_5x5_4x).

  • Run Generate_Data_for_Test.m to generate test data. The generated data will be saved in ./data_for_test/ (SR_5x5_2x, SR_5x5_4x).

Benchmark

We benchmark several methods on above datasets, and PSNR and SSIM metrics are used for quantitative evaluation.

PSNR and SSIM values achieved by different methods for 2xSR:

Method Scale #Params. EPFL HCInew HCIold INRIA STFgantry Average
Bilinear x2 -- 28.479949/0.918006 30.717944/0.919248 36.243278/0.970928 30.133901/0.945545 29.577468/0.931030 31.030508/0.936951
Bicubic x2 -- 29.739509/0.937581 31.887011/0.935637 37.685776/0.978536 31.331483/0.957731 31.062631/0.949769 32.341282/0.951851
VDSR x2
EDSR x2 33.088922/0.962924 34.828374/0.959156 41.013989/0.987400 34.984982/0.976397 36.295865/0.981809
RCSN x2
resLF x2
LFSSR x2 33.670594/0.974351 36.801555/0.974910 43.811050/0.993773 35.279443/0.983202 37.943969/0.989818
LF-ATO x2 34.271635/0.975711 37.243620/0.976684 44.205264/0.994202 36.169943/0.984241 39.636445/0.992862
LF-InterNet x2
LF-DFnet x2
MEG-Net x2
LFT x2

PSNR and SSIM values achieved by different methods for 4xSR:

Method Scale #Params. EPFL HCInew HCIold INRIA STFgantry Average
Bilinear x4 -- 24.567490/0.815793 27.084949/0.839677 31.688225/0.925630 26.226265/0.875682 25.203262/0.826105 26.954038/0.856577
Bicubic x4 -- 25.264206/0.832389 27.714905/0.851661 32.576315/0.934428 26.951718/0.886740 26.087451/0.845230 27.718919/0.870090
VDSR x4
EDSR x4
RCSN x4
resLF x4
LFSSR x4
LF-ATO x4
LF-InterNet x4
LF-DFnet x4
MEG-Net x4
LFT x4

Train

  • Run train.py to perform network training. Example for training [model_name] on 5x5 angular resolution for 2x/4x SR:
    $ python train.py --model_name [model_name] --angRes 5 --scale_factor 2 --batch_size 8
    $ python train.py --model_name [model_name] --angRes 5 --scale_factor 4 --batch_size 4
    
  • Checkpoints and Logs will be saved to ./log/, and the ./log/ has following structure:
    ├──./log/
    │    ├── SR_5x5_2x
    │    │    ├── [dataset_name]
    │    │         ├── [model_name]
    │    │         │    ├── [model_name]_log.txt
    │    │         │    ├── checkpoints
    │    │         │    │    ├── [model_name]_5x5_2x_epoch_01_model.pth
    │    │         │    │    ├── [model_name]_5x5_2x_epoch_02_model.pth
    │    │         │    │    ├── ...
    │    │         │    ├── results
    │    │         │    │    ├── VAL_epoch_01
    │    │         │    │    ├── VAL_epoch_02
    │    │         │    │    ├── ...
    │    │         ├── [other_model_name]
    │    │         ├── ...
    │    ├── SR_5x5_4x
    

Test

  • Run test.py to perform network inference. Example for test [model_name] on 5x5 angular resolution for 2x/4xSR:

    $ python test.py --model_name [model_name] --angRes 5 --scale_factor 2  
    $ python test.py --model_name [model_name] --angRes 5 --scale_factor 4 
    
  • The PSNR and SSIM values of each dataset will be saved to ./log/, and the ./log/ is following structure:

    ├──./log/
    │    ├── SR_5x5_2x
    │    │    ├── [dataset_name]
    │    │        ├── [model_name]
    │    │        │    ├── [model_name]_log.txt
    │    │        │    ├── checkpoints
    │    │        │    │   ├── ...
    │    │        │    ├── results
    │    │        │    │    ├── Test
    │    │        │    │    │    ├── evaluation.xls
    │    │        │    │    │    ├── [dataset_1_name]
    │    │        │    │    │    │    ├── [scene_1_name]
    │    │        │    │    │    │    │    ├── [scene_1_name]_CenterView.bmp
    │    │        │    │    │    │    │    ├── [scene_1_name]_SAI.bmp
    │    │        │    │    │    │    │    ├── views
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_0.bmp
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_0_1.bmp
    │    │        │    │    │    │    │    │    ├── ...
    │    │        │    │    │    │    │    │    ├── [scene_1_name]_4_4.bmp
    │    │        │    │    │    │    ├── [scene_2_name]
    │    │        │    │    │    │    ├── ...
    │    │        │    │    │    ├── [dataset_2_name]
    │    │        │    │    │    ├── ...
    │    │        │    │    ├── VAL_epoch_01
    │    │        │    │    ├── ...
    │    │        ├── [other_model_name]
    │    │        ├── ...
    │    ├── SR_5x5_4x
    

Recources

We provide some original super-resolved images and useful resources to facilitate researchers to reproduce the above results.

Other Recources

Contact

Any question regarding this work can be addressed to [email protected].

Owner
Squidward
Squidward
An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise Weight Sharing) by Sensetime Research.

An integration of several popular automatic augmentation methods, including OHL (Online Hyper-Parameter Learning for Auto-Augmentation Strategy) and AWS (Improving Auto Augment via Augmentation Wise

45 Dec 08, 2022
Code for "Training Neural Networks with Fixed Sparse Masks" (NeurIPS 2021).

Code for "Training Neural Networks with Fixed Sparse Masks" (NeurIPS 2021).

Varun Nair 37 Dec 30, 2022
This repository contains the source code of an efficient 1D probabilistic model for music time analysis proposed in ICASSP2022 venue.

Jump Reward Inference for 1D Music Rhythmic State Spaces An implementation of the probablistic jump reward inference model for music rhythmic informat

Mojtaba Heydari 25 Dec 16, 2022
Repository for the semantic WMI loss

Installation: pip install -e . Installing DL2: First clone DL2 in a separate directory and install it using the following commands: git clone https:/

Nick Hoernle 4 Sep 15, 2022
Vision Transformer for 3D medical image registration (Pytorch).

ViT-V-Net: Vision Transformer for Volumetric Medical Image Registration keywords: vision transformer, convolutional neural networks, image registratio

Junyu Chen 192 Dec 20, 2022
Black-Box-Tuning - Black-Box Tuning for Language-Model-as-a-Service

Black-Box-Tuning Source code for paper "Black-Box Tuning for Language-Model-as-a

Tianxiang Sun 149 Jan 04, 2023
Cross-view Transformers for real-time Map-view Semantic Segmentation (CVPR 2022 Oral)

Cross View Transformers This repository contains the source code and data for our paper: Cross-view Transformers for real-time Map-view Semantic Segme

Brady Zhou 363 Dec 25, 2022
Official PyTorch implementation of "Adversarial Reciprocal Points Learning for Open Set Recognition"

Adversarial Reciprocal Points Learning for Open Set Recognition Official PyTorch implementation of "Adversarial Reciprocal Points Learning for Open Se

Guangyao Chen 78 Dec 28, 2022
CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes (AAAI2022)

CMUA-Watermark The official code for CMUA-Watermark: A Cross-Model Universal Adversarial Watermark for Combating Deepfakes (AAAI2022) arxiv. It is bas

50 Nov 26, 2022
Tensorflow implementation of Character-Aware Neural Language Models.

Character-Aware Neural Language Models Tensorflow implementation of Character-Aware Neural Language Models. The original code of author can be found h

Taehoon Kim 751 Dec 26, 2022
[CVPR 2021] Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach

Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach This is the repo to host the dataset TextSeg and code for TexRNe

SHI Lab 174 Dec 19, 2022
Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning

structshot Code and data for paper "Simple and Effective Few-Shot Named Entity Recognition with Structured Nearest Neighbor Learning", Yi Yang and Arz

ASAPP Research 47 Dec 27, 2022
DAN: Unfolding the Alternating Optimization for Blind Super Resolution

DAN-Basd-on-Openmmlab DAN: Unfolding the Alternating Optimization for Blind Super Resolution We reproduce DAN via mmediting based on open-sourced code

AlexZou 72 Dec 13, 2022
Object DGCNN and DETR3D, Our implementations are built on top of MMdetection3D.

This repo contains the implementations of Object DGCNN (https://arxiv.org/abs/2110.06923) and DETR3D (https://arxiv.org/abs/2110.06922). Our implementations are built on top of MMdetection3D.

Wang, Yue 539 Jan 07, 2023
prior-based-losses-for-medical-image-segmentation

Repository for papers: Benchmark: Effect of Prior-based Losses on Segmentation Performance: A Benchmark Midl: A Surprisingly Effective Perimeter-based

Rosana EL JURDI 9 Sep 07, 2022
An evaluation toolkit for voice conversion models.

Voice-conversion-evaluation An evaluation toolkit for voice conversion models. Sample test pair Generate the metadata for evaluating models. The direc

30 Aug 29, 2022
Read number plates with https://platerecognizer.com/

HASS-plate-recognizer Read vehicle license plates with https://platerecognizer.com/ which offers free processing of 2500 images per month. You will ne

Robin 69 Dec 30, 2022
Styleformer - Official Pytorch Implementation

Styleformer -- Official PyTorch implementation Styleformer: Transformer based Generative Adversarial Networks with Style Vector(https://arxiv.org/abs/

Jeeseung Park 159 Dec 12, 2022
GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data

GeneGAN: Learning Object Transfiguration and Attribute Subspace from Unpaired Data By Shuchang Zhou, Taihong Xiao, Yi Yang, Dieqiao Feng, Qinyao He, W

Taihong Xiao 141 Apr 16, 2021
A boosting-based Multiple Instance Learning (MIL) package that includes MIL-Boost and MCIL-Boost

A boosting-based Multiple Instance Learning (MIL) package that includes MIL-Boost and MCIL-Boost

Jun-Yan Zhu 27 Aug 08, 2022