A-ESRGAN aims to provide better super-resolution images by using multi-scale attention U-net discriminators.

Related tags

Deep LearningA-ESRGAN
Overview

A-ESRGAN: Training Real-World Blind Super-Resolution with Attention-based U-net Discriminators

The authors are hidden for the purpose of double blind in the process of review.

Main idea

Introduce attention U-net into the field of blind real world image super resolution. We aims to provide a super resolution method with sharper result and less distortion.

Sharper:

Less distortion:

Network Architecture

The overall architecture of the A-ESRGAN, where the generator is adopted from ESRGAN:

The architecture of a single attention U-net discriminator:

The attention block is modified from 3D attention U-net's attention gate:

Attention Map

We argue it is the attention map that plays the main role in improving the quality of super resolution images. To support our idea, we visualize how the attention coefficients changes in time and space.

We argue that during the training process the attention will gradually focus on regions where color changes abruptly, i.e. edges. And attention layer in different depth will give us edges of different granularity.

Attention coefficients changes across time.

Attention coefficients changes across space.

Multi Scale

Multi scale discriminator has to learn whether parts of the image is clear enough from different receptive fields. From this perspective, different discriminator can learn complementary knowledge. From the figure below, normal discriminator learn to focus on edges, while down-sampled discriminator learn patch-like patterns such as textures.

Thus, comparing with the single attention u-net discriminator, multi-scale u-net discriminator can generate more realistic and detailed images.

Better Texture:

Test Sets

The datasets for test in our A-ESRGAN model are the standard benchmark datasets Set5, Set14, BSD100, Sun-Hays80, Urban100. Noted that we directly apply 4X super resolution to the original real world images and use NIQE to test the perceptual quality of the result. As shown in the figure below, these 5 datasets have covered a large variety of images.

A combined dataset can be find in DatasetsForSR.zip.

We compare with ESRGAN, RealSR, BSRGAN, RealESRGAN on the above 5 datasets and use NIQE as our metrics. The result can be seen in the table below:

Note a lower NIQE score shows a better perceptual quality.

Quick Use

Inference Script

! We now only provides 4X super resolution now.

Download pre-trained models: A-ESRGAN-Single.pth to the experiments/pretrained_models.

wget https://github.com/aergan/A-ESRGAN/releases/download/v1.0.0/A_ESRGAN_Single.pth

Inference:

python inference_aesrgan.py --model_path=experiments/pretrained_models/A_ESRGAN_Single.pth --input=inputs

Results are in the results folder

NIQE Script

The NIQE Script is used to give the Mean NIQE score of a certain directory of images.

Cacluate NIQE score:

cd NIQE_Script
python niqe.py --path=../results

Visualization Script

The Visualization Script is used to visualize the attention coefficient of each attention layer in the attention based U-net discriminator. It has two scripts. One script discriminator_attention_visual(Single).py is used to visualize how the attention of each layer is updated during the training process on a certain image. Another Script combine.py is used to combine the heat map together with original image.

Generate heat maps:

First download single.zip and unzip to experiments/pretrained_models/single

cd Visualization_Script
python discriminator_attention_visual(Single).py --img_path=../inputs/img_015_SRF_4_HR.png

The heat maps will be contained in Visualization_Script/Visual

If you want to see how the heat map looks when combining with the original image, run:

python combine.py --img_path=../inputs/img_015_SRF_4_HR.png

The combined images will be contained in Visualization_Script/Combined

! Multi-scale discriminator attention map visualization:

Download multi.zip and unzip to experiments/pretrained_models/multi

Run discriminator_attention_visual(Mulit).py similar to discriminator_attention_visual(Single).py.

!See what the multi-scale discriminator output

Run Multi_discriminator_Output.py and you could see the visualization of pixel-wise loss from the discriminators.

! Note we haven't provided a combined script for multi attention map yet.

Model_Zoo

The following models are the generators, used in the A-ESRGAN

The following models are discriminators, which are usually used for fine-tuning.

The following models are the checkpoints of discriminators during A-ESRGAN training process, which are provided for visualization attention.

Training and Finetuning on your own dataset

We follow the same setting as RealESRGAN, and a detailed guide can be found in Training.md.

Acknowledgement

Our implementation of A-ESRGAN is based on the BasicSR and Real-ESRGAN.

You might also like...
The deployment framework aims to provide a simple, lightweight, fast integrated, pipelined deployment framework that ensures reliability, high concurrency and scalability of services.

savior是一个能够进行快速集成算法模块并支持高性能部署的轻量开发框架。能够帮助将团队进行快速想法验证(PoC),避免重复的去github上找模型然后复现模型;能够帮助团队将功能进行流程拆解,很方便的提高分布式执行效率;能够有效减少代码冗余,减少不必要负担。

[CVPR 2022] Official PyTorch Implementation for
[CVPR 2022] Official PyTorch Implementation for "Reference-based Video Super-Resolution Using Multi-Camera Video Triplets"

Reference-based Video Super-Resolution (RefVSR) Official PyTorch Implementation of the CVPR 2022 Paper Project | arXiv | RealMCVSR Dataset This repo c

PyTorch code for our paper
PyTorch code for our paper "Image Super-Resolution with Non-Local Sparse Attention" (CVPR2021).

Image Super-Resolution with Non-Local Sparse Attention This repository is for NLSN introduced in the following paper "Image Super-Resolution with Non-

PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

HAN PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network" This repository is for HAN introduced in the

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022)
A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022)

A Text Attention Network for Spatial Deformation Robust Scene Text Image Super-resolution (CVPR2022) https://arxiv.org/abs/2203.09388 Jianqi Ma, Zheto

PyTorch implementation of a Real-ESRGAN model trained on custom dataset

Real-ESRGAN PyTorch implementation of a Real-ESRGAN model trained on custom dataset. This model shows better results on faces compared to the original

My usage of Real-ESRGAN to upscale anime, some test and results in the test_img folder
My usage of Real-ESRGAN to upscale anime, some test and results in the test_img folder

anime upscaler My usage of Real-ESRGAN to upscale anime, I hope to use this on a proper GPU cuz doing this on CPU is completely shit 😂 , I even tried

Comments
  • About the pre-trained model

    About the pre-trained model

    Hi, is the A-ESRGAN-multi pertained model available?

    the link below seems broken.

    https://github.com/aergan/A-ESRGAN/releases/download/v1.0.0/A_ESRGAN_Multi.pth

    opened by ShiinaMitsuki 1
  • some error

    some error

    /media/xyt/software/anaconda3/envs/basicSR/bin/python /media/xyt/data/github/SR/code/A-ESRGAN/train.py -opt options/train_aesrgan_x4plus.yml --debug 2022-02-09 18:17:12,962 INFO: Dataset [RealESRGANDataset] - DF2K is built. 2022-02-09 18:17:12,962 INFO: Training statistics: Number of train images: 500 Dataset enlarge ratio: 1 Batch size per gpu: 6 World size (gpu number): 1 Require iter number per epoch: 84 Total epochs: 4762; iters: 400000. Traceback (most recent call last): File "/media/xyt/data/github/SR/code/A-ESRGAN/train.py", line 11, in train_pipeline(root_path) File "/media/xyt/software/anaconda3/envs/basicSR/lib/python3.7/site-packages/basicsr/train.py", line 128, in train_pipeline model = build_model(opt) File "/media/xyt/software/anaconda3/envs/basicSR/lib/python3.7/site-packages/basicsr/models/init.py", line 27, in build_model model = MODEL_REGISTRY.get(opt['model_type'])(opt) File "/media/xyt/software/anaconda3/envs/basicSR/lib/python3.7/site-packages/basicsr/utils/registry.py", line 65, in get raise KeyError(f"No object named '{name}' found in '{self._name}' registry!") KeyError: "No object named 'RealESRGANModel' found in 'model' registry!"

    opened by xiayutong 1
High accurate tool for automatic faces detection with landmarks

faces_detanator High accurate tool for automatic faces detection with landmarks. The library is based on public detectors with high accuracy (TinaFace

Ihar 7 May 10, 2022
Hunt down social media accounts by username across social networks

Hunt down social media accounts by username across social networks Installation | Usage | Docker Notes | Contributing Installation # clone the repo $

1 Dec 14, 2021
A minimalist environment for decision-making in autonomous driving

highway-env A collection of environments for autonomous driving and tactical decision-making tasks An episode of one of the environments available in

Edouard Leurent 1.6k Jan 07, 2023
SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking

SPLADE 🍴 + 🥄 = 🔎 This repository contains the weights for four models as well as the code for running inference for our two papers: [v1]: SPLADE: S

NAVER 170 Dec 28, 2022
Official PyTorch Implementation of Rank & Sort Loss [ICCV2021]

Rank & Sort Loss for Object Detection and Instance Segmentation The official implementation of Rank & Sort Loss. Our implementation is based on mmdete

Kemal Oksuz 229 Dec 20, 2022
A Python library for adversarial machine learning focusing on benchmarking adversarial robustness.

ARES This repository contains the code for ARES (Adversarial Robustness Evaluation for Safety), a Python library for adversarial machine learning rese

Tsinghua Machine Learning Group 377 Dec 20, 2022
A weakly-supervised scene graph generation codebase. The implementation of our CVPR2021 paper ``Linguistic Structures as Weak Supervision for Visual Scene Graph Generation''

README.md shall be finished soon. WSSGG 0 Overview 1 Installation 1.1 Faster-RCNN 1.2 Language Parser 1.3 GloVe Embeddings 2 Settings 2.1 VG-GT-Graph

Keren Ye 35 Nov 20, 2022
Exploring Simple Siamese Representation Learning

G-SimSiam A PyTorch implementation which refers to repo for the paper Exploring Simple Siamese Representation Learning by Xinlei Chen & Kaiming He Add

zhuyun 1 Dec 19, 2021
K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce (EMNLP Founding 2021)

Introduction K-PLUG: Knowledge-injected Pre-trained Language Model for Natural Language Understanding and Generation in E-Commerce. Installation PyTor

Xu Song 21 Nov 16, 2022
Metric learning algorithms in Python

metric-learn: Metric Learning in Python metric-learn contains efficient Python implementations of several popular supervised and weakly-supervised met

1.3k Jan 02, 2023
Detection of drones using their thermal signatures from thermal camera through YOLO-V3 based CNN with modifications to encapsulate drone motion

Drone Detection using Thermal Signature This repository highlights the work for night-time drone detection using a using an Optris PI Lightweight ther

Chong Yu Quan 6 Dec 31, 2022
PyTorch implementations of the paper: "Learning Independent Instance Maps for Crowd Localization"

IIM - Crowd Localization This repo is the official implementation of paper: Learning Independent Instance Maps for Crowd Localization. The code is dev

tao han 91 Nov 10, 2022
本步态识别系统主要基于GaitSet模型进行实现

本步态识别系统主要基于GaitSet模型进行实现。在尝试部署本系统之前,建立理解GaitSet模型的网络结构、训练和推理方法。 系统的实现效果如视频所示: 演示视频 由于模型较大,部分模型文件存储在百度云盘。 链接提取码:33mb 具体部署过程 1.下载代码 2.安装requirements.txt

16 Oct 22, 2022
Concept drift monitoring for HA model servers.

{Fast, Correct, Simple} - pick three Easily compare training and production ML data & model distributions Goals Boxkite is an instrumentation library

98 Dec 15, 2022
(CVPR 2022) A minimalistic mapless end-to-end stack for joint perception, prediction, planning and control for self driving.

LAV Learning from All Vehicles Dian Chen, Philipp Krähenbühl CVPR 2022 (also arXiV 2203.11934) This repo contains code for paper Learning from all veh

Dian Chen 300 Dec 15, 2022
Learning Compatible Embeddings, ICCV 2021

LCE Learning Compatible Embeddings, ICCV 2021 by Qiang Meng, Chixiang Zhang, Xiaoqiang Xu and Feng Zhou Paper: Arxiv We cannot release source codes pu

Qiang Meng 25 Dec 17, 2022
Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis Implementation This project attempted to implement the paper Putting NeRF on a

254 Dec 27, 2022
Ludwig Benchmarking Toolkit

Ludwig Benchmarking Toolkit The Ludwig Benchmarking Toolkit is a personalized benchmarking toolkit for running end-to-end benchmark studies across an

HazyResearch 17 Nov 18, 2022
HeartRate detector with ArduinoandPython - Use Arduino and Python create a heartrate detector.

Syllabus of Contents Syllabus of Contents Introduction Of Project Features Develop With Python code introduction Installation License Developer Contac

1 Jan 05, 2022
FCN (Fully Convolutional Network) is deep fully convolutional neural network architecture for semantic pixel-wise segmentation

FCN_via_Keras FCN FCN (Fully Convolutional Network) is deep fully convolutional neural network architecture for semantic pixel-wise segmentation. This

Kento Watanabe 48 Aug 30, 2022