Code for Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid

Related tags

Deep LearningSPN
Overview

SPN: Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid

Code for Fully Context-Aware Image Inpainting with a Learned Semantic Pyramid, submitted to IEEE. Pretrained models have been uploaded.

This project is for our new inpainting method SPN which has been submitted to IEEE under peer review. This work is an extension version of our previous work SPL (IJCAI'21). If you have any questions, feel free to make issues. Thanks for your interests!

Paper on Arxiv. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible

Introduction:

Briefly speaking, in this work, we still focus on the key insight that learning semantic priors from specific pretext tasks can benefit image inpainting, and we further strengthen the modeling of the learned priors in this work from the following aspects:

  1. We exploit multi-scale semantic priors in a feature pyramid manner to achieve consistent understanding of both gloabl and local context. The image generator is also improved to incorporate the prior pyramid.
  2. We extend our prior learned in a probabilistic manner which enables our method to handle probabilistic image inpainting problem.
  3. Besides, more analyses of the learned prior pyramid and the choices of the semantic supervision are provided in our experiment part.

Prerequisites (same with SPL)

  • Python 3.7
  • PyTorch 1.8 (1.6+ may also work)
  • NVIDIA GPU + CUDA cuDNN
  • Inplace_Abn (only needed for training our model, used in ASL_TRresNet model)
  • torchlight (We only use it to record the printed information. You can change it as you want.)

Datasets

We use Places2, CelebA and Paris Street-View datasets for determinstic image inpainting which is same with SPL, and CelebA-HQ dataset is used for probabilistic image inpainting. We also used the irregular mask provided by Liu et al. which can be downloaded from their website. For the detailed processes of these datasets please refer to SPL and our paper.

Getting Strated

Since our approach can be applied for both deterministic and probabilistic image inpainting, so we seperate the codes under these two setups in different files and each file contains corresponding training and testing commonds.

For all setups, the common pre-preparations are list as follows:

  1. Download the pre-trained models and copy them under ./checkpoints directory.

  2. (For training) Make another directory, e.g ./pretrained_ASL, and download the weights of TResNet_L pretrained on OpenImage dataset to this directory.

  3. Install torchlight

cd ./torchlight
python setup.py install
Auditing Black-Box Prediction Models for Data Minimization Compliance

Data-Minimization-Auditor An auditing tool for model-instability based data minimization that is introduced in "Auditing Black-Box Prediction Models f

Bashir Rastegarpanah 2 Mar 24, 2022
World Models with TensorFlow 2

World Models This repo reproduces the original implementation of World Models. This implementation uses TensorFlow 2.2. Docker The easiest way to hand

Zac Wellmer 234 Nov 30, 2022
Official implementation of SynthTIGER (Synthetic Text Image GEneratoR) ICDAR 2021

๐Ÿฏ SynthTIGER: Synthetic Text Image GEneratoR Official implementation of SynthTIGER | Paper | Datasets Moonbin Yim1, Yoonsik Kim1, Han-cheol Cho1, Sun

Clova AI Research 256 Jan 05, 2023
A Context-aware Visual Attention-based training pipeline for Object Detection from a Webpage screenshot!

CoVA: Context-aware Visual Attention for Webpage Information Extraction Abstract Webpage information extraction (WIE) is an important step to create k

Keval Morabia 41 Jan 01, 2023
PyTorch-lightning implementation of the ESFW module proposed in our paper Edge-Selective Feature Weaving for Point Cloud Matching

Edge-Selective Feature Weaving for Point Cloud Matching This repository contains a PyTorch-lightning implementation of the ESFW module proposed in our

5 Feb 14, 2022
An algorithm study of the 6th iOS 10 set of Boost Camp Web Mobile

์•Œ๊ณ ๋ฆฌ์ฆ˜ ์Šคํ„ฐ๋”” ๐Ÿ”ฅ ๋ถ€์ŠคํŠธ์บ ํ”„ ์›น๋ชจ๋ฐ”์ผ 6๊ธฐ iOS 10์กฐ์˜ ์•Œ๊ณ ๋ฆฌ์ฆ˜ ์Šคํ„ฐ๋”” ์ž…๋‹ˆ๋‹ค. ๊ฐœ์ธ์ ์ธ ์‚ฌ์ • ๋“ฑ์œผ๋กœ S034, S055๋งŒ ์ฐธ๊ฐ€ํ•˜์˜€์Šต๋‹ˆ๋‹ค. ์Šคํ„ฐ๋”” ๋ชฉ์  ์ƒ์ง„: ์ฝ”ํ…Œ ํ•ฉ๊ฒฉ + ๋ถ€์บ ๋๋‚˜๊ณ  ์•„์นจ์— ์ผ์–ด๋‚˜๊ธฐ ์œ„ํ•ด ํ•„์š”ํ•œ ์‚ฌ์ดํด ๊ธฐ์™„: ๊พธ์ค€ํ•˜๊ฒŒ ์ž๋ฆฌ์— ์•‰์•„ ๊ณต๋ถ€ํ•˜๊ธฐ +

2 Jan 11, 2022
Tensorflow 2 implementation of our high quality frame interpolation neural network

FILM: Frame Interpolation for Large Scene Motion Project | Paper | YouTube | Benchmark Scores Tensorflow 2 implementation of our high quality frame in

Google Research 1.6k Dec 28, 2022
Using VapourSynth with super resolution models and speeding them up with TensorRT.

VSGAN-tensorrt-docker Using image super resolution models with vapoursynth and speeding them up with TensorRT. Using NVIDIA/Torch-TensorRT combined wi

111 Jan 05, 2023
Multiple Object Tracking with Yolov5!

Tracking with yolov5 This implementation is for who need to tracking multi-object only with detector. You can easily track mult-object with your well

9 Nov 08, 2022
tree-math: mathematical operations for JAX pytrees

tree-math: mathematical operations for JAX pytrees tree-math makes it easy to implement numerical algorithms that work on JAX pytrees, such as iterati

Google 137 Dec 28, 2022
A short code in python, Enchpyter, is able to encrypt and decrypt words as you determine, of course

Enchpyter Enchpyter is a program do encrypt and decrypt any word you want (just letters). You enter how many letters jumps and write the word, so, the

Joรฃo Assalim 2 Oct 10, 2022
A flexible and extensible framework for gait recognition.

A flexible and extensible framework for gait recognition. You can focus on designing your own models and comparing with state-of-the-arts easily with the help of OpenGait.

Shiqi Yu 335 Dec 22, 2022
ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction

ViSER: Video-Specific Surface Embeddings for Articulated 3D Shape Reconstruction. NeurIPS 2021.

Gengshan Yang 59 Nov 25, 2022
Compressed Video Action Recognition

Compressed Video Action Recognition Chao-Yuan Wu, Manzil Zaheer, Hexiang Hu, R. Manmatha, Alexander J. Smola, Philipp Krรคhenbรผhl. In CVPR, 2018. [Proj

Chao-Yuan Wu 479 Dec 26, 2022
Official code base for the poster "On the use of Cortical Magnification and Saccades as Biological Proxies for Data Augmentation" published in NeurIPS 2021 Workshop (SVRHM)

Self-Supervised Learning (SimCLR) with Biological Plausible Image Augmentations Official code base for the poster "On the use of Cortical Magnificatio

Binxu 8 Aug 17, 2022
What can linearized neural networks actually say about generalization?

What can linearized neural networks actually say about generalization? This is the source code to reproduce the experiments of the NeurIPS 2021 paper

gortizji 11 Dec 09, 2022
SMPLpix: Neural Avatars from 3D Human Models

subject0_validation_poses.mp4 Left: SMPL-X human mesh registered with SMPLify-X, middle: SMPLpix render, right: ground truth video. SMPLpix: Neural Av

Sergey Prokudin 292 Dec 30, 2022
A collection of SOTA Image Classification Models in PyTorch

A collection of SOTA Image Classification Models in PyTorch

sithu3 85 Dec 30, 2022
A fuzzing framework for SMT solvers

yinyang A fuzzing framework for SMT solvers. Given a set of seed SMT formulas, yinyang generates mutant formulas to stress-test SMT solvers. yinyang c

Project Yin-Yang for SMT Solver Testing 145 Jan 04, 2023
Code for the upcoming CVPR 2021 paper

The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel J. Brostow and Michael

Niantic Labs 496 Dec 30, 2022