Using image super resolution models with vapoursynth and speeding them up with TensorRT

Last update: Aug 23, 2022

Overview

vs-RealEsrganAnime-tensorrt-docker

Using image super resolution models with vapoursynth and speeding them up with TensorRT. Also a docker image since TensorRT is hard to install. Testing showed ~70% more speed on my 1070ti compared to normal PyTorch in 480p. Using the 2x model with TensorRT and 848x480 input was 0.517x realtime speed for 24fps video.

I was forced to use onnx/onnx-tensorrt instead of NVIDIA/Torch-TensorRT because of convertion errors with PyTorch, but the only disadvantage should be that a new onnx model needs to be created for a different input resolution, which takes a bit time.

This repo uses a lot of code from HolyWu/vs-realesrgan and xinntao/Real-ESRGAN. The models are from here.

Usage:

# install docker, command for arch
yay -S docker nvidia-docker nvidia-container-toolkit
# Put the dockerfile in a directory and run that inside that directory
docker build -t realsr_tensorrt:latest .
# run with a mounted folder
docker run --privileged --gpus all -it --rm -v /home/Desktop/tensorrt:/workspace/tensorrt realsr_tensorrt:latest
# you can use it in various ways, ffmpeg example
vspipe --y4m inference.py - | ffmpeg -i pipe: example.mkv

If docker does not want to start, try this before you use docker:

# fixing docker errors
systemctl start docker
sudo chmod 666 /var/run/docker.sock

If you don't want to use docker, vapoursynth install commands are here and a TensorRT example is here.

Set the input video path in inference.py and access videos with the mounted folder. You can also choose between the 4x and 2x model.

It is also possible to directly pipe the video into mpv. Change the mounted folder path to your own videofolder and use the mpv dockerfile instead. Only tested in Manjaro.

yay -S pulseaudio

# i am not sure if it is needed, but go into pulseaudio settings and check "make pulseaudio network audio devices discoverable in the local network" and reboot

# start docker
docker run --rm -i -t \
    --network host \
    -e DISPLAY \
    -v /home/Schreibtisch/test/:/home/mpv/media \
    --ipc=host \
    --privileged \
    --gpus all \
    -e PULSE_COOKIE=/run/pulse/cookie \
    -v ~/.config/pulse/cookie:/run/pulse/cookie \
    -e PULSE_SERVER=unix:${XDG_RUNTIME_DIR}/pulse/native \
    -v ${XDG_RUNTIME_DIR}/pulse/native:${XDG_RUNTIME_DIR}/pulse/native \
    realsr_tensorrt:latest
    
# run mpv
vspipe --y4m inference.py - | mpv -

Using image super resolution models with vapoursynth and speeding them up with TensorRT

Related tags

Overview

vs-RealEsrganAnime-tensorrt-docker

Owner

A PyTorch implementation of NeRF (Neural Radiance Fields) that reproduces the results.

Embeds a story into a music playlist by sorting the playlist so that the order of the music follows a narrative arc.

An MQA (Studio, originalSampleRate) identifier for lossless flac files written in Python.

A basic duplicate image detection service using perceptual image hash functions and nearest neighbor search, implemented using faiss, fastapi, and imagehash

Gym-TORCS is the reinforcement learning (RL) environment in TORCS domain with OpenAI-gym-like interface.

Code release for NeRF (Neural Radiance Fields)

An experiment on the performance of homemade Q-learning AIs in Agar.io depending on their state representation and available actions

Simulation code and tutorial for BBHnet training data

This repo provides the official code for TransBTS: Multimodal Brain Tumor Segmentation Using Transformer (https://arxiv.org/pdf/2103.04430.pdf).

Predicting lncRNA–protein interactions based on graph autoencoders and collaborative training

A framework for annotating 3D meshes using the predictions of a 2D semantic segmentation model.

A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

BABEL: Bodies, Action and Behavior with English Labels [CVPR 2021]

TorchMD-Net provides state-of-the-art graph neural networks and equivariant transformer neural networks potentials for learning molecular potentials

This is Official implementation for "Pose-guided Feature Disentangling for Occluded Person Re-Identification Based on Transformer" in AAAI2022

Edge-oriented Convolution Block for Real-time Super Resolution on Mobile Devices, ACM Multimedia 2021

ATAC: Adversarially Trained Actor Critic

Official pytorch implementation of the paper: "SinGAN: Learning a Generative Model from a Single Natural Image"

TensorFlow Similarity is a python package focused on making similarity learning quick and easy.

The official homepage of the (outdated) COCO-Stuff 10K dataset.