Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

Download MPI sintel dataset from here

2. GMA optical flow estimator

To obtain optical flow estimations for pretraining, we are using GMA from here. Note that it dose not have to do with our identity.

3. Training

Training neural residual flow fields (NRFF)

# frame 0 - 6
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 0 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start0_jq98_hf96
# frame 7 - 13
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 7 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start7_jq98_hf96
# frame 14 - 20
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 14 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start14_jq98_hf96
# frame 21 - 27
python train_video_flow_midkey.py --use-estimator --lr 0.0005 --training-step 30000 --data-dir {sintel dataset training directory} --video-name alley_1 --start-frame 21 --num-frames 7 --jpeg-quality 98 --hidden-features 96 --use-estimator --tag start21_jq98_hf96

Training baseline (SIREN)

python train_video.py --data-dir {sintel dataset training directory} --video-name alley_1 --hidden-features 256 --num-frames 28 --lr 0.001 --training-step 30000 --tag baseline_siren_hf256

4. Examples

alley_2.mp4

HoneyBee.mp4

Eff video representation - Efficient video representation through neural fields

Related tags

Overview

Neural Residual Flow Fields for Efficient Video Representations

1. Download MPI sintel dataset

2. GMA optical flow estimator

3. Training

4. Examples

Owner

A multi-functional library for full-stack Deep Learning. Simplifies Model Building, API development, and Model Deployment.

AirPose: Multi-View Fusion Network for Aerial 3D Human Pose and Shape Estimation

Official pytorch implementation of Rainbow Memory (CVPR 2021)

PyTorch implementation of SmoothGrad: removing noise by adding noise.

Code for the CIKM 2019 paper "DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting".

Swin-Transformer is basically a hierarchical Transformer whose representation is computed with shifted windows.

[CVPR'21] MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation

Deep Video Matting via Spatio-Temporal Alignment and Aggregation [CVPR2021]

Systemic Evolutionary Chemical Space Exploration for Drug Discovery

A CROSS-MODAL FUSION NETWORK BASED ON SELF-ATTENTION AND RESIDUAL STRUCTURE FOR MULTIMODAL EMOTION RECOGNITION

Hummingbird compiles trained ML models into tensor computation for faster inference.

Train an RL agent to execute natural language instructions in a 3D Environment (PyTorch)

An official source code for paper Deep Graph Clustering via Dual Correlation Reduction, accepted by AAAI 2022

Efficient Householder transformation in PyTorch

InferPy: Deep Probabilistic Modeling with Tensorflow Made Easy

Definition of a business problem according to Wilson Lower Bound Score and Time Based Average Rating

Machine learning notebooks in different subjects optimized to run in google collaboratory

Speech Emotion Recognition with Fusion of Acoustic- and Linguistic-Feature-Based Decisions

PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

This reposityory contains the PyTorch implementation of our paper "Generative Dynamic Patch Attack".