SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Last update: Aug 30, 2022

Related tags

Deep Learning SAAVN

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

These codes are based on the SoundSpaces code base.

Usage

This repo supports AudioGoal Task on Replica and Matterport3D datasets.

Below we show the commands for training and evaluating AudioGoal with Depth sensor on Replica, but it applies to Matterport dataset as well.

Training

python main.py --default av_nav --run-type train --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Validation (evaluate each checkpoint and generate a validation curve)

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Test the best validation checkpoint based on validation curve

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Generate demo video with audio

python main.py --default av_nav --run-type eval --exp-config [exp_config_file] --model-dir data/models/replica/av_nav/e0000/audiogoal_depth --tag-config [tag_config_file] TORCH_GPU_ID 0 SIMULATOR_GPU_ID 0

Note: [exp_config_file] is the main parameter configuration file of the experiment, while [tag_config_file] is special parameter configuration file for abalation experiments.

Citation

If you use this model in your research, please cite the following paper:

@inproceedings{YinfengICLR2022saavn,
	title = {Sound Adversarial Audio-Visual Navigation},
	author = {Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, Xiaohong Liu},
	year = {2022},
        booktitle={ICLR},
}

SAAVN - Sound Adversarial Audio-Visual Navigation,ICLR2022 (In PyTorch)

Related tags

Overview

SAAVN

SAAVN Code release for paper "Sound Adversarial Audio-Visual Navigation,ICLR2022" (In PyTorch)

These code are under cleaning! Some of bugs maybe happen, please tell me if you have any trouble.

Thanks

Usage

Citation

Owner

YinfengYu

Like ThreeJS but for Python and based on wgpu

Code release for "Detecting Twenty-thousand Classes using Image-level Supervision".

Official code for "Stereo Waterdrop Removal with Row-wise Dilated Attention (IROS2021)"

A Python package for time series augmentation

Contextualized Perturbation for Textual Adversarial Attack, NAACL 2021

Hysterese plugin with two temperature offset areas

Generative Adversarial Networks(GANs)

A implemetation of the LRCN in mxnet

Implementation of the algorithm shown in the article "Modelo de Predicción de Éxito de Canciones Basado en Descriptores de Audio"

This repository contains the source code for the paper First Order Motion Model for Image Animation

Pytorch implementation of our method for regularizing nerual radiance fields for few-shot neural volume rendering.

"Projelerle Yapay Zeka Ve Bilgisayarlı Görü" Kitabımın projeleri

Tiny Kinetics-400 for test

Codes for the compilation and visualization examples to the HIF vegetation dataset

Tutorial for the PERFECTING FACTORY 5.0 WITH EDGE-POWERED AI workshop

Code for Mining the Benefits of Two-stage and One-stage HOI Detection

A semismooth Newton method for elliptic PDE-constrained optimization

ONNX Runtime: cross-platform, high performance ML inferencing and training accelerator

Pytorch implementation of Supporting Clustering with Contrastive Learning, NAACL 2021

Deep Learning Based EDM Subgenre Classification using Mel-Spectrogram and Tempogram Features"