RARA: Zero-shot Sim2Real Visual Navigation with Following Foreground Cues

Related tags

Deep Learningfgbg
Overview

RARA: Zero-shot Sim2Real Visual Navigation with Following Foreground Cues

FGBG (foreground-background) pytorch package for defining and training models. For a demo, please watch: https://youtu.be/nnnhLXBl8J8

Install Imitation-learning codebase for data collection and evaluation in simulation

See instruction here: https://github.com/kkelchte/imitation-learning-codebase. If the installation went fluently you should be able to create a dataset from within your sourced singularity environment:

python3.8 src/sim/ros/src/data_collection_fg_bg.py

This will create a json and hdf5 file of a number of flewn trajectories in the line world.

Install FGBG in a conda environment

conda create --yes --name venv python=3.6
conda activate venv
conda install --yes --file requirements-conda
conda install --yes pytorch torchvision cudatoolkit=11.0 -c pytorch 
python -m pip install -r requirements-pip

Train your models for extracting the foreground and background

Pretrain a model with bg augmentation from MITplaces stored in data/datasets/places

python run.py --config_file configs/deep_supervision_triplet.json --texture_directory data/datasets/places --target line --output_dir data/mymodel

Finetune the final layers for waypoint prediction with

python run.py --config_file configs/deep_supervision_triplet.json --texture_directory data/datasets/places --target line --encoder_ckpt_dir data/mymodel --output_dir data/mymodel/waypoints --task waypoints

Evaluate neural network on both simulated and real bebop drone

From within the singularity environment, you can run the following files. Make sure you adjust each file to the correct task (waypoints) and the correct checkpoint directory (data/mymodel/waypoints).

For evaluation in simulation:

python3.8 src/sim/ros/src/online_evaluation_fgbg.py

For evaluation on the real bebop drone, make sure you connect to the wifi of the drone before launching:

python3.8 src/sim/ros/src/online_evaluation_fgbg_real.py
rosrun imitation-learning-ros-package fgbg_actor.py

If everything goes according to plan, a console view should pop up with the life mask predictions as well as the waypoints. In order to start the autonomous flight, you can either use the keyboard or the joystick interface to publish an emtpy message on the '/go' topic. You can over take the experiments with publishing an empty message on the '/overtake' topic.

Troubleshoot

Just email me on kkelchtermans AT gmail.com. Thanks!

Owner
Klaas Kelchtermans
I was born as Klaas Kelchtermans
Klaas Kelchtermans
Bayes-Newton—A Gaussian process library in JAX, with a unifying view of approximate Bayesian inference as variants of Newton's algorithm.

Bayes-Newton Bayes-Newton is a library for approximate inference in Gaussian processes (GPs) in JAX (with objax), built and actively maintained by Wil

AaltoML 165 Nov 27, 2022
This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detection, built on SECOND.

3D-CVF This is the official implementation of 3D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object

YecheolKim 97 Dec 20, 2022
This repository is for DSA and CP scripts for reference.

dsa-script-collections This Repo is the collection of DSA and CP scripts for reference. Contents Python Bubble Sort Insertion Sort Merge Sort Quick So

Aditya Kumar Pandey 9 Nov 22, 2022
🐥A PyTorch implementation of OpenAI's finetuned transformer language model with a script to import the weights pre-trained by OpenAI

PyTorch implementation of OpenAI's Finetuned Transformer Language Model This is a PyTorch implementation of the TensorFlow code provided with OpenAI's

Hugging Face 1.4k Jan 05, 2023
Talk covering the features of skorch

Skorch Talk Skorch - A Union of Scikit-learn and PyTorch Presentation The slides can be downloaded at: download link. Google Colab Part One - MNIST Pa

Thomas J. Fan 3 Oct 20, 2020
Loopy belief propagation for factor graphs on discrete variables, in JAX!

PGMax implements general factor graphs for discrete probabilistic graphical models (PGMs), and hardware-accelerated differentiable loopy belief propagation (LBP) in JAX.

Vicarious 62 Dec 23, 2022
A boosting-based Multiple Instance Learning (MIL) package that includes MIL-Boost and MCIL-Boost

A boosting-based Multiple Instance Learning (MIL) package that includes MIL-Boost and MCIL-Boost

Jun-Yan Zhu 27 Aug 08, 2022
Research using Cirq!

ReCirq Research using Cirq! This project contains modules for running quantum computing applications and experiments through Cirq and Quantum Engine.

quantumlib 230 Dec 29, 2022
Official Implementation of CVPR 2022 paper: "Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning"

(CVPR 2022) Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning ArXiv This repo contains Official Implementat

Yujun Shi 24 Nov 01, 2022
PyTorch implementation of the paper Dynamic Data Augmentation with Gating Networks

Dynamic Data Augmentation with Gating Networks This is an official PyTorch implementation of the paper Dynamic Data Augmentation with Gating Networks

九州大学 ヒューマンインタフェース研究室 3 Oct 26, 2022
Save-restricted-v-3 - Save restricted content Bot For telegram

Save restricted content Bot Contact: Telegram A stable telegram bot to get restr

DEVANSH 11 Dec 21, 2022
codes for Self-paced Deep Regression Forests with Consideration on Ranking Fairness

Self-paced Deep Regression Forests with Consideration on Ranking Fairness This is official codes for paper Self-paced Deep Regression Forests with Con

Learning in Vision 4 Sep 11, 2022
Binary Passage Retriever (BPR) - an efficient passage retriever for open-domain question answering

BPR Binary Passage Retriever (BPR) is an efficient neural retrieval model for open-domain question answering. BPR integrates a learning-to-hash techni

Studio Ousia 147 Dec 07, 2022
Pytorch Implementation for Dilated Continuous Random Field

DilatedCRF Pytorch implementation for fully-learnable DilatedCRF. If you find my work helpful, please consider our paper: @article{Mo2022dilatedcrf,

DunnoCoding_Plus 3 Nov 13, 2022
Unofficial Implementation of MLP-Mixer, gMLP, resMLP, Vision Permutator, S2MLPv2, RaftMLP, ConvMLP, ConvMixer in Jittor and PyTorch.

Unofficial Implementation of MLP-Mixer, gMLP, resMLP, Vision Permutator, S2MLPv2, RaftMLP, ConvMLP, ConvMixer in Jittor and PyTorch! Now, Rearrange and Reduce in einops.layers.jittor are support!!

130 Jan 08, 2023
System Design course at HSE (2021)

System Design course at HSE (2021) Wiki-страница курса Структура репозитория: slides - директория с презентациями с занятий tasks - материалы для выпо

22 Dec 25, 2022
UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning This is the official PyTorch implementation for UniMoCo pape

dddzg 49 Jan 02, 2023
Locationinfo - A script helps the user to show network information such as ip address

Description This script helps the user to show network information such as ip ad

Roxcoder 1 Dec 30, 2021
Code for Private Recommender Systems: How Can Users Build Their Own Fair Recommender Systems without Log Data? (SDM 2022)

Private Recommender Systems: How Can Users Build Their Own Fair Recommender Systems without Log Data? (SDM 2022) We consider how a user of a web servi

joisino 20 Aug 21, 2022
The official PyTorch implementation of Curriculum by Smoothing (NeurIPS 2020, Spotlight).

Curriculum by Smoothing (NeurIPS 2020) The official PyTorch implementation of Curriculum by Smoothing (NeurIPS 2020, Spotlight). For any questions reg

PAIR Lab 36 Nov 23, 2022