Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation

Overview

NVIDIA Source Code License Python 3.7

OSCAR

Project Page | Paper

This repository contains the codebase used in OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation.

More generally, this codebase is a modular framework built upon IsaacGym, and intended to support future robotics research leveraging large-scale training.

Of note, this repo contains:

  • High-quality controller implementations of OSC, IK, and Joint-Based controllers that have been fully parallelized for PyTorch
  • Complex Robot Manipulation tasks for benchmarking learning algorithms
  • Modular structure enabling rapid prototyping of additional robots, controllers, and environments

Requirements

  • Linux machine
  • Conda
  • NVIDIA GPU + CUDA

Getting Started

First, clone this repo and initialize the submodules:

git clone https://github.com/NVlabs/oscar.git
cd oscar
git submodule update --init --recursive

Next, create a new conda environment to be used for this repo and activate the repo:

bash create_conda_env_oscar.sh
conda activate oscar

This will create a new conda environment named oscar and additional install some dependencies. Next, we need IsaacGym. This repo itself does not contain IsaacGym, but is compatible with any version >= preview 3.0.

Install and build IsaacGym HERE.

Once installed, navigate to the python directory and install the package to this conda environment:

(oscar) cd <ISAACGYM_REPO_PATH>/python
(oscar) pip install -e .

Now with IsaacGym installed, we can finally install this repo as a package:

(oscar) cd <OSCAR_REPO_PATH>
(oscar) pip install -e .

That's it!

Training

Provided are helpful scripts for running training, evaluation, and finetuning. These are found in the Examples directory. You can set the Task, Controller, and other parameters directly at the top of the example script. They should run out of the box, like so:

cd examples
bash train.sh

For evaluation (including zero-shot), you can modify and run:

bash eval.sh

For finetuning on the published out-of-distribution task settings using a pretrained model, you can modify and run:

bash finetune.sh

To pretrain the initial OSCAR base network, you can modify and run:

bash pretrain_oscar.sh

Reproducing Paper Results

We provide all of our final trained models used in our published results, found in trained_models section.

Adding Custom Modules

This repo is designed to be built upon and enable future large-scale robot learning simulation research. You can add your own custom controller by seeing an example controller like the OSC controller, your own custom robot agent by seeing an example agent like the Franka agent, and your own custom task by seeing an example task like the Push task.

License

Please check the LICENSE file. OSCAR may be used non-commercially, meaning for research or evaluation purposes only. For business inquiries, please contact [email protected].

Citation

Please cite OSCAR if you use this framework in your publications:

@inproceedings{wong2021oscar,
  title={OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation},
  author={Josiah Wong and Viktor Makoviychuk and Anima Anandkumar and Yuke Zhu},
  booktitle={arXiv preprint arXiv:2110.00704},
  year={2021}
}
EMNLP 2021 Findings' paper, SCICAP: Generating Captions for Scientific Figures

SCICAP: Scientific Figures Dataset This is the Github repo of the EMNLP 2021 Findings' paper, SCICAP: Generating Captions for Scientific Figures (Hsu

Edward 26 Nov 21, 2022
Game Agent Framework. Helping you create AIs / Bots that learn to play any game you own!

Serpent.AI - Game Agent Framework (Python) Update: Revival (May 2020) Development work has resumed on the framework with the aim of bringing it into 2

Serpent.AI 6.4k Jan 05, 2023
Linear Variational State Space Filters

Linear Variational State Space Filters To set up the environment, use the provided scripts in the docker/ folder to build and run the codebase inside

0 Dec 13, 2021
Code for EMNLP'21 paper "Types of Out-of-Distribution Texts and How to Detect Them"

ood-text-emnlp Code for EMNLP'21 paper "Types of Out-of-Distribution Texts and How to Detect Them" Files fine_tune.py is used to finetune the GPT-2 mo

Udit Arora 19 Oct 28, 2022
Implementation of RegretNet with Pytorch

Dependencies are Python 3, a recent PyTorch, numpy/scipy, tqdm, future and tensorboard. Plotting with Matplotlib. Implementation of the neural network

Horris zhGu 1 Nov 05, 2021
Homepage of paper: Paint Transformer: Feed Forward Neural Painting with Stroke Prediction, ICCV 2021.

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction [Paper] [Official Paddle Implementation] [Huggingface Gradio Demo] [Unofficial

442 Dec 16, 2022
PyTorch implementation of the ExORL: Exploratory Data for Offline Reinforcement Learning

ExORL: Exploratory Data for Offline Reinforcement Learning This is an original PyTorch implementation of the ExORL framework from Don't Change the Alg

Denis Yarats 52 Jan 01, 2023
Deep Image Search is an AI-based image search engine that includes deep transfor learning features Extraction and tree-based vectorized search.

Deep Image Search - AI-Based Image Search Engine Deep Image Search is an AI-based image search engine that includes deep transfer learning features Ex

139 Jan 01, 2023
The pyrelational package offers a flexible workflow to enable active learning with as little change to the models and datasets as possible

pyrelational is a python active learning library developed by Relation Therapeutics for rapidly implementing active learning pipelines from data management, model development (and Bayesian approximat

Relation Therapeutics 95 Dec 27, 2022
PyTorch inference for "Progressive Growing of GANs" with CelebA snapshot

Progressive Growing of GANs inference in PyTorch with CelebA training snapshot Description This is an inference sample written in PyTorch of the origi

320 Nov 21, 2022
3D-printable hand-strapped keyboard

Note: This repo has not been cleaned up and prepared for general consumption at all. This is just a dump of the project files. If there is any interes

Wojciech Baranowski 41 Dec 31, 2022
The code for our paper CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention.

CrossFormer This repository is the code for our paper CrossFormer: A Versatile Vision Transformer Based on Cross-scale Attention. Introduction Existin

cheerss 238 Jan 06, 2023
Self-supervised spatio-spectro-temporal represenation learning for EEG analysis

EEG-Oriented Self-Supervised Learning and Cluster-Aware Adaptation This repository provides a tensorflow implementation of a submitted paper: EEG-Orie

Wonjun Ko 4 Jun 09, 2022
Demo for Real-time RGBD-based Extended Body Pose Estimation paper

Real-time RGBD-based Extended Body Pose Estimation This repository is a real-time demo for our paper that was published at WACV 2021 conference The ou

Renat Bashirov 118 Dec 26, 2022
Decoding the Protein-ligand Interactions Using Parallel Graph Neural Networks

Decoding the Protein-ligand Interactions Using Parallel Graph Neural Networks Requirements python 0.10+ rdkit 2020.03.3.0 biopython 1.78 openbabel 2.4

Neeraj Kumar 3 Nov 23, 2022
In this project, we'll be making our own screen recorder in Python using some libraries.

Screen Recorder in Python Project Description: In this project, we'll be making our own screen recorder in Python using some libraries. Requirements:

Hassan Shahzad 4 Jan 24, 2022
Hierarchical probabilistic 3D U-Net, with attention mechanisms (โ€”๐˜ˆ๐˜ต๐˜ต๐˜ฆ๐˜ฏ๐˜ต๐˜ช๐˜ฐ๐˜ฏ ๐˜œ-๐˜•๐˜ฆ๐˜ต, ๐˜š๐˜Œ๐˜™๐˜ฆ๐˜ด๐˜•๐˜ฆ๐˜ต) and a nested decoder structure with deep supervision (โ€”๐˜œ๐˜•๐˜ฆ๐˜ต++).

Hierarchical probabilistic 3D U-Net, with attention mechanisms (โ€”๐˜ˆ๐˜ต๐˜ต๐˜ฆ๐˜ฏ๐˜ต๐˜ช๐˜ฐ๐˜ฏ ๐˜œ-๐˜•๐˜ฆ๐˜ต, ๐˜š๐˜Œ๐˜™๐˜ฆ๐˜ด๐˜•๐˜ฆ๐˜ต) and a nested decoder structure with deep supervision (โ€”๐˜œ๐˜•๐˜ฆ๐˜ต++). Built in TensorFlow 2.5. Configured for vox

Diagnostic Image Analysis Group 32 Dec 08, 2022
End-to-End Object Detection with Fully Convolutional Network

This project provides an implementation for "End-to-End Object Detection with Fully Convolutional Network" on PyTorch.

472 Dec 22, 2022
Implementation of Memformer, a Memory-augmented Transformer, in Pytorch

Memformer - Pytorch Implementation of Memformer, a Memory-augmented Transformer, in Pytorch. It includes memory slots, which are updated with attentio

Phil Wang 60 Nov 06, 2022
"Projelerle Yapay Zeka Ve Bilgisayarlฤฑ Gรถrรผ" Kitabฤฑmฤฑn projeleri

"Projelerle Yapay Zeka Ve Bilgisayarlฤฑ Gรถrรผ" Kitabฤฑmฤฑn projeleri Bu Github Reposundaki tรผm projeler; kaleme almฤฑลŸ olduฤŸum "Projelerle Yapay Zekรข ve Bi

รœmit Aksoylu 4 Aug 03, 2022