CARL provides highly configurable contextual extensions to several well-known RL environments.

Related tags

Deep LearningCARL
Overview

The CARL Benchmark Library

CARL (context adaptive RL) provides highly configurable contextual extensions to several well-known RL environments. It's designed to test your agent's generalization capabilities in all scenarios where intra-task generalization is important.

Benchmarks include:

  • OpenAI gym classic control suite extended with several physics context features like gravity or friction

  • OpenAI gym Box2D BipedalWalker, LunarLander and CarRacing, each with their own modification possibilities like new vehicles to race

  • All Brax locomotion environments with exposed internal features like joint strength or torso mass

  • Super Mario (TOAD-GAN), a procedurally generated jump'n'run game with control over level similarity

  • RNADesign, an environment for RNA design given structure constraints with structures from different datasets to choose from

Screenshot of each environment included in CARL.

Installation

We recommend you use a virtual environment (e.g. Anaconda) to install CARL and its dependencies. We recommend and test with python 3.9 under Linux.

First, clone our repository and install the basic requirements:

git clone https://github.com/automl/CARL.git --recursive
cd CARL
pip install .

This will only install the basic classic control environments, which should run on most operating systems. For the full set of environments, use the install options:

pip install -e .[box2d, brax, rna, mario]

These may not be compatible with Windows systems. Box2D environment may need to be installed via conda on MacOS systems:

conda install -c conda-forge gym-box2d

In general, we test on Linux systems, but aim to keep the benchmark compatible with MacOS as much as possible. Mario at this point, however, will not run on any operation system besides Linux

To install the additional requirements for ToadGAN:

javac src/envs/mario/Mario-AI-Framework/**/*.java

If you want to use the RNA design environment:

cd src/envs/rna/learna
make requirements
make data

In case you want to run our experiments or use our training files, also install the experiment dependencies:

pip install -e .[experiments]

Train an Agent

To get started with CARL, you can use our 'train.py' script. It will train a PPO agent on the environment of your choice with custom context variations that are sampled from a standard deviation.

To use MetaCartPole with variations in gravity and friction by 20% compared to the default, run:

python train.py 
--env CARLCartPoleEnv 
--context_args gravity friction
--default_sample_std_percentage 0.2
--outdir <result_location>

You can use the plotting scripts in src/eval to view the results.

CARL's Contextual Extension

CARL contextually extends the environment by making the context visible and configurable. During training we therefore can encounter different contexts and train for generalization. We exemplarily show how Brax' Fetch is extended and embedded by CARL. Different instiations can be achieved by setting the context features to different values.

CARL contextually extends Brax' Fetch.

Cite Us

@misc{CARL,
  author    = {C. Benjamins and 
               T. Eimer and 
               F. Schubert and 
               A. Biedenkapp and 
               B. Rosenhahn and 
               F. Hutter and 
               M. Lindauer},
  title     = {CARL: A Benchmark for Contextual and Adaptive Reinforcement Learning},
  howpublished = {https://github.com/automl/CARL},
  year      = {2021},
  month     = aug,
}

References

OpenAI gym, Brockman et al., 2016. arXiv preprint arXiv:1606.01540

Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation, Freeman et al., NeurIPS 2021 (Dataset & Benchmarking Track)

TOAD-GAN: Coherent Style Level Generation from a Single Example, Awiszus et al., AIIDE 2020

Learning to Design RNA, Runge et al., ICRL 2019

License

CARL falls under the Apache License 2.0 (see file 'LICENSE') as is permitted by all work that we use. This includes CARLMario, which is not based on the Nintendo Game, but on TOAD-GAN and TOAD-GUI running under an MIT license. They in turn make use of the Mario AI framework (https://github.com/amidos2006/Mario-AI-Framework). This is not the original game but a replica, explicitly built for research purposes and includes a copyright notice (https://github.com/amidos2006/Mario-AI-Framework#copyrights ).

Comments
  • Rna fixup

    Rna fixup

    RNA is now better documented and more easily runnable. There's also an option to subsample the datasets instead of always using all instances per context.

    The thing that's missing right now are more context options like filtering by solvers or GC-content, but those aren't easily extractable from our data right now, so that's a separate work package all together.

    opened by TheEimer 6
  • Gym 0.22.0

    Gym 0.22.0

    • update required minimum gym version number
    • added pygame as a requirement because it is not picked up by the gym requirements
    • getting rid of CustomBipedalWalkerEnv because the functionality of changing the gravity is covered by CARLEnv (same for CustomLunarLanderEnv)
    • add high game over penalty for LunarLander by a wrapper
    opened by benjamc 6
  • Instance selection

    Instance selection

    Instance selection now is a class. Default is still roundrobin selection. An instance is only selected when env.reset() (or to be more specific, _progress_instance() is called.

    opened by benjamc 4
  • Added Encoders

    Added Encoders

    Context encoders have been added as a folder and an experiment for running the encoder added in the experiments folders. Since the working directory is the experiment one, I had to add an absolute path for the saved weights. This might need to be changed in the config file

    opened by amsks 4
  • Update References with correct conference

    Update References with correct conference

    Thanks for the pointer to the survey, but it hasn't been published anywhere, so that detail is incorrect (I wouldn't want to claim that it's published somewhere when it isn't).

    opened by RobertKirk 3
  • Performance Deviations in Brax

    Performance Deviations in Brax

    Comparing HalfCheetah in Brax (via gym.make and then wrapped as here: https://github.com/google/brax/blob/main/notebooks/training_torch.ipynb) vs in CARL makes a big difference in return even when the context is kept static. Do we do any unexpected reward normalization? Does the way we reset the env make a difference compared to theirs (as we actually update the simluation)?

    bug 
    opened by TheEimer 2
  • Integrate DM Control

    Integrate DM Control

    • [ ] (convert test file to jupyter notebook. I would like to keep that)
    • [ ] check tests / write more to increase coverage
    • [x] update README.md
    • [x] update documentation
    • [x] add dm_control to requirements
    • [x] support dict observation space
    documentation tests 
    opened by benjamc 2
  • Fix gym version

    Fix gym version

    Gym released a new version where the signature of the step function has changed. This affects our code and requires a separate PR. For now, fix the gym version.

    opened by benjamc 1
  • Initial statedistrs #48

    Initial statedistrs #48

    #48 Make initial state distribution configurable. So far, only uniform distributions are used and the bounds can be adjusted.

    Classic control:

    • [x] Acrobot
    • [x] Pendulum
    • [x] MountainCar (normal distribution instead of uniform)
    • [x] MountainCarContinuous (uniform distribution)
    • [x] CartPole

    Box2d

    • [x] LunarLander

    • [ ] (maybe/later) Make distributions fully configurable by passing the distribution class and its parameters.

    • [x] Update documentation: Contexts are automatically filled with the default context if underspecified.

    opened by benjamc 1
  • Integrate dmcontrol

    Integrate dmcontrol

    Add support for dm control environments. Integrated walker, quadruped and fish.

    In dmc environments there is an additional setting for the context, namely the context mask, which can reduce the amount of context features.

    opened by sebidoe 1
  • use appropriate library for building states

    use appropriate library for building states

    So far, when we do not hide the context, we concatenate the context to the state. For jax based environments (brax) this means that the state is converted from a jax to a numpy array. Now, the state builder checks which library to use and keeps jax states as jax arrays and numpy states as numpy arrays.

    Noticed in #42.

    opened by benjamc 1
  • AttributeError: 'System' object has no attribute 'body_idx' in brax

    AttributeError: 'System' object has no attribute 'body_idx' in brax

    when running test/test_all_envs.py, there is AttributeError: 'System' object has no attribute 'body_idx' in carl_fetch and carl_humanoid environments.

    opened by andy-james0310 3
Releases(v0.2.0)
  • v0.2.0(Jul 12, 2022)

    • Integrate dm control environments (#55)
    • Add context masks to only append those to the state (#54)
    • Extend classic control environments to parametrize initial state distributions (#52)
    • Remove RNA environment for maintenance (#61)
    • Fixed pre-commit (mypy, black, flake8, isort) (#62)
    Source code(tar.gz)
    Source code(zip)
Owner
AutoML-Freiburg-Hannover
AutoML-Freiburg-Hannover
Source code for From Stars to Subgraphs

GNNAsKernel Official code for From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awareness Visualizations GNN-AK(+) GNN-AK(+) with Subgra

44 Dec 19, 2022
Plaything for Autistic Children (demo for PaddlePaddle/Wechaty/Mixlab project)

星星的孩子 - 一款为孤独症孩子设计的聊天机器人游戏 孤独症儿童是目前常常被忽视的一类群体。他们有着类似性格内向的特征,实际却受着广泛性发育障碍的折磨。 项目背景 这类儿童在与人交往时存在着沟通障碍,其特点表现在: 社交交流差,互动障碍明显 认知能力有限,被动认知 兴趣狭窄,重复刻板,缺乏变化和想象

Tianyi Pan 35 Nov 24, 2022
QR2Pass-project - A proof of concept for an alternative (passwordless) authentication system to a web server

QR2Pass This is a proof of concept for an alternative (passwordless) authenticat

4 Dec 09, 2022
This repo contains implementation of different architectures for emotion recognition in conversations.

Emotion Recognition in Conversations Updates 🔥 🔥 🔥 Date Announcements 03/08/2021 🎆 🎆 We have released a new dataset M2H2: A Multimodal Multiparty

Deep Cognition and Language Research (DeCLaRe) Lab 1k Dec 30, 2022
Offline Multi-Agent Reinforcement Learning Implementations: Solving Overcooked Game with Data-Driven Method

Overcooked-AI We suppose to apply traditional offline reinforcement learning technique to multi-agent algorithm. In this repository, we implemented be

Baek In-Chang 14 Sep 16, 2022
Open source code for the paper of Neural Sparse Voxel Fields.

Neural Sparse Voxel Fields (NSVF) Project Page | Video | Paper | Data Photo-realistic free-viewpoint rendering of real-world scenes using classical co

Meta Research 647 Dec 27, 2022
Pytorch code for paper "Image Compressed Sensing Using Non-local Neural Network" TMM 2021.

NL-CSNet-Pytorch Pytorch code for paper "Image Compressed Sensing Using Non-local Neural Network" TMM 2021. Note: this repo only shows the strategy of

WenxueCui 7 Nov 07, 2022
This game was designed to encourage young people not to gamble on lotteries, as the probablity of correctly guessing the number is infinitesimal!

Lottery Simulator 2022 for Web Launch Application Developed by John Seong in Ontario. This game was designed to encourage young people not to gamble o

John Seong 2 Sep 02, 2022
Experiments with Fourier layers on simulation data.

Factorized Fourier Neural Operators This repository contains the code to reproduce the results in our NeurIPS 2021 ML4PS workshop paper, Factorized Fo

Alasdair Tran 57 Dec 25, 2022
codes for paper Combining Dynamic Local Context Focus and Dependency Cluster Attention for Aspect-level sentiment classification

DLCF-DCA codes for paper Combining Dynamic Local Context Focus and Dependency Cluster Attention for Aspect-level sentiment classification. submitted t

15 Aug 30, 2022
PyTorch implementation of "VRT: A Video Restoration Transformer"

VRT: A Video Restoration Transformer Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, Luc Van Gool Computer

Jingyun Liang 837 Jan 09, 2023
SimDeblur is a simple framework for image and video deblurring, implemented by PyTorch

SimDeblur (Simple Deblurring) is an open source framework for image and video deblurring toolbox based on PyTorch, which contains most deep-learning based state-of-the-art deblurring algorithms. It i

220 Jan 07, 2023
Genpass - A Passwors Generator App With Python3

Genpass Welcom again into another python3 App this is simply an Passwors Generat

Mal4D 1 Jan 09, 2022
Generate Contextual Directory Wordlist For Target Org

PathPermutor Generate Contextual Directory Wordlist For Target Org This script generates contextual wordlist for any target org based on the set of UR

8 Jun 23, 2021
Megaverse is a new 3D simulation platform for reinforcement learning and embodied AI research

Megaverse Megaverse is a new 3D simulation platform for reinforcement learning and embodied AI research. The efficient design of the engine enables ph

Aleksei Petrenko 191 Dec 23, 2022
Parallel and High-Fidelity Text-to-Lip Generation; AAAI 2022 ; Official code

Parallel and High-Fidelity Text-to-Lip Generation This repository is the official PyTorch implementation of our AAAI-2022 paper, in which we propose P

Zhying 77 Dec 21, 2022
Efficient training of deep recommenders on cloud.

HybridBackend Introduction HybridBackend is a training framework for deep recommenders which bridges the gap between evolving cloud infrastructure and

Alibaba 111 Dec 23, 2022
Decoding the Protein-ligand Interactions Using Parallel Graph Neural Networks

Decoding the Protein-ligand Interactions Using Parallel Graph Neural Networks Requirements python 0.10+ rdkit 2020.03.3.0 biopython 1.78 openbabel 2.4

Neeraj Kumar 3 Nov 23, 2022
Powerful and efficient Computer Vision Annotation Tool (CVAT)

Computer Vision Annotation Tool (CVAT) CVAT is free, online, interactive video and image annotation tool for computer vision. It is being used by our

OpenVINO Toolkit 8.6k Jan 01, 2023
GANfolk: Using AI to create portraits of fictional people to sell as NFTs

GANfolk are AI-generated renderings of fictional people. Each image in the collection was created by a pair of Generative Adversarial Networks (GANs) with names and backstories also created with AI.

Robert A. Gonsalves 32 Dec 02, 2022