I've just been trying out maze and tried out the step-by-step tutorial.

In Step 5 (5. Training the MazeEnv) the instructions are incomplete or wrong.

I was able to get it running in the end, but it took (us) quite some time. I'm not sure if this is a bug in maze or hydra, of if just some newer version of either library changes the behavior a little bit. But you should update the documentation such that it works out of the box for new users of the library.

The setup (under Ubuntu 2020.04):

>> mkdir maze5 && cd maze5
>> pyenv local 3.8.8
>> python -m venv .venv
>> source .venv/bin/activate
>> pip install maze-rl torch
>> pip list
Package                 Version
----------------------- -----------
hydra-core              1.1.0
hydra-nevergrad-sweeper 1.1.5
maze-rl                 0.1.7
torch                   1.9.0
...

Then just copy-pasted the files from the https://github.com/enlite-ai/maze-examples/tree/main/tutorial_maze_env/part03_maze_env repo and adjusted the _target paths in the config yamls (e.g. from _target_: tutorial_maze_env.part03_maze_env.env.maze_env.maze_env_factory to _target_: env.maze_env.maze_env_factory).

Problem 1:

When you run the suggested training command, Hydra will just complain that it can't find the configuration files.

>> maze-run -cn conf_train env=tutorial_cutting_2d_basic wrappers=tutorial_cutting_2d_basic \
    model=tutorial_cutting_2d_basic algorithm=ppo
In 'conf_train': Could not find 'model/tutorial_cutting_2d_basic'

Available options in 'model':
        flatten_concat
        flatten_concat_shared_embedding
        pixel_obs
        pixel_obs_rnn
        rllib
        vector_obs
        vector_obs_rnn
Config search path:
        provider=hydra, path=pkg://hydra.conf
        provider=main, path=pkg://maze.conf
        provider=schema, path=structured://

Fix:

You can just define the config directory for hydra with maze-run -cd conf -cn conf_train .... Then Hydra will find the 3 config files and load them correctly.

Problem 2:

After loading the config files, hydra tries to load the modules defined in the _target fields. And that fails immediatly with:

  ...
  File "***/maze5-uWAZh5bh/lib/python3.8/site-packages/hydra/_internal/instantiate/_instantiate2.py", line 104, in _resolve_target
    return _locate(target)
  File "***/maze5-uWAZh5bh/lib/python3.8/site-packages/hydra/_internal/utils.py", line 563, in _locate
    raise ImportError(f"Error loading module '{path}'") from e

ImportError: Error loading module 'env.maze_env.maze_env_factory'

Fix:

For some reason Hydra doesn't know the path to the directory from where we call maze-run. And therefore it doesn't find the env directory containing the maze_env file.

This is fixable by just setting the environment variable: export PYTHONPATH="$PYTHONPATH:$PWD/".

Thanks for using Hydra! I see that you are using Hydra 1.1 already which is great. One thing that is really recent is the ability to configure the config searchpath from the primary config. You can learn about it here.

This can probably eliminate the need of your users to even know what a ConfigSearchpathPlugin is.

Feel free to jump into the Hydra chat if you have any questions.

• Adds Soft Actor-Critic (SAC) Trainer (supporting Dictionary Observations and Actions)
• Simplifies the reward aggregation interface (now also supports multi-agent training)
• Extends PPO and A2C to multi-agent capable actor-critic trainers (individual agents vs. centralized critic)
• Adds option for custom rollout evaluators
• Adds option for shared weights in actor-critic settings
• Adds experiment and multi-run support for RunContext Python API

Changes

• made Maze compatible to Rllib 1.4
• updated to the recently released hydra 1.1.0
• Simpified API (RunContext): Experiment and evaluation support
• Fixed support of the nevergrad sweeper: made the LocalLauncher hydra plugin part of the wheel
• Replaced the (policy id, actor id) tuple with an ActorID class

Other

• various documentation improvements
• added ready-to-go Docker containers
• contribution guidelines, pull request templates etc. on GitHub

Features:

• Adds documentation for run_context
• Changes of simulated environment interfaces step_without_observation -> fast_step
• Adds seeding to environments, models and trainers
• Initial commit of the Maze Python API
• Adds an ExportGifWrapper
• Adds network architecture visualizations to Tensorboard Images
• adds incremental min/max stats
• adds categorical (support-based) value networks
• added value transformations

• Adds seeding to environments, models and trainers
• Initial commit of the Maze Python API
• Adds an ExportGifWrapper
• Adds network architecture visualizations to Tensorboard Images

• improved docs
• switch to RLlib version 1.3.0.
• full structured env support
  • policy interface now selects policy based on actor_id
• added testing dependencies to main package

• adds PointNetFeatureBlock to perception module
• adds Tensorboard hyper paramter visualization for hydra multiruns
• merges parallel and sequential dataset into a single InMemoryDataset

Improvements:

• Enable event collection from within the Wrapper stack
• Aligned StepSkipWrapper with the event system
• MonitoringWrapper: Logging of observations, actions and rewards throughout the wrapper stack, useful for diagnosis
• Make _recursive_ in Hydra config files compatible with Maze object instantiation

Features:

• Imitation Learning:
  • Added Evaluation Rollouts
  • Unified dataset structures (InMemoryDataset)
• GlobalPoolingBlock: now supports sum and max pooling
• ObservationNormalizationWrapper: Adds observation and observation distribution visualization to Tensorboard logging.
• Distribution: Introduced VectorEnv, refactored the single and multi process parallelization wrappers.

Features:

• hyper parameter optimization via grid search and Nevergrad
• plain python training example
• local hydra job launcher
• extend attention/transformer perception blocks

Fixes:

• cumulative stats logging