Describe the bug Ray memory issue occurred when running rnd ppo on montezuma's revenge of Atari env.

To Reproduce Run rnd ppo on montezuma's revenge

Expected behavior Memory issue occurs

Screenshots

Development Env. (OS, version, libraries): Linux Ubuntu, Python 3.8, requirement (jorldy0.0.2)

Additional context Add any other context about the problem here.

:star2: Hello! Thanks for contributing JORLDY!

Checklist

Please check if you consider the following items.

• [v] My code follows the style guidelines of this project
• [v] My code follows the naming convention of documentation
• [v] I have commented my code, particularly in hard-to-understand areas
• [v] My changes generate no new warnings or errors

Types of changes

Bugfix

Test Configuration

• OS: Windows 10
• Python version: 3.8
• Additional libraries: None

Description

• Fixed #44

The basic idea is that eval_manager in the child process should create its env. For now, distributed_train.py process doesn’t use env after creating agent config.

I am tried running V-MPO on atari Breakout, and it didn't seem to gain any momentum; Any reason why this might be? I tried changing some of the parameters in the config file and I still didn't get any improvement. Is this how it suppose to be at the beginning of training?

:star2: Hello! Thanks for contributing JORLDY!

Checklist

Please check if you consider the following items.

• [v] My code follows the style guidelines of this project
• [v] My code follows the naming convention of documentation
• [v] I have commented my code, particularly in hard-to-understand areas
• [v] My changes generate no new warnings or errors

Types of changes

Please describe the types of changes! (ex. Bugfix, New feature, Documentation, ...) New feature

Test Configuration

• OS: Linux Ubuntu
• Python version: 3.8
• Additional libraries: None

Description

Please describe the details of your contribution Envs which have Multi modal (image, vector) input can be applied to all agents.

Describe the bug A clear and concise description of what the bug is.

To Reproduce python main.py --async --config config.r2d2.atari --env.name breakout python main.py --async --config config.muzero.atari --env.name qbert

Expected behavior RayOutOfMemoryError

Screenshots

Development Env. (OS, version, libraries): Linux python 3.7.11 jorldy:0.3.0

Additional context Add any other context about the problem here. https://stackoverflow.com/questions/60175137/out-of-memory-with-ray-python-framework https://github.com/ray-project/ray/issues/5572

It seems that GC for ray shared memory doesn't work properly.

Describe the bug A clear and concise description of what the bug is.

Samples of Multistep agent has trash value about post-terminal state.

To Reproduce Steps to reproduce the behavior:

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Development Env. (OS, version, libraries): Please describe current development environment

Additional context Add any other context about the problem here.

update put&timeout to put_nowait

:star2: Hello! Thanks for contributing JORLDY!

Checklist

Please check if you consider the following items.

• [x] My code follows the style guidelines of this project contributing
• [x] My code follows the naming convention of documentation
• [x] I have commented my code, particularly in hard-to-understand areas
• [x] My changes generate no new warnings or errors

Types of changes

Please describe the types of changes! (ex. Bugfix, New feature, Documentation, ...)

Test Configuration

• OS:
• Python version:
• Additional libraries:

Description

Please describe the details of your contribution

optimize put method

Describe the bug A clear and concise description of what the bug is.

agent.memory.size is not defined correctly

To Reproduce Steps to reproduce the behavior:

run pytest after uncomment agent.memory.size

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Development Env. (OS, version, libraries): Please describe current development environment Linux Ubuntu

Additional context Add any other context about the problem here.

Describe the bug

mlagents_envs.exception.UnityEnvironmentException:

Couldn't launch the ./core/env/mlagents/DroneDelivery/Server/DroneDelivery environment. 
Provided filename does not match any environments.

To Reproduce

# docker
docker build -t jorldy -f ./docker/Dockerfile .
docker run -it --rm --name jorldy -v `pwd`:/JORLDY jorldy /bin/bash

python sync_distributed_train.py --config=config.ppo.drone_delivery_mlagent

Expected behavior A clear and concise description of what you expected to happen.

Screenshots

Development Env. (OS, version, libraries): Ubuntu 18.04.5 LTS", mlagents-envs 0.26.0

Additional context Add any other context about the problem here.

Describe the bug

1. Not running mlagents until I stalled hiredis
2. DroneDelivery env error, I think it's corrupted.

To Reproduce pip install -r requirements.txt python sync_distributed_train.py --config=config.ppo.drone_delivery_mlagent

Expected behavior

First, After I had installed requirements.txt I followed the commands "python sync_distributed_train.py --config=config.ppo.drone_delivery_mlagent" Then I saw "redis-py works best with hiredis please consider installing" in my case it's not causing any problem to run mlagents. but one on my friend couldn't run it until he installed hiredis.

Second, When I run mlagents. I could barely see Drone, Destination points. (please see the pic I attached) By overwriting files with this I could solve the problem.

Please check these errors. Thanks

Screenshots

Development Env. (OS, version, libraries): Windows 10, Anaconda, Python3.8.8

Is your feature request related to a problem? Please describe. Hi, thank you for sharing this project. For now it seems DQN doesn't check discrete or continuous in advance. When I change dqn.cartpole config

env = {
    "name":"cartpole",
    "render":False,
}

to

env = {
    "name":"cartpole",
    "render":False,
    "mode":"continuous",
}

it doesn't give any errors and isn't trained well. Since DQN is an algorithm for discrete action and buffer gives integer actions so continuous Cartpole env only run action = 1. (and I didn't really look into that other algorithms check the actions, but DQN doesn't)

Describe the solution you'd like It might be possible to insert assert statement in each algorithm codes.

Describe alternatives you've considered x

Additional context x

Describe the bug Unavailable moduels ['mlagent', 'mujoco', 'nes', 'procgen'] module: mlagent error: Traceback (most recent call last): File "e:\study\machineStudy\project\Jorldy\JORLDY\jorldy\core\env_init_.py", line 21, in module = import(module_path, fromlist=[None]) File "e:\study\machineStudy\project\Jorldy\JORLDY\jorldy\core\env\mlagent.py", line 1, in from mlagents_envs.environment import UnityEnvironment, ActionTuple ModuleNotFoundError: No module named 'mlagents_envs'

and ModuleNotFoundError: No module named 'mujoco_py' ModuleNotFoundError: No module named 'nes_py'

and ImportError: cannot import name 'ProcgenEnv' from partially initialized module 'procgen' (most likely due to a circular import) (e:\study\machineStudy\project\Jorldy\JORLDY\jorldy\core\env\procgen.py)

To Reproduce Steps to reproduce the behavior: main.py default_config_path = "config.ppo.pong_mlagent" and run

when i pip install mlagents-envs Couldn't launch the ./core/env/mlagents/Pong/Windows/Pong environment. Provided filename does not match any environments. File "E:\study\machineStudy\project\Jorldy\JORLDY\jorldy\core\env\mlagent.py", line 37, in init self.env = UnityEnvironment(

I change the mlagent code

    rootPath = os.path.abspath(os.path.dirname(__file__))+"/../../"
    env_path =rootPath+ f"./core/env/mlagents/{env_name}/{match_build()}/{env_name}"

and it is run

when it is run end program is no end when use async_distributed_train mlagent

the last log: Interact process done.

Expected behavior no error and run train success,and end success

Development Env. (OS, version, libraries): windows 10

Is your feature request related to a problem? Please describe. Currently, the state type stored as a transition in R2D2 is too large as float64. And if the sequence length is lengthened accordingly, the existing buffer size is too large.

Describe the solution you'd like

• Change the state type of transition to unit8.
• Reduce the buffer size of the config.
• R2D2 atari benchmark

Describe alternatives you've considered

• Fixed size when adding state to _transition in agent interact callback.

Additional context

• R2D2 atari benchmark

Describe the bug MuZero shows very good performance in some environment such as cartpole, pong mlagent, atari (pong, breakout). However, it shows bad performance in most of the Atari environment (spaceinvaders, qbert, enduro, seaquest, ...)

To Reproduce Try running MuZero algorithm in environments other than pong and breakout

Expected behavior It shows worse performance when compared to other algorithms.

Screenshots

Development Env. (OS, version, libraries): Linux, Python 3.8, jorldy 0.3.0 requirement

Additional context Add any other context about the problem here.

Please describe the feature you want to add. A clear and concise description of what the feature. Ex. I'm going to implement ...

Use Multi-GPU

Additional requirement A clear and concise description of additional requirement for the new feature

Reference Please append the reference about the feature

Describe the bug RuntimeError when running mpo

To Reproduce

python main.py --config.mpo.atari --env.name breakout --sync

When config is modified with the values shown in the paper, it occurs faster and more frequently.

Expected behavior

• An error occurred when calculating multinomial method with pi from Actor network.
• RuntimeError: probability tensor contains either inf, nan or element < 0

Screenshots

training graph

• default config, green, also causes an error at 7M.

error txt

mpo generated agent code

Development Env. (OS, version, libraries):

• linux
• V4XLARGE
• python 3.7.11
• jorldy:0.3.0

Additional context

• Even with default config, an error sometimes occurs after a lot of learning.
• If you set the config to the value shown in the paper, you get a much higher score at the beginning, but an error quickly occurs.