[Bug Report] mujoco rendering is not deterministic

[famishedrover]

Describe the bug

Describe the bug
Upon initializing a mujoco environment through gym (the issue is with mujoco_py and other packages like metaworld etc as well), when one resets the env and renders it the expected behavior would be that any number of renders would give the same image observation. This is true if the rendering backend is glfw (as on mac), however for other backends egl & omesa this is not true.

Code example

Code example
When running a simple code :

import gym 
x = gym.make('Humanoid-v2', render_mode='rgb_array')
x.reset(seed=2)
a = x.render()
b = x.render()

print((a!=b).sum())

The expected result should be 0. It is 0 if infact i'm running glfw (either onscreen / ofscreen on mac, or ofscreen with xvfb on headless server). But for egl / omesa backend this is non zero & significantly varies.

System Info

Describe the characteristic of your environment:

• gym version 0.26.2, mujoco 2.1.2.14, mujoco 2.2.0
• OS : CentOS7, Ubuntu 20x (tried on both)
• Python 3.9x

While I'm using gym 0.26.2 primarily, I can confirm this error appears even with latest gymnasium 0.29.1

Checklist

• I have checked that there is no similar issue in the repo

[pseudo-rnd-thoughts]

My mac is struggling to install v2 with mujoco-py but I can confirm that v4 doesn't have this issue

import numpy as np
import gymnasium as gym

env = gym.make('Humanoid-v2', render_mode='rgb_array')

# test reset render
env.reset(seed=123)
render_1 = env.render()
env.reset(seed=123)
render_2 = env.render()
print(f'Humanoid-v2 - reset render: {np.all(render_1 == render_2)}')

# test repeat render
env.reset(seed=123)
render_1 = env.render()
render_2 = env.render()
print(f'Humanoid-v2 - render render: {np.all(render_1 == render_2)}')

env = gym.make('Humanoid-v4', render_mode='rgb_array')

# test reset render
env.reset(seed=123)
render_1 = env.render()
env.reset(seed=123)
render_2 = env.render()
print(f'Humanoid-v4 - reset render: {np.all(render_1 == render_2)}')

# test repeat render
env.reset(seed=123)
render_1 = env.render()
render_2 = env.render()
print(f'Humanoid-v4 - render render: {np.all(render_1 == render_2)}')

@famishedrover Could you run the code above, you will need to install pip install "gymnasium[mujoco-py, mujoco]"
@rodrigodelazcano Any ideas / can you replicate?

[famishedrover]

I get this :

Found 8 GPUs for rendering. Using device 0.
Humanoid-v2 - reset render: False
Humanoid-v2 - render render: False
/home/mverma13/.local/lib/python3.9/site-packages/glfw/__init__.py:912: GLFWError: (65544) b'X11: The DISPLAY environment variable is missing'
  warnings.warn(message, GLFWError)
/home/mverma13/.local/lib/python3.9/site-packages/glfw/__init__.py:912: GLFWError: (65537) b'The GLFW library is not initialized'
  warnings.warn(message, GLFWError)
Humanoid-v4 - reset render: False
Humanoid-v4 - render render: False

Mac uses glfw for both onscreen and offscreen render <- which is working for me as well. If I force my headless ubuntu to use glfw (by using xvfb) then its working as well. But when I use the sim.render() i.e. mjrendercontextoffscreen in offscreen=True mode then the issues start. Therefore I feel that there maybe something wrong with the choice of rendering.

[pseudo-rnd-thoughts]

@rodrigodelazcano Do you know why this might be?
@famishedrover Why do you need deterministic rendering?

[rodrigodelazcano]

@famishedrover this issue is related to rendering with mujoco_py and we don't give support to the mujoco_py repository. We are keeping the old mujoco environment versions that depend on mujoco_py for reproducibility but we don't maintain them anymore. I suggest you use later versions that use the new official mujoco bindings (>v4).
@famishedrover, are you able to replicate @pseudo-rnd-thoughts's code and confirm that it works for you with the later environments?

[famishedrover]

I get the same result (False for all cases) :
I created a new conda env python=3.9 and installed
and ran pip install "gymnasium[mujoco-py, mujoco]"

I get the following :

Found 8 GPUs for rendering. Using device 0.
Humanoid-v2 - reset render: False
Humanoid-v2 - render render: False
/home/local/ASUAD/mverma13/anaconda3/envs/mujoco/lib/python3.9/site-packages/glfw/__init__.py:916: GLFWError: (65544) b'X11: The DISPLAY environment variable is missing'
  warnings.warn(message, GLFWError)
/home/local/ASUAD/mverma13/anaconda3/envs/mujoco/lib/python3.9/site-packages/glfw/__init__.py:916: GLFWError: (65537) b'The GLFW library is not initialized'
  warnings.warn(message, GLFWError)
Humanoid-v4 - reset render: False
Humanoid-v4 - render render: False

[pseudo-rnd-thoughts]

@famishedrover What system are you on? Looking at the rendering code on gymnasium's end, there is no randomness involved from what I can see.
Therefore, I would suspect that there is some form of randomness either in mujoco or the system.

Also, why do you need determinisim in the rendering?

[rodrigodelazcano]

Also, @famishedrover does this past issue fix your problem? #411 (comment)

[reginald-mclean]

@pseudo-rnd-thoughts @rodrigodelazcano the motivation for this issue is related to an issue they opened in Meta-World # 439

Motivation
I care about obtaining the image observation from a past state. Since storing images on the go is not feasible for my setup, I wanted to explore solutions where I can store less data intensive information like the observation etc. that is enough to exactly restore the environment state to get the image (or if there are other faster methods to get the image back).

[pseudo-rnd-thoughts]

env = gym.make('Humanoid-v4', render_mode='rgb_array')

for seed in range(100):

    # test reset render
    env.reset(seed=seed)
    render_1 = env.render()
    env.reset(seed=seed)
    render_2 = env.render()

    # test repeat render
    env.reset(seed=seed)
    render_3 = env.render()
    render_4 = env.render()
    print(f'reset render: {np.all(render_1 == render_2)}, render render: {np.all(render_3 == render_4)}')

This code all pass on my mac so I suspect that there is an issue with ubuntu for some reason

[famishedrover]

@rodrigodelazcano thanks for the suggestion but as I point out here xvfb works, but it doesn't use GPU rendering, so its very very slow.

@pseudo-rnd-thoughts If you inspect the mujoco_py rendering code, mac uses glfw by default <- which doesn't have issues (xvfb allows for using glfw rendering). The issue is more of it doesn't work with headless ubuntu. (Although I did arrange an Ubuntu machine where it was failing as well).

I'm still investigating other methods to install mujoco_py that may help with the issue, but I agree that this is a mujoco_py issue (but affects downstream gym / and other mujoco based envs). The issue remains with deepmind's mujoco as well.

[saran-t]

BTW the rendering helper in mujoco hasn't relied on GLFW for offscreen rendering on macOS for some time: https://github.com/google-deepmind/mujoco/blob/main/python/mujoco/cgl/__init__.py

[reginald-mclean]

@saran-t is that also true for MuJoco-py?

[saran-t]

No, no one's modified anything in mujoco_py for years.

[famishedrover]

More recently, I tried switching to docker devcontainers full time to have more consistent mujoco setup. I ended up rediscovering a known issue - EGL based rendering is non-deterministic apparently due to hardware / driver etc. google-deepmind/dm_control#100

I tried two ways of resetting the environment :

1. Store the actions and replay the actions from a copy of the env initialized from same seed.
2. Save environment state and restore state [MjData state]

Environments used : Metaworld and Gymnasum MuJoCo
Non determinism with respect to image rendering.

Findings :

1. EGL : both methods are nondeterministic
2. OSMESA : method 1 works fine and method 2 does not for Metaworld. For Gymnasium both methods work.
3. GLFW : both methods work.

TorchRL docs already state that MUJOCO_GL uses gpu rendering only with EGL.

I am yet to test with Mujoco3 but since this seems to be more of an EGL issue I do not have high hopes.

However, it would be really helpful to get a deterministic version of EGL based mujoco environments.

[saran-t]

Might be worth clarifying what "nondeterminism" means here -- the point is that the same sequence of OpenGL commands will lead to non-identical pixels depending on the implementation (i.e. driver).

There's isn't anything that makes EGL inherently more or less deterministic than GLFW. These are just different APIs that can be used to set up an OpenGL context (in fact GLFW isn't even a context setup API in itself but rather a middleware that can dispatch to GLX or EGL, among others).

[famishedrover]

@saran-t im still trying to figure out GLFW/EGL/OpenGL internals but it seems that using GLFW resolves all issues, albeit without gpu rendering which is much slower, than EGL. I support you in that EGL rendering should not be non-deterministic (produce different rendered images upon same call ) but it turns out that it does. It maybe in part MuJoCo implementation and hopefully it will be resolved in the future.

[saran-t]

It almost certainly can't be MuJoCo's code since the code doesn't even know how the OpenGL context was set up!

Could you please provide pixel diffs so that we know how much things change between renders? (Personally I think striving for pixel-identical 3D rendering is not a worthwhile endeavour)

[Kallinteris-Andreas]

Requesting close this issue and add label label:wontfix
This is only affecting the old mujoco_py based environments and causes a few pixels to be slightly different
it does not affect the new mujoco based environment with egl or glfw omesa.

This bug would require mujoco_py to be updated.