Add rendering wrapper with noise

[sparisi]

Description

Added two wrappers that add white noise to the rendering of the environments.
The first simply randomly add white noise to every pixel according to some probability.
The second adds biggers "patches" (the number of patches depends on the size of the patch and on what percentage of pixels we want to mask).
Noise is different at every step.

This is not a fix, but a new feature. It allows for noisy pixel observations and can easily make the environment partially-observable (especially the "patch" version). Helpful to turn MDPs into POMDPs from pixels.

The reasons why I added it as a rendering wrapper and not an observation wrapper are:

1. It can be used with both RGB and grayscale rendering,
2. It can be used with other observation wrappers (e.g., both dictionary with both pixels and state),
3. It can be used with "human" rendering.

However, it currently does not work with "rgb_array_list", because the render() function expects a single frame but it receives a list of frames. I couldn't figure out how "rgb_array_list" work. The best thing would be to have the noise added right after the frame is created and before it is appened to the list, but I couldn't find where this happens. I am happy to look more into this and make this even more generic.

One example of learning from pixels with noisy observation would be (matplotlib is just to show the noisy frame):

import gymnasium as gym
from gymnasium.wrappers.rendering import ObstructView
from gymnasium.wrappers.transform_observation import AddRenderObservation

env = gym.make("LunarLander-v3", render_mode="rgb_array")
env = ObstructView(env, obstructed_pixels_ratio=0.5, obstruction_width=100)
env = AddRenderObservation(env)
obs, _ = env.reset(seed=123)

from matplotlib import pyplot as plt
plt.imshow(obs)
plt.show()

I am not sure if the name "ObstructedView" is the best, I am happy to rename it to a better fit.

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

Screenshots

Here are some screenshots of the wrapper in action

import gymnasium as gym
from gymnasium.wrappers.rendering import HumanRendering, AddWhiteNoise

env = gym.make("LunarLander-v3", render_mode="rgb_array")
env = AddWhiteNoise(env, probability_of_noise_per_pixel=0.5)
env = HumanRendering(env)
obs, _ = env.reset(seed=123)
obs, *_ = env.step(env.action_space.sample())

import gymnasium as gym
from gymnasium.wrappers.rendering import HumanRendering, ObstructView

env = gym.make("LunarLander-v3", render_mode="rgb_array")
env = ObstructView(env, obstructed_pixels_ratio=0.5, obstruction_width=100)
env = HumanRendering(env)
obs, _ = env.reset(seed=123)
obs, *_ = env.step(env.action_space.sample())

Checklist:

• I have run the pre-commit checks with pre-commit run --all-files (see CONTRIBUTING.md instructions to set it up)
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[pseudo-rnd-thoughts]

I'm certainly interested in these wrappers, my only worry is that I don't want to add features that are uncommonly used and just add more wrappers to update which can be just part of a more specialised repo.

@sparisi Are these common wrappers within the literature? Why are these applied to the render and not the observation?

[sparisi]

@pseudo-rnd-thoughts
I am not super about literature. I have some students working on RL with noisy observations and having white noise on pixels seems the most obvious thing to do. About having patches, I know past work did something similar, but rather than having random patches the authors were fully obstructing a section of the frame for the whole episode (imagine having a gray bar in the middle of the screen). I can follow up later with some references.

I have a version where they are implemented as observation wrappers, but

1. The rendered image was still fully observable. Rendering the noisy observation makes it easier to debug and understand what the agent is doing (because we can see what it sees).
2. Applying the noise to the rendered imagine makes it compatible with any other observation wrapper. Just apply noisy rendering first and then AddRenderObservation and maybe even the one for grayscale frames. They stack nicely. And if in the future there will be more pixel-based wrappers, the noisy observation will still be compatible.

At least, that's based on my understanding of the current wrappers.

Side note. As I wrote, my wrappers do not work with "rgb_array_list" rendering. I can't find how this rendering works, ie, where the frames are concatenated into list. My guess is that this is used for vectorized environments and video rendering, but I cannot find exactly where/how they are called. Ideally, my wrappers should be called right after the environment is rendered and before frames are put into the list of frames.

[pseudo-rnd-thoughts]

I remembered a version of the dm-control environments where a background is added to make the environment more noisy. See https://github.com/nicklashansen/dmcontrol-generalization-benchmark
Given that Gymnasium doesn't have any random noise wrappers, it would make sense to add.
But that would be more towards adding randomness in the observation rather than rendering.

@sparisi would you be interested in changing this to be more observation focused? Or do you think that only rendering is important?

For rgb_array_list, the environment renders a list of renderings from the environment, list[RenderFrame] therefore, we would need to apply the noise to all individual frames or prevent users using it

[sparisi]

@pseudo-rnd-thoughts
Yes I can do that.
But when I made it as observation wrapper, the rendered environment was without noise. How can I add noise to the pixels as observation wrapper and also render the noise? It would make using the wrapper more user-friendly, because users can see what the agent sees as well.

[pseudo-rnd-thoughts]

I agree with the idea that if applying noise to an image observation, then we also apply noise to the rendering.

To implement, I would have an argument apply_noise_to_render where the noise is applied to the renders when render is called.
Also, your previous implementation only works with image Boxes, could we support continuous Boxes as well?

[sparisi]

Great, will do it.

What do you mean with "image Boxes"? I thought Box was the classic continuous (non-image) space. Do you mean applying white noise to small-dimensional observation space as well? Like, adding noise to the default (non-image) LunarLander space? I can also do that.

[pseudo-rnd-thoughts]

Box allows you to specify a dtype, allowing both continuous float32 inputs for classic control environment and discrete inputs like uint8 with Atari.
I'm not sure how we transfer a continuous control input noise pattern to a rendering.

If users used the RenderObservation wrapper then they can make their rendering the observation but I think otherwise, we have to disable applying the noise to the rendering unless the two are identical shapes