[rllib] Improve IMPALA examples and premerge

[pseudo-rnd-thoughts]

Description

Updates the IMPALA examples and premerge with CartPole and TicTacToe.
We only have a minimal number of examples as most users should use PPO or APPO rather than IMPALA.

[gemini-code-assist]

Code Review

This pull request significantly improves the IMPALA and APPO examples and associated tests. The changes include refactoring release tests for clarity, renaming cluster configuration files for better descriptiveness, and replacing old example scripts with new, well-documented ones. The new examples for APPO and IMPALA are a great addition.

My review focuses on ensuring consistency in the new documentation and configurations. I've pointed out a few places where comments, tables, and default values are out of sync. Addressing these will improve the maintainability and usability of these examples and tests. Specifically, I've noted:

• Inconsistent test matrices for APPO between release_tests.yaml and BUILD.bazel.
• A copy-paste error in the IMPALA test matrix header in BUILD.bazel.
• Several new example scripts have placeholders or outdated information in their docstrings regarding expected results.

Overall, this is a valuable cleanup and improvement. Great work!

[gemini-code-assist]

The APPO test matrix table here seems to be inconsistent with the one in rllib/BUILD.bazel. For example, the success criteria for CartPole is 450 here, but 200 in BUILD.bazel. Similarly for Stateless Cartpole (350 vs 150), TicTacToe (0 vs -0.2), and Atari (18 vs 5). Also, the column structure is different. It would be great to unify these tables to avoid confusion and improve maintainability.

[gemini-code-assist]

It seems there's a copy-paste error in the comment table for IMPALA tests. The header says "APPO (14 total tests)" but the tests below are for IMPALA. This should be corrected to "IMPALA".

Suggested change

	# | APPO (14 total tests) | | Number of Learners (Device) |
	# | Environment | Success | Local (CPU) | Single (CPU) | Single (GPU) | Multi (GPU) |
	# |--------------------------------|---------|-------------|-----------------|--------------|-------------|
	# | (SA/D) Cartpole | 200 | ✅ | ❌ | ❌ | ❌ |
	# | (MA/D) TicTacToe | -2.0 | ❌ | ❌ | ❌ | ✅ |
	# | IMPALA (2 total tests) | | Number of Learners (Device) |
	# | Environment | Success | Local (CPU) | Single (CPU) | Single (GPU) | Multi (GPU) |
	# |--------------------------------|---------|-------------|-----------------|--------------|-------------|
	# | (SA/D) Cartpole | 200 | ✅ | ❌ | ❌ | ❌ |
	# | (MA/D) TicTacToe | -2.0 | ❌ | ❌ | ❌ | ✅ |

[gemini-code-assist]

The docstring mentions that the algorithm should reach a reward threshold of "XX on Breakout". This seems to be a placeholder. Could you please update "XX" with the actual expected reward (e.g., 18.0 as per default_reward) and change "Breakout" to "Pong" to match the default environment for this script?

Suggested change

	The algorithm should reach the default reward threshold of XX on Breakout
	within 10 million timesteps (40 million frames with 4x frame stacking,
	see: `default_timesteps` in the code).
	The algorithm should reach the default reward threshold of 18.0 on Pong
	within 10 million timesteps (40 million frames with 4x frame stacking,
	see: `default_timesteps` in the code).

[gemini-code-assist]

The "Results to expect" section in the docstring seems to contain information for HalfCheetah ("reward threshold of 9000.0 within 2 million timesteps"), but the script defaults are for Humanoid-v4 (default_reward=800.0, default_timesteps=3_000_000). This looks like a copy-paste from the old halfcheetah_appo.py example. Please update the docstring to reflect the correct environment and expected results for Humanoid-v4.

Suggested change

	The algorithm should reach the default reward threshold of 9000.0 within
	2 million timesteps (see: `default_timesteps` in the code).
	The number of environment steps can be changed through
	The algorithm should reach the default reward threshold of 800.0 on Humanoid-v4
	within 3 million timesteps (see: `default_timesteps` in the code).

[gemini-code-assist]

The docstring mentions an expected reward of 300.0 within 2 million timesteps, but the script's defaults are default_reward=350.0 and default_timesteps=5_000_000. Please update the docstring to be consistent with the code defaults.

Suggested change

	The algorithm should reach the default reward threshold of 300.0 (500.0 is the
	maximum) within approximately 2 million timesteps (see: `default_timesteps`
	in the code). The number of environment
	steps can be changed through argparser's `default_timesteps`.
	The algorithm should reach the default reward threshold of 350.0 (500.0 is the
	maximum) within approximately 5 million timesteps (see: `default_timesteps`
	in the code). The number of environment
	steps can be changed through argparser's `default_timesteps`.

[gemini-code-assist]

The docstring has a placeholder "XX" for the expected mean return and an outdated value for default_timesteps. Could you please replace "XX" with the actual expected value (e.g., -0.5) and update the number of timesteps to match the script's default of 10,000,000?

Suggested change

	Training will run for 100 thousand timesteps (see: `default_timesteps` in the
	code) for p0 (policy 0) to achieve a mean return of XX compared to the
	random policy. The number of environment steps can be changed through
	`default_timesteps`. The trainable policies should gradually improve their
	Training will run for 10 million timesteps (see: `default_timesteps` in the
	code) for p0 (policy 0) to achieve a mean return of -0.5 compared to the
	random policy. The number of environment steps can be changed through
	`default_timesteps`. The trainable policies should gradually improve their