Save observation history config with checkpoint and use it on play

[saikishor]

This PR allows to save the observation history of actor or critic groups or at the term level in the checkpoints configuration, and this can be later reused to setup the observation while playing.

Right now, the user has to edit the files to set the history length for playing the checkpoint. If not, the following error appears

RuntimeError: Error(s) in loading state_dict for MLPModel:
	size mismatch for obs_normalizer._mean: copying a param with shape torch.Size([1, 111]) from checkpoint, the shape in current model is torch.Size([1, 99]).
	size mismatch for obs_normalizer._var: copying a param with shape torch.Size([1, 111]) from checkpoint, the shape in current model is torch.Size([1, 99]).
	size mismatch for obs_normalizer._std: copying a param with shape torch.Size([1, 111]) from checkpoint, the shape in current model is torch.Size([1, 99]).
	size mismatch for mlp.0.weight: copying a param with shape torch.Size([512, 111]) from checkpoint, the shape in current model is torch.Size([512, 99]).

With the proposed change, we save the checkpoint with the following information:

Saving checkpoint to logs/rsl_rl/g1_velocity/2026-04-12_10-05-00_mjlab_1204_g1_lin_vel_obs_hist/model_550.pt with obs_history_cfg: {'actor': {'history_length': None, 'flatten_history_dim': True, 'terms': {'base_lin_vel': {'history_length': 5, 'flatten_history_dim': True}, 'base_ang_vel': {'history_length': 0, 'flatten_history_dim': True}, 'projected_gravity': {'history_length': 0, 'flatten_history_dim': True}, 'joint_pos': {'history_length': 0, 'flatten_history_dim': True}, 'joint_vel': {'history_length': 0, 'flatten_history_dim': True}, 'actions': {'history_length': 0, 'flatten_history_dim': True}, 'command': {'history_length': 0, 'flatten_history_dim': True}}}, 'critic': {'history_length': None, 'flatten_history_dim': True, 'terms': {'base_lin_vel': {'history_length': 5, 'flatten_history_dim': True}, 'base_ang_vel': {'history_length': 0, 'flatten_history_dim': True}, 'projected_gravity': {'history_length': 0, 'flatten_history_dim': True}, 'joint_pos': {'history_length': 0, 'flatten_history_dim': True}, 'joint_vel': {'history_length': 0, 'flatten_history_dim': True}, 'actions': {'history_length': 0, 'flatten_history_dim': True}, 'command': {'history_length': 0, 'flatten_history_dim': True}, 'foot_height': {'history_length': 0, 'flatten_history_dim': True}, 'foot_air_time': {'history_length': 0, 'flatten_history_dim': True}, 'foot_contact': {'history_length': 0, 'flatten_history_dim': True}, 'foot_contact_forces': {'history_length': 0, 'flatten_history_dim': True}}}}

I observed the issue when I tried to launch a training of one of my observation terms to 5, and then tried to run the checkpoint it failed until I explicitly set the history length of the particular term to the appropriate value

[louislelay]

Hi @saikishor, drive-by comment: do we actually need to persist the critic's observation history here? At eval time we only roll out the actor, don't we?

[saikishor]

Hi @saikishor, drive-by comment: do we actually need to persist the critic's observation history here? At eval time we only roll out the actor, don't we?

Yes, we do only that. I thought of updating both to have consistency. Anyway, I'm right now doing a different way to not save only history length but also to persist other parts