Sequential memory sampling

[elle-miller]

Description

Hi,

In the PPO implementation, the update uses minibatches from sample_all function of Memory which are sequentially sampled e.g. [1,2,3,4,5,6......].

https://github.com/Toni-SM/skrl/blob/main/skrl/memories/torch/base.py#L327C9-L327C19

Was this intentional? I thought the sampling should be random, to avoid temporal correlations in the data e.g. point 6

I'm seeing minor improvements in my particular environment, but have not run conclusive multiseed tests.

if random:
     indexes = RandomSampler(indexes)

batches = BatchSampler(indexes, batch_size=len(indexes) // mini_batches, drop_last=True)

Just a heads up!

What skrl version are you using?

---

What ML framework/library version are you using?

No response

Additional system information

No response

[r2xd889zyn-dot]

Hi, quick question regarding PPO minibatch sampling. In the current implementation, sample_all() returns sequential indices which are then passed directly to BatchSampler, resulting in temporally ordered minibatches (e.g., [1,2,3,4,...]).

Was this intentional? I was expecting the indices to be shuffled prior to minibatching to reduce temporal correlations, as is common in many PPO implementations. I experimented with wrapping the indices in a RandomSampler before batching and observed minor improvements in my environment (not yet validated with multi-seed runs).

Just wanted to flag this and ask if the sequential sampling is by design or if randomization is expected to be handled elsewhere.