libero-pi0 retraining cannot earn official points

[Yang-Jiahe]

I trained dexbotic/playground/benchmarks/libero/libero-pi0.py on the latest version of the dexbotic codebase using libero_pi0_all data. And tested with dexbotic-benchmark, the results are as follows:

db_pi0_all_ft paper | 97 | 98.2 | 94 | 86.4 | 93.9
db_pi0_all_ft test | 96.6 | 95.6 | 93.4 | 87.2 | 93.2
my_db_pi0_all_ft | 93.4 | 96.8 | 86.6 | 73.2 | 87.5

The "db_pi0_all_ft paper" is the official result provided. The "db_pi0_all_ft test" is the result I obtained using the official provided model (libero-pi0) for testing. The "my_db_pi0_all_ft" is the result of my retraining and testing as required.

It can be seen that there is a gap between the results and the official points, and the difference in the results obtained from retraining is even greater. My initial guess is that the reason is due to the use of a different code version, as I noticed that the model's config. json provided by the official source is different from the config. json I retrained. The retrained config. json has many additional fields on top of the original config. json, and I noticed that the transformer version is different.
Official model: "transformers_version": "4.38.0. dev0".
Current version (my model): "transformers_version": "4.51.0".
The official code pyproject.toml for PI0's transformer is 4.53.2.

Could this be the reason for this? The training parameter settings are completely consistent with the provided code, and the testing parameters are consistent with dexbotic-benchmark/evaluation/config/libero/sample_pi0ulibero.yaml.

I also conducted several experiments to change the replan_step parameter, but the results were inconsistent. 15 would be better than 5. Why is this

[zed0630]

Thanks for following our work.

While we have noticed some variance in LIBERO results between experiments, the discrepancy was not as significant as what you reported. We are going to re-run our experiments to investigate this further to see if we can reproduce the issue and identify the potential cause.

To help us further isolate the problem, we would greatly appreciate it if you could provide the following information:

1. Details of your training setup, specifically the hardware environment (e.g., GPU model and number of GPUs).
2. Results from a repeated training run to see if the issue is consistently reproducible.

[Yang-Jiahe]

I used h20x8 during training，4090x1 during testing
h20 Driver Version: 535.161.08 CUDA Version: 12.2. Ubuntu 20.04.6 LTS
4090 Driver Version: 535.154.05 CUDA Version: 12.2. Ubuntu 22.04.3 LTS
dexbotic Environment: Based on the Docker image dexmalhub-cn-beijing.cr.volces.com/dexbotic/training:0826
dexbotic-benchmark Environment: Based on the Docker image dexmalhub-cn-beijing.cr.volces.com/default/dexbotic/benchmark:latest
If I rely too much, I won't list it. I haven't made any modifications to these two environments

Both training and testing scripts follow the current code repository

training：I only made changes to save_steps and output_dir
bf16: bool = field(default=True)
num_train_steps: int = field(default=30000)
save_steps: int = field(default=5000)
per_device_train_batch_size: int = field(default=4)
gradient_accumulation_steps: int = field(default=1)
model_max_length: int = field(default=48)
lr_scheduler_type: str = field(default="cosine_with_min_lr")
lr_scheduler_kwargs: dict = field(default_factory=lambda: {"min_lr_rate": 0.1})

testing：Modified the model_name_or_path in Pi0InferenceConfig
dexbotic-benchmark/evaluation/configs/libero/example_pi0_libero.yaml：I only made modifications to base_url and output_dir
benchmark: libero_10
num_trails_per_task: 50
num_steps_wait: 10
seed: 7
replan_step: 5
send_state: true
send_image:

• image
• wrist_image
  discrete_gripper: false
  use_text_template: false

[zed0630]

Hi @Yang-Jiahe,

We’ve reviewed the information you provided—thank you very much for your help! It appears that your experiment configuration matches the latest version exactly, yet it fails to achieve the expected results.

We also recently conducted an internal reproducing experiment. On one hand, we did observe result fluctuations of less than 2%. We are currently attempting a more stable experimental setup to address this and plan to release this version once verification is complete. On the other hand, we are still unable to reproduce your specific results.

We would be very grateful if you could provide results from a repeated experiment.

[Yang-Jiahe]

Thank you for your experiment. However, the results from my repeated tests consistently fell below 90, and the success rate of the long suite remained at 70-75. I am considering whether I made a mistake in using the dataset. I would like to ask if you used all four suites from Dexmal/libero for training libero_pi0_all. If you could share this information, I would be extremely grateful

[polinazhang]

From a previous look at the codebase, my understanding was that libero_90 might not be included in the training data for libero_pi0_all (please correct me if I’m mistaken).