Support for LLaVA-format VLM training and release plans for DexboticVLM (weights/code/datasets)

[EmberNoWeither]

Hello! Thank you for your great work! I’m wondering whether this project also supports training VLMs in the LLaVA format. Additionally, your paper mentions training the DexboticVLM model—do you have any plans to release the DexboticVLM model weights, as well as the complete code for VLM pretraining, the full pretraining pipeline from VLM to VLA, and possibly the related datasets?

[yuzhms]

Hi, thanks for your interest.

Here are answers to your questions:

1. LLaVA format support: Yes, we already support LLaVA-style data. You can use the conversations key in our episode data format. Please refer to the documentation here:
   https://github.com/dexmal/dexbotic/blob/main/docs/Data.md#episode-data-format

2. DexboticVLM weights: At the moment, we do not have plans to release the DexboticVLM model weights.

3. VLM training code: The current codebase already supports VLM training. You can start from the base experiment setup here:
   https://github.com/dexmal/dexbotic/blob/main/dexbotic/exp/base_exp.py This can serve as a reference for building the VLM stage and further extending it toward VLA training.