Dataset Preparation Script

[mmderakhshani]

Hi, thanks for this excellent work.

Do you mind if I ask you to release the dataset preparation part, such as downloading the dataset and recaptioning it using ShareGPT4v? I could not find them in the repo.

Thanks.

[Sierkinhane]

Hi, unfortunately, the datasets are not prepared by ourselves. You can try this download tool to download large-scale datasets as tars, which can be directly loaded using our code.

We do use ShareGPT4V to recaption the data and the prompt is Analyze the image in a comprehensive and detailed manner.

[mmderakhshani]

Thanks for replying back indeed! I knew this repo as it is trying to provide dataset in webdataset format. However, when I was checking the project config file, I have noticed that ‘laion-auestics-12m’ is used during pretraining. When googling, I could not find this dataset and that is why I have asked for it because without this data we cannot reproduce your experiments. Thanks!

[Sierkinhane]

Hi, laion-aesthetics-12m is available here https://huggingface.co/datasets/dclure/laion-aesthetics-12m-umap