Misalignment between audio and video frames

[zailongchen]

Thank you so much for your fantastic work!
I found a few videos shorter than 60s in your dataset. When using your frame extraction script to extract frames from a video in the 1fps manner, I could not get 60 frames, however, the shape of the corresponding audio feature was [60, 128] in vggish folder.
It would be so grateful if you let me know how to align the audio and frames from the same video.

[ayameyao]

Thank you very much for your interest in our work.

In the MUSIC-AVQA dataset, there are few videos whose length is shorter than 60 seconds. The way we process this part of the video is as follows:

1. Extract video frames: Copy the content of the last second multiple times until its length reaches 60 seconds (we recommend executing the extract_frames_adaptive_script.py file, and some explanations have been updated);
2. Extract audio features: The value of the audio length parameter T in the code is set to 60, and its specific operation is the same as 1), that is, copy the audio of the last second until its length reaches 60 seconds, and then extract its audio features.

Hope the above replies solve your problem!

Thank you again for your interest in our work! If you have other questions and suggestions, please do not hesitate to contact me.