Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding [Paper][Project Page]
Rohan Myer Krishnan*, Zitian Tang*, Zhiqiu Yu, Chen Sun (*Equal Contribution)

1. Clone this repository

git clone https://github.com/brown-palm/Spacewalk-18-benchmark.git
cd Spacewalk-18-benchmark

2. Download videos

cd data
python download_videos.py

The videos are downloaded into data/videos. There are in total 18 videos and each one is the recording of a spacewalk mission. All the videos are at 29.97 FPS with frame size 720 x 1280.

The transcripts of the videos are in data/transcripts. They contain the global start and end times of each sentence in seconds. Please use encoding='cp1252' when reading them.

At the beginning of each video, an animation demonstrates the mission to be done by the astronauts. Based off the animation, the mission is devided into dozens of steps, which are listed in data/steps.json.

We annotate the spacewalk recordings by segmenting them into the steps. The annotations are in data/annotation.json. Note that one step may occur multiple times and the steps may appear interwoven, because there are two astronauts working in parallel in a mission.

We have two tasks on Spacewalk-18 benchmark - step recognition and intra-video retrieval. They share the same training, validation, and test video splits, which are listed in tasks/split.json.

Given a query timestamp $t$ in a video, step recognition aims to determine the step happening at $t$. The training, validation, and test samples are in tasks/step_recognition/*.json. We show in construct_samples.ipynb how they are constructed. You may use it to generate more training samples if they benefit your method. Moreover, we encourage you to go beyond these samples and fully use the training set videos to develop your model.

Our tasks enables model evaluation under varying temporal context window lengths $w$. In this setting, a model needs to determine frame $t$'s corresponding step given the video clip $[t-w/2, t+w/2]$. We provide the code to calculate the video start and end times and collect the transcripts in the clip for any specified context length $w$. See construct_context_window.ipynb.

We use accuracy and mAP to evaluate the performance on the step recognition task. The code is in evaluation.py. To use it, please organize your predictions in a JSON file in the following format:

{
      "03152022": [
            {
                  "prediction": 1,
                  "score": [0.4, 0.8, ..., 0.2]
            },
            {
                  "prediction": 0,
                  "score": [0.9, 0.3, ..., 0.6]
            },
            ... ...
      ],
      "11152022": [
            ... ...
      ]
}

The $i$-th item under each video ID is your prediction to the $i$-th sample. prediction is a predicted label used to compute accuracy. score is a list of scores for each category used to compute mAP. The length of score should be the same as the number of categories in the corresponding video. If your method does not provide category scores, you can omit score in your file.

You can use --gt to specify the ground truth file and --pred to specify your prediction file. For example, to evaluate your predictions on the validation set, run

python evaluation.py \
--gt val.json \
--pred YOUR_PREDICTION_FILE.json

Given a query timestamp $t_q$ and two candidate timestamps $t_1$ and $t_2$, intra-video retrieval aims to determine which candidate comes from the same step as the query. The training, validation, and test samples are in tasks/intra-video_retrieval/*.json. We show in construct_samples.ipynb how they are constructed. Again, we encourage you to go beyond the provided training samples to develop your model.

With a temporal context window length $w$, the model can access to video clips $[t_q-w/2, t_q+w/2]$, $[t_1-w/2, t_1+w/2]$, and $[t_2-w/2, t_2+w/2]$. We provide the code to calculate the video start and end times and collect the transcripts for any specified context length $w$. See construct_context_window.ipynb.

We use accuracy evaluate the performance on the intra-video retrieval task. The code is in evaluation.py. Please organize your predictions in a JSON file in the following format:

{
      "03152022": [1, 0, 1, 1, ..., 1],
      "11152022": [1, 0, 0, 1, ..., 0],
      ... ...
}

The list under each video ID is your predictions to the samples. Their values should be either 0 or 1.

You can use --gt to specify the ground truth file and --pred to specify your prediction file. For example, to evaluate your predictions on the validation set, run

python evaluation.py \
--gt val.json \
--pred YOUR_PREDICTION_FILE.json

We will be happy if you find Spacewalk-18 useful. Please cite it using this BibTeX:

@misc{krishnan2024spacewalk18,
      title={Spacewalk-18: A Benchmark for Multimodal and Long-form Procedural Video Understanding}, 
      author={Rohan Myer Krishnan and Zitian Tang and Zhiqiu Yu and Chen Sun},
      year={2024},
      publisher={arXiv:2311.18773}
}

Spacewalk-18 benchmark is released under MIT License.