Temporal Localization of Moments in Video Collections with Natural Language

Introduction

TODO: layman explanation of the content in the repo

In case you find this work relevant for your research please cite

TODO

Getting started

1. Install all the required dependencies:

   The main requirements of this project are Python==3.6, numpy, matplotlib, jupyter, pytorch and unity-agents. To ease its installation, I recommend the following procedure:

   • Install miniconda.

     Feel free to skip this step, if you already have anaconda or miniconda installed in your machine.

   • Creating the environment.

     conda env create -n moments-retrieval-devel -f environment-devel.yml

   • Activate the environment

     conda activate moments-retrieval-devel

2. Download data

   A snapshot of the processed data ready to train new models is available here

   • Download it and unzip it. You should see a single directory called data.

     Let's assume that you place this folder in [path]/data.

   • Copy it into the data folder of the repo.

     cd moments-retrieval
     `cp -r [path]/data .`

     Please remember to replace [path] with the actual folder of the downloaded data in your machine.

   TODO: write a bash script to do this.

Instructions

Training a new model

• The following line will launch the training of CAL in Charades-STA dataset.

  dataset_dir=data/processed/charades-sta
  parameters="--arch SMCN --feat rgb --train-list $dataset_dir/train.json --val-list $dataset_dir/val-01.json --test-list $dataset_dir/test.json --h5-path $dataset_dir/rgb_resnet152_max_cs-3.h5"
  python train.py --gpu-id 0 $parameters

  In case you want to save the logs into a file, add --logfile [filename]. For example:

  dataset_dir=data/processed/charades-sta
  parameters="--logfile 1 --arch SMCN --feat rgb --train-list $dataset_dir/train.json --val-list $dataset_dir/val-01.json --test-list $dataset_dir/test.json --h5-path $dataset_dir/rgb_resnet152_max_cs-3.h5"
  python train.py --gpu-id 0 $parameters

  This time we write all the verbosity into the file 1.log. Moreover, a model snapshot and the configuration setup are serialized into 1.pth.tar and 1.json, respectively.

  In case you close the terminal, don't forget to activate the environment (conda activate moments-retrieval).

TODO: corpus video retrieval evaluation

TODO: single video retrieval evaluation

TODO: dashboards

Do you like the project?

Please give us ⭐️ in the GitHub banner 😉. We are also open for discussions especially accompany with ☕,🍺 or 🍸 (preferably spritz).

LICENSE

MIT

We highly appreciate that you leave attribution notes when you copy portions of our codebase in yours.

---

Usage

Below you will find how to use the programs that comes with our codebase.

Training

Training a model

data_dir=data/processed/didemo
parameters="--proposal-interface DidemoICCV17SS --arch SMCN --feat rgb --train-list $data_dir/train-03.json --val-list $data_dir/train-03_01.json --test-list $data_dir/val-03.json --h5-path $data_dir/resnet152_rgb_max_cl-2.5.h5 --nms-threshold 1"
python -m ipdb train.py --gpu-id 0 $parameters --debug --epochs 1 --h5-path-nis workers/tyler/data/interim/smcn_40/b/train/1_corpus-eval.h5 --num-workers 0 --snapshot workers/tyler/data/interim/smcn_40/b/1.json

Corpus retrieval

Any corpus retrieval experiment required a pre-trained model.

Exhaustive search with single stage

python corpus_retrieval_eval.py --test-list data/processed/activitynet-captions/val.json --h5-path data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot workers/skynet-base/data/interim/mcn_43/b/1.json --dump-per-instance-results --dump

Two-stage corpus retrieval

python corpus_retrieval_2nd_eval.py --test-list data/processed/didemo/test-03.json --h5-path data/processed/didemo/resnet152_rgb_max_cl-5.h5 --snapshot workers/ibex-scratch/data/interim/mcn_41/a/1.json --h5-1ststage workers/ibex-scratch/data/interim/mcn_41/b/1_corpus-eval.h5 --k-first 200 --nms-threshold 1.0 --debug

Approximated setup 2

Evaluating model, CAL + CAL-TEF:

python corpus_retrieval_2nd_eval.py --corpus-setup TwoStageClipPlusGeneric --test-list data/processed/didemo/test-03.json --h5-path data/processed/didemo/resnet152_rgb_max_cl-2.5.h5 --snapshot workers/tyler/data/interim/smcn_40/a/1.json --h5-1ststage data/processed/didemo/resnet152_rgb_max_cl-2.5.h5 --snapshot-1ststage workers/tyler/data/interim/smcn_40/b/1.json --k-first 200

• Dumping data for re-training

  1. Apply the patch

     git apply scripts/patches/dump-clip-retrieval-results

     For ActivityNet, it's better use this or have enough storage

     git apply scripts/patches/dump-clip-retrieval-results-activitynet

  2. Launch

     python corpus_retrieval_eval.py --test-list data/processed/didemo/  train-03.json --h5-path data/processed/didemo/resnet152_rgb_max_cl-2.5.h5   --snapshot data/interim/smcn_40/b/4.json --clip-based   --dump-per-instance-results

     python corpus_retrieval_eval.py --test-list data/processed/charades-sta/  train-01.json --h5-path data/processed/charades-sta/  resnet152_rgb_max_cl-3.h5 --snapshot data/interim/smcn_42/b/3.json   --clip-based --dump-per-instance-results

     python corpus_retrieval_eval.py --test-list data/processed/activitynet-captions/train.json --h5-path data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot data/interim/smcn_43/b/3.json --clip-based --dump-per-instance-results

• Re-training

  1. git checkout eb7f2d73bd20bb4fbc22a8d2f4dc003807248cef

  2. Update bash script used for smcn_[47-48].

• Launch re-trained models

python corpus_retrieval_2nd_eval.py --corpus-setup TwoStageClipPlusGeneric --test-list data/processed/didemo/test-03.json --h5-path data/processed/didemo/resnet152_rgb_max_cl-2.5.h5 --snapshot data/interim/smcn_50/b/1.json --h5-1ststage data/processed/didemo/resnet152_rgb_max_cl-2.5.h5 --snapshot-1ststage data/interim/smcn_40/b/4.json --k-first 200 --dump --output-prefix cr-msm_approx-smcn-40b_nms-1

python corpus_retrieval_2nd_eval.py --corpus-setup TwoStageClipPlusGeneric --test-list data/processed/charades-sta/test-01.json --h5-path data/processed/charades-sta/resnet152_rgb_max_cl-3.h5 --snapshot data/interim/smcn_51/a/1.json --h5-1ststage data/processed/charades-sta/resnet152_rgb_max_cl-3.h5 --snapshot-1ststage data/interim/smcn_42/b/3.json --k-first 200 --dump --output-prefix cr-msm_approx-smcn-42b_nms-1

python corpus_retrieval_2nd_eval.py --corpus-setup TwoStageClipPlusGeneric --test-list data/processed/activitynet-captions/val.json --h5-path data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot data/interim/smcn_52/a/1.json --h5-1ststage data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot-1ststage data/interim/smcn_43/a/3.json --k-first 200 --dump --output-prefix cr-msm_approx-smcn-43b_nms-1

Aproximate setup fast

CAL - MCN

python corpus_retrieval_2nd_eval.py --corpus-setup TwoStageClipPlusMCNFast --test-list data/processed/activitynet-captions/val.json --h5-path data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot workers/tyler/data/interim/mcn_43/a/3.json --h5-1ststage data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot-1ststage workers/tyler/data/interim/smcn_43/b/3.json --k-first 200

CAL - CAL

python -m ipdb corpus_retrieval_2nd_eval.py --corpus-setup TwoStageClipPlusCALFast --test-list data/processed/activitynet-captions/val.json --h5-path data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot data/interim/smcn_43/a/3.json --h5-1ststage data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot-1ststage data/interim/smcn_43/b/3.json --k-first 200 --debug

Debugging fast-retrieval branch

python -m ipdb corpus_retrieval_2nd_eval.py --corpus-setup TwoStageClipPlusCALFast --test-list data/processed/activitynet-captions/val.json --h5-path data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot data/interim/smcn_49/a/1.json --h5-1ststage data/processed/activitynet-captions/resnet152_rgb_max_cl-5.h5 --snapshot-1ststage data/interim/smcn_43/b/3.json --k-first 200 --output-prefix replicate-fast

Baselines for moment retrieval on a video corpus

Example on Charades-STA

python mfp_corpus_eval.py \
    --test-list data/processed/charades-sta/test-01.json \
    --snapshot data/processed/charades-sta/mfp/1.json \
    --logfile data/processed/charades-sta/mfp/1_corpus-eval \
    --dump --topk 1 10 100 \

Example on ActivityNet-Captions

python mfp_corpus_eval.py \
    --test-list data/processed/activitynet-captions/val.json \
    --snapshot data/processed/activitynet-captions/mfp/1.json \
    --logfile  data/processed/activitynet-captions/mfp/1_corpus-eval \
    --topk 1 10 100 --dump

To compute chance supply the --chance to the commands above.

(optional) video retrieval evaluation

python eval_video_retrieval.py --test-list data/processed/activitynet-captions/val.json --snapshot data/processed/3rdparty/mee_video-retrieval_activitynet-captions_val.h5 --h5-1ststage data/processed/3rdparty/mee_video-retrieval_activitynet-captions_val.h5 --topk 1 10 100 1000 --dump

Single video moment retrieval

DiDeMo evaluation

python single_video_retrieval_didemo.py --val-list data/processed/didemo/val-03.json --test-list data/processed/didemo/test-03.json --h5-path data/processed/didemo/resnet152_rgb_max_cl-2.5.h5 --snapshot workers/tyler/data/interim/mcn_41/a/1.json --dump

Moment frequecy prior for moment retrieval on single video

Example Charades-STA:

python moment_freq_prior.py \
    --train-list data/processed/charades-sta/train-01.json \
    --test-list data/processed/charades-sta/test-01.json \
    --logfile data/processed/charades-sta/mfp/1
    --clip-length 3 --bins 75 \
    --proposal-interface SlidingWindowMSRSS \
    --min-length 3 --scales 2 3 4 5 6 7 8 \
    --stride 0.3 --nms-threshold 0.6 \

Example ActivityNet-Captions:

python moment_freq_prior.py \
    --train-list data/processed/activitynet-captions/train.json \
    --test-list data/processed/activitynet-captions/val.json \
    --logfile data/processed/activitynet-captions/mfp/1
    --clip-length 5 --bins 500 \
    --proposal-interface SlidingWindowMSRSS \
    --min-length 5 --scales 2 4 6 8 10 12 14 16 18 20 22 24 26 \
    --stride 0.3 --nms-threshold 0.6 \

Remove the logfile if you prefer to to print into shell and copy-paste results.

Debug CAL-Chamfer

data_dir=data/processed/didemo parameters="--test-list $dataset_dir/test-03.json --h5-path $dataset_dir/resnet152_rgb_max_cl-5.h5 --corpus-setup LoopOverKMoments --snapshot data/interim/calchamfer_deploy/DIDEMO/whole_trainingset/ModelD_TEF_5s_7.json --k-first 10 --h5-1ststage workers/tyler/data/interim/smcn_40/b/test/1_corpus-eval.h5" python corpus_retrieval_2nd_eval.py $parameters --k-first 1 --debug