MARS: Motion-Augmented RGB Stream for Action Recognition

By Nieves Crasto, Philippe Weinzaepfel, Karteek Alahari and Cordelia Schmid

MARS is a strategy to learn a stream that takes only RGB frames as input but leverages both appearance and motion information from them. This is achieved by training a network to minimize the loss between its features and the Flow stream, along with the cross entropy loss for recognition. For more details, please refer to our CVPR 2019 paper and our website.

We release the testing code along trained models.

Citing MARS

@inproceedings{crasto2019mars,
  title={{MARS: Motion-Augmented RGB Stream for Action Recognition}},
  author={Crasto, Nieves and Weinzaepfel, Philippe and Alahari, Karteek and Schmid, Cordelia},
  booktitle={CVPR},
  year={2019}
}

Contents

1. Requirements
2. Datasets
3. Models
4. Testing

Requirements

• Python3
• Pytorch 1.0

conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

• ffmpeg version 3.2.4

• OpenCV with GPU support (will not be providing support in compiling this part)

• Directory tree

   dataset/
       HMDB51/ 
           ../(dirs of class names)
               ../(dirs of video names)
       HMDB51_labels/
   results/
       test.txt
   trained_models/
       HMDB51/
           ../(.pth files)

Datasets

• The datsets and splits can be downloaded from

  Kinetics400

  UCF101

  HMDB51

  SomethingSomethingv1

• To extract only frames from videos

python utils1/extract_frames.py path_to_video_files path_to_extracted_frames start_class end_class

• To extract optical flows + frames from videos
  • Build

  export OPENCV=path_where_opencv_is_installed
  
  g++ -std=c++11 tvl1_videoframes.cpp -o tvl1_videoframes -I${OPENCV}include/opencv4/ -L${OPENCV}lib64 -lopencv_objdetect -lopencv_features2d -lopencv_imgproc -lopencv_highgui -lopencv_core -lopencv_imgcodecs -lopencv_cudaoptflow -lopencv_cudaarithm
  
  python utils1/extract_frames_flows.py path_to_video_files path_to_extracted_flows_frames start_class end_class gpu_id
  

Models

Trained models can be found here. The names of the models are in the form of

stream_dataset_frames.pth     

RGB_Kinetics_16f.pth indicates --modality RGB --dataset Kinetics --sample_duration 16

For HMDB51 and UCF101, we have only provided trained models for the first split.

Testing script

For RGB stream:

python test_single_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB --sample_duration 16 --split 1 --only_RGB  \
--resume_path1 "trained_models/HMDB51/RGB_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"

For Flow stream:

python test_single_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality Flow --sample_duration 16 --split 1  \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"

For single stream MARS:

python test_single_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB --sample_duration 16 --split 1 --only_RGB  \
--resume_path1 "trained_models/HMDB51/MARS_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"

For two streams RGB+MARS:

python test_two_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB --sample_duration 16 --split 1 --only_RGB  \
--resume_path1 "trained_models/HMDB51/RGB_HMDB51_16f.pth" \
--resume_path2 "trained_models/HMDB51/MARS_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"

For two streams RGB+Flow:

python test_two_stream.py --batch_size 1 --n_classes 51 --model resnext --model_depth 101 \
--log 0 --dataset HMDB51 --modality RGB_Flow --sample_duration 16 --split 1 \
--resume_path1 "trained_models/HMDB51/RGB_HMDB51_16f.pth" \
--resume_path2 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--frame_dir "dataset/HMDB51/HMDB51_frames/" \
--annotation_path "dataset/HMDB51_labels" \
--result_path "results/"

Training script

For RGB stream:

From scratch:

 python train.py --dataset Kinetics --modality RGB --only_RGB \
--n_classes 400 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101  \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--result_path "results/"

From pretrained Kinetics400:

 python train.py --dataset HMDB51 --modality RGB --split 1 --only_RGB \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/RGB_Kinetics_16f.pth" \
--result_path "results/"

From checkpoint:

 python train.py --dataset HMDB51 --modality RGB --split 1 --only_RGB \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/RGB_Kinetics_16f.pth" \
--resume_path1 "results/HMDB51/PreKin_HMDB51_1_RGB_train_batch32_sample112_clip16_nestFalse_damp0.9_weight_decay1e-05_manualseed1_modelresnext101_ftbeginidx4_varLR2.pth" \
--result_path "results/"

For Flow stream

From scratch:

 python train.py --dataset Kinetics --modality Flow \
--n_classes 400 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101  \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--result_path "results/"

From pretrained Kinetics400:

 python train.py --dataset HMDB51 --modality Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--result_path "results/"

From checkpoint:

 python train.py --dataset HMDB51 --modality Flow --split 1 \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 32 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--resume_path1 "results/HMDB51/PreKin_HMDB51_1_Flow_train_batch32_sample112_clip16_nestFalse_damp0.9_weight_decay1e-05_manualseed1_modelresnext101_ftbeginidx4_varLR2.pth" \
--result_path "results/"

For MARS:

From scratch:

python MARS_train.py --dataset Kinetics --modality RGB_Flow \
--n_classes 400 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--resume_path1 "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--result_path "results/" --checkpoint 1

From pretrained Kinetics400:

python MARS_train.py --dataset HMDB51 --modality RGB_Flow --split 1  \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MARS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--result_path "results/" --checkpoint 1

From checkpoint:

python MARS_train.py --dataset HMDB51 --modality RGB_Flow --split 1  \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MARS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--MARS_resume_path "results/HMDB51/MARS_HMDB51_1_train_batch16_sample112_clip16_lr0.001_nesterovFalse_manualseed1_modelresnext101_ftbeginidx4_layeravgpool_alpha50.0_1.pth" \
--result_path "results/" --checkpoint 1

For MERS:

From scratch:

python MERS_train.py --dataset Kinetics --modality RGB_Flow \
--n_classes 400 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/Kinetics" \
--annotation_path "dataset/Kinetics_labels" \
--resume_path1 "trained_models/Kinetics/Flow_Kinetics_16f.pth" \
--result_path "results/" --checkpoint 1

From pretrained Kinetics400:

python MERS_train.py --dataset HMDB51 --modality RGB_Flow --split 1  \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MERS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--result_path "results/" --checkpoint 1

From checkpoint:

python MERS_train.py --dataset HMDB51 --modality RGB_Flow --split 1  \
--n_classes 400 --n_finetune_classes 51 \
--batch_size 16 --log 1 --sample_duration 16 \
--model resnext --model_depth 101 --ft_begin_index 4 \
--output_layers 'avgpool' --MARS_alpha 50 \
--frame_dir "dataset/HMDB51" \
--annotation_path "dataset/HMDB51_labels" \
--pretrain_path "trained_models/Kinetics/MARS_Kinetics_16f.pth" \
--resume_path1 "trained_models/HMDB51/Flow_HMDB51_16f.pth" \
--MARS_resume_path "results/HMDB51/MERS_HMDB51_1_train_batch16_sample112_clip16_lr0.001_nesterovFalse_manualseed1_modelresnext101_ftbeginidx4_layeravgpool_alpha50.0_1.pth" \
--result_path "results/" --checkpoint 1