Reproduce the results

[chaneyddtt]

Hi @markendo, thanks a lot for the interesting work. I have been trying to reproduce the results reported in Table 1. I train the model from scratch following the training command, but get an accuracy of 0.53. May I know whether there is additional steps or parameters I need to set to get the reported performance of 0.578? Thanks in advance for your help.

[ishantamrakar]

Hi @markendo. I also got lower overall accuracy reporting (0.526718) when I ran the evaluation script. It is worth noting that I did run the code on CPU only mode and pytorch 1.11.0 with python 3.8. Not sure the precision and/or random number generation process contributed to the difference. Could you provide the complete evaluation configuration (hyper parameters, hardware), if you remember?

[markendo]

Hello, sorry for the delay! Here is the complete info for the conda environment used for this project. Note I had to add export SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True as the sklearn install is now deprecated.

channels:
  - defaults
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - ca-certificates=2023.01.10=h06a4308_0
  - certifi=2022.12.7=py37h06a4308_0
  - ld_impl_linux-64=2.38=h1181459_1
  - libffi=3.3=he6710b0_2
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - ncurses=6.4=h6a678d5_0
  - openssl=1.1.1t=h7f8727e_0
  - pip=22.3.1=py37h06a4308_0
  - python=3.7.11=h12debd9_0
  - readline=8.2=h5eee18b_0
  - setuptools=65.6.3=py37h06a4308_0
  - sqlite=3.41.1=h5eee18b_0
  - tk=8.6.12=h1ccaba5_0
  - wheel=0.38.4=py37h06a4308_0
  - xz=5.2.10=h5eee18b_1
  - zlib=1.2.13=h5eee18b_0
  - pip:
    - absl-py==1.4.0
    - astunparse==1.6.3
    - backcall==0.2.0
    - brotlipy==0.7.0
    - cachetools==5.3.0
    - cffi==1.15.1
    - charset-normalizer==2.0.4
    - click==8.0.3
    - cryptography==39.0.1
    - cycler==0.11.0
    - cython==0.29.33
    - decorator==5.1.1
    - dpcpp-cpp-rt==2023.0.0
    - editdistpy==0.1.3
    - flatbuffers==2.0.7
    - flit-core==3.6.0
    - fonttools==4.38.0
    - gast==0.4.0
    - google-auth==2.16.2
    - google-auth-oauthlib==0.4.6
    - google-pasta==0.2.0
    - grpcio==1.51.3
    - h5py==3.8.0
    - idna==3.4
    - importlib-metadata==6.0.0
    - intel-cmplr-lib-rt==2023.0.0
    - intel-cmplr-lic-rt==2023.0.0
    - intel-opencl-rt==2023.0.0
    - intel-openmp==2023.0.0
    - ipdb==0.13.11
    - ipython==7.34.0
    - jacinle==1.0.2
    - jedi==0.18.2
    - joblib==1.2.0
    - keras==2.7.0
    - keras-preprocessing==1.1.2
    - kiwisolver==1.4.4
    - lark==1.1.3
    - libclang==15.0.6.1
    - markdown==3.4.1
    - markupsafe==2.1.2
    - matplotlib==3.5.3
    - matplotlib-inline==0.1.6
    - mkl==2023.1.0
    - mkl-fft==1.3.1
    - mkl-random==1.2.2
    - mkl-service==2.4.0
    - nltk==3.6.5
    - numpy==1.21.6
    - oauthlib==3.2.2
    - opt-einsum==3.3.0
    - packaging==23.0
    - pandas==1.3.5
    - parso==0.8.3
    - peewee==3.14.8
    - pexpect==4.8.0
    - pickleshare==0.7.5
    - pillow==9.4.0
    - pkgconfig==1.5.5
    - plotly==5.13.1
    - plyfile==0.7.4
    - prompt-toolkit==3.0.38
    - protobuf==3.20.3
    - ptyprocess==0.7.0
    - pyasn1==0.4.8
    - pyasn1-modules==0.2.8
    - pycparser==2.21
    - pygments==2.14.0
    - pyopenssl==23.0.0
    - pyparsing==3.0.9
    - pysocks==1.7.1
    - python-dateutil==2.8.2
    - pytz==2022.7.1
    - pyyaml==6.0
    - pyzmq==25.0.0
    - regex==2022.10.31
    - requests==2.28.1
    - requests-oauthlib==1.3.1
    - rsa==4.9
    - scikit-learn==1.0.2
    - scipy==1.7.3
    - shapely==2.0.1
    - six==1.16.0
    - sklearn==0.0.post1
    - smplx==0.1.28
    - symspellpy==6.7.7
    - tabulate==0.9.0
    - tbb==2021.9.0
    - tenacity==8.2.2
    - tensorboard==2.11.2
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.1
    - tensorboardx==2.6
    - tensorflow==2.7.0
    - tensorflow-estimator==2.7.0
    - tensorflow-io-gcs-filesystem==0.31.0
    - termcolor==2.2.0
    - threadpoolctl==3.1.0
    - tomli==2.0.1
    - torch==1.11.0
    - torchaudio==0.11.0
    - torchvision==0.12.0
    - tqdm==4.64.1
    - traitlets==5.9.0
    - typing-extensions==4.5.0
    - urllib3==1.26.14
    - wcwidth==0.2.6
    - werkzeug==2.2.3
    - wrapt==1.15.0
    - zipp==3.15.0

For the Jacinle package, I am using commit 262d477dd793c107245846e4d589e74374ee2967.

I just rebuilt the conda environment and get the following when running with a titan rtx:

jac-crun 0 nspose/trainval.py \
--desc nspose/desc_nspose.py \
--evaluate \
--load /path/to/f_45_o_15_conv1d.pth \
--expr test_f_45_o_15_conv1d \
--data-dir /path/to/BABEL-QA/ \
--batch-size 4 \
--temporal_operator conv1d \
--no_gt_segments

test/acc/qa = 0.587786
test/acc/query_action = 0.633333
test/acc/query_action_after = 0.647887
test/acc/query_action_before = 0.629032
test/acc/query_action_in_between = 0.666667
test/acc/query_action_no_relation = 0.703333
test/acc/query_body_part = 0.320513
test/acc/query_body_part_after = 0.529412
test/acc/query_body_part_before = 0.305556
test/acc/query_body_part_no_relation = 0.142857
test/acc/query_direction = 0.602504
test/acc/query_direction_after = 0.607143
test/acc/query_direction_before = 0.416667
test/acc/query_direction_in_between = 0.750000
test/acc/query_direction_no_relation = 0.586777
test/filter_action = 0.524384
test/filter_body_part = 0.536232
test/filter_direction = 0.531566
time/data = 0.012975
time/step = 0.065634

Note that the exact number from the paper (0.578) is an averaged score from 3 different runs. I hope this helps!