CAMPUS GARDEN2 Dataset wrong Evaluation Results

[Mr-Akbari]

Hello @chengche6230,

I have conducted an evaluation on the CAMPUS - GARDEN2 dataset on Website and rename file in dataset followed format:

view-HC1.txt -> view-HC0.txt
view-HC2.txt -> view-HC1.txt
view-HC3.txt -> view-HC2.txt
view-HC4.txt -> view-HC3.txt

with the following configuration:

MODEL:
    DEVICE: "cuda"
    DEVICE_ID: ('0')
    MODE: 'test'
    DETECTION: 'gt' # {'gt'}
    RESUME: false
    LAST_CKPT_FILE: './'

DATASET:
    DIR: './datasets/'
    NAME: 'CAMPUS'
    SEQUENCE: ['Garden2'] # {'Garden1' | 'Garden2' | 'Parkinglot'}
    CAMS: 4
    TOTAL_FRAMES: 6000 # 'Garden1':2849, 'Garden2': 6000, 'Parkinglot': 6475

FE: # Feature Extractor
    CHOICE: 'CNN'
    INPUT_SIZE: (256, 128)

SOLVER:
    TYPE: 'TG' # {'SG': Spatial Graph, 'TG': Temporal Graph}
    EPOCHS: 100
    EVAL_EPOCH: 2
    BATCH_SIZE: 100
    LR: 0.01 # Learning Rate
    MAX_PASSING_STEPS: 4
    W: 3 # Temporal Window Size
    W_TEST: 5
    FOCAL_ALPHA: 0.98
    FOCAL_GAMMA: 5

OUTPUT:
    VISUALIZE: False
    LOG: True
    CKPT_DIR: './logs/ckpts'
    INFERENCE_DIR: './logs/inference'

TEST:
    CKPT_FILE_SG: './Model Zoo/CAMPUS_Garden2_SG_epoch12_train80.pth'
    CKPT_FILE_TG: './Model Zoo/CAMPUS_Garden2_TG_epoch6_train80.pth'
    FRAME_START: 4800 # 'Garden1': 2280; 'Garden2': 4800; 'Parkinglot': 5828
    EDGE_THRESH: 0.9

I obtained the following results:

can you help me?

[lvmlvm]

Hi. It seems like the renaming of the videos was not done sequentially, leading to each video being associated with the wrong homography matrix. From trial and error, I was able to get more acceptable IDF1 scores on CAMPUS's Parkinglot sequence by suffling the homography matrices in the metainfo.json files. That said, the paper only provided IDF1 scores for the Wildtrack dataset so it is unknown how the expected IDF1 scores should be.

@chengche6230 please provide us with info on how we should rename the videos in CAMPUS so that the homography matrices are correctly associated with the different camera views.