Bugs in the evaluation code

[sashamaru]

For the baseline methods, the results are stored using a dictionary with the key (method, iters). However, the if condition that updates these results is based solely on the iteration count (iters). As a result, for each batch, the results are reassigned as an empty list, causing only the results from the last batch to be retained. This is the primary reason why all the baseline methods perform poorly.

superansac/tests/essential_matrix/tester_baseline.py

Lines 377 to 382 in e2c5214

	for iters in [10, 25, 50, 100, 250, 500, 750, 1000, 1500, 2500, 5000, 7500, 10000]:
	key = (method, iters)
	if iters not in pose_errors:
	pose_errors[key] = []
	runtimes[key] = []
	inlier_numbers[key] = []

In superansac, it's correct.

superansac/tests/essential_matrix/tester_superansac.py

Lines 224 to 229 in e2c5214

	for iters in [10, 25, 50, 100, 250, 500, 750, 1000, 1500, 2500, 5000, 7500, 10000]:
	key = iters
	if iters not in pose_errors:
	pose_errors[key] = []
	runtimes[key] = []
	inlier_numbers[key] = []

In my experiment, I avoided this issue because my implementation uses only the iteration count (iters) as the key.

You should address this bug, otherwise, others running the evaluation may obtain results similar to those in your paper, where the code is incorrect. Consequently, all the reported results in the paper need to be corrected.

[danini]

Good catch, thanks! I will check it out and fix it.

[sashamaru]

Many codes have been updated, but why does the baseline evaluation code still contain errors?

superansac/tests/essential_matrix/tester_baseline.py

Lines 376 to 379 in c3cbdbe

	for method in ["poselib", "vsac", "MAGSAC OpenCV", "GCRANSAC OpenCV", "RANSAC OpenCV", "LMEDS OpenCV", "gcransac", "magsac", "magsac++", "pycolmap"]:
	for iters in [10, 25, 50, 100, 250, 500, 750, 1000, 1500, 2500, 5000, 7500, 10000]:
	key = (method, iters)
	if iters not in pose_errors:

[danini]

Updated!