Skip to content

Comments

Apply custom thresholds and weights to Streamlit IQB score computation#160

Open
atulbhardwaj-io wants to merge 1 commit intom-lab:mainfrom
atulbhardwaj-io:fix/streamlit-custom-config-score
Open

Apply custom thresholds and weights to Streamlit IQB score computation#160
atulbhardwaj-io wants to merge 1 commit intom-lab:mainfrom
atulbhardwaj-io:fix/streamlit-custom-config-score

Conversation

@atulbhardwaj-io
Copy link

Summary

This PR fixes a correctness bug in the Streamlit prototype where edited thresholds/weights in the configuration editor were not applied to IQB score computation.

Why this was happening

The editor updated thresholds/weights in Streamlit session state, but score calculation paths still used the default calculator instance (state.iqb) initialized with default config.
So UI changes were visible, but the score logic ignored them.

Root cause

  • Prototype score paths called state.iqb.calculate_iqb_score(...) directly.
  • state.iqb used default config only.
  • IQBCalculator.set_config did not support in-memory dict config.
  • Custom-config builder existed but was not connected to score computation paths.

What changed

  • Added dict-config support in calculator: library/src/iqb/calculator.py
  • Added custom-settings score helper: prototype/utils/calculation_utils.py
  • Added calculate_iqb_score_with_custom_settings(...) helper usage in scoring paths.
  • Rewired score paths in prototype/Home.py and prototype/pages/IQB_Map.py.
  • Added regression tests in library/tests/iqb/calculator_test.py.
  • Added regression tests in prototype/tests/calculation_utils_test.py.

Validation

Automated tests

Ran:
uv run pytest library/tests/iqb/calculator_test.py prototype/tests/calculation_utils_test.py

Result:

  • 30 passed in 4.02s

Key tests:

  • calculator_test.py::TestIQBCalculatorScoreCalculation::test_calculate_iqb_score_changes_with_custom_thresholds
  • calculation_utils_test.py::test_calculate_iqb_score_with_custom_settings_changes_result

Behavior proof (script)

With fixed baseline inputs and strict thresholds:

  • baseline=0.571429
  • strict=0.000000
  • decreased=True

Manual proof of concept

Fixed inputs:

  • Download: 15.00
  • Upload: 20.00
  • Latency: 75.00
  • Packet Loss: 0.700

Strict thresholds (all use-case tabs):

  • Download: 500
  • Upload: 500
  • Latency: 5
  • Packet Loss: 0.100

Observed:

  • IQB score decreases after edits (no longer stays unchanged).

Screenshots

Before score (proof_before_score.png):
before_after_correct

Config changed (proof_config_changed.png):
check_after_corect

After score (proof_after_score.png):
after_after_corect

Impact

  • Fixes user-visible scoring correctness in prototype.
  • Makes “what-if” config editing reliable.
  • Keeps displayed scores aligned with edited thresholds/weights.

Issue

Closes #158

@bassosimone
Copy link
Collaborator

After listing our project for GSoC, we received a large amount of pull requests across several repositories. We are dealing with the backlog, but this would take time. We will get back to this pull request eventually. In the meanwhile, if you are a GSoC applicant, please read our updated GSoC policy: https://github.com/m-lab/gsoc/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Streamlit editor thresholds/weights are not applied to IQB score calculation

2 participants