Skip to content

Editing Evaluation, Reproducibility, Benchmarks Meeting 36

Jump to bottom Edit New page
Nicholas Heller edited this page Jun 25, 2025 · 1 revision

Minutes of Meeting 36

Date: 25th June, 2025

Present

  • Carole
  • Olivier
  • Nick
  • Michela

Metrics Reloaded Implementations

  • It would be nice to have metrics reloaded integrated into training, but the MONAI team said we would need to do that ourselves
  • Michela had a student reach out for a summer project -- not sure about coding experience
  • Let's reach out to the team and see how much support they can provide

Confidence Interval Implementation

  • What's the timeline on this? Quite flexible
  • What about when user does something against findings/recommendations? It should give a warning
    • When the user gives a sample of values, it can test each metric's/aggregation method's assumptions, esp. regarding distribution shape
  • Should investigate the python skore package -- seems to be a metrics/aggregation companion to scikit-learn

Decathlon Data

  • We need permission from decathlon participants to use their data as an example with our MONAI implementations
  • It sounds like Michela would prefer that Annika/Carole send out request
    • We can just mention in the email that Michela is onboard/approves
  • We can't just use the test data -- the labels are held out!
    • So we're asking for the cross-validation results -- maybe just ask Fabian

Confidence Intervals Paper

  • Rejected after rebuttal
  • Resubmitted to the BRIDGE workshop
  • Journal version -- targeting a submission by July 15th
  • Have lots of data on segmentation -- working on similar data for classification
  • Have strong plots to show how common it is to deviate from normality
  • Using coverage
    • What fraction of generated CIs contain true mean/median?
  • Some metrics require more observations to achieve good coverage
    • Dice lower than ASSD for mean, for example

Upcoming Meetings

  • WG website suggestion box?
    • It sounds like they would like us to fork the repo and PR any changes in
  • Next generation of the BIAS initiative?
    • Talk about this next time
    • Annika has looked into other similar guidelines, borrowing items that we seem to be missing
  • Summer break is coming up
  • Maybe we should re-arrange some of these monthly meetings
    • Keep 16th July
    • Move the August meeting earlier
    • Annika won't be available next month

Copyright (c) MONAI Consortium

Working Groups

Clone this wiki locally