Adding Mean F1 Score Difference and Hitting Rate Metrics #39
Conversation
…ixing a preprocessing bug and changing the name of the syntheval metric base class.
emersodb
requested review from
amrit110,
lotif,
fatemetkl,
sarakodeiri,
masi-sh and
bzamanlooy
|
|
||
|
|
||
| class SynthEvalQualityMetric(MetricBase, ABC): | ||
| class SynthEvalMetric(MetricBase, ABC): |
There was a problem hiding this comment.
Name change to be more general as it now underpins more than just quality metrics (Hitting Rate, with others to follow).
src/midst_toolkit/evaluation/quality/mean_f1_score_difference.py
Outdated
Show resolved
Hide resolved
src/midst_toolkit/evaluation/quality/mean_f1_score_difference.py
Outdated
Show resolved
Hide resolved
src/midst_toolkit/evaluation/quality/mean_f1_score_difference.py
Outdated
Show resolved
Hide resolved
src/midst_toolkit/evaluation/quality/mean_f1_score_difference.py
Outdated
Show resolved
Hide resolved
| A smaller rate is better. | ||
| NOTE: Categorical variables must be encoded in some way (ordinal or vector) for the evaluation to work. This |
There was a problem hiding this comment.
Should we perhaps implement the compute function in a way that takes care of the encoding for categorical variables? With do_preprocess only attributed to the numerical ones perhaps?
There was a problem hiding this comment.
I'm trying to allow a bit of flexibility in the way a user can process the datasets they are sending to this function, while also providing a "sensible" default (i.e. the one that SynthEval provides).
I don't want to force preprocessing on the user if they want to do it some other way. For example, perhaps imputation needs to be done or you are going to collapse under represented classes before encoding. So this was the route I thought might work best. Happy to discuss it a bit more though!
…ering.py (#8) Moving the clustering code into its own clustering.py module and adding docstrings. Also, moving some common parameter type definitions to a params.py module.
PR Type
Feature
Short Description
Clickup Ticket(s): https://app.clickup.com/t/868fk3gbd
This PR adds a metric to the the quality and privacy metrics in the library respectively. These are Mean F1 Score Difference and Hitting Rate.
Two additional components have also trivially changed.
Tests Added
Tests have been added for both metrics. Hitting Rate is very thoroughly tested and Mean F1 Score Difference is run on a large set of data that requires training multiple models. I've sanity checked the results and they all make sense.