Adding in Hellinger and pMSE metrics #38
Conversation
emersodb
requested review from
amrit110,
lotif,
fatemetkl,
sarakodeiri,
masi-sh and
bzamanlooy
| "D104", # Ignore package level docstrings requirement | ||
| "D205", # 1 blank line required between summary line and description | ||
| "D212", # Multi-line docstring summary should start at the first line | ||
| "D301", # r-strings for docstrings with backslashes |
There was a problem hiding this comment.
This needs to be brought in if we're going to have latex in our docstrings.
src/midst_toolkit/evaluation/quality/mean_hellinger_distance.py
Outdated
Show resolved
Hide resolved
src/midst_toolkit/evaluation/quality/mean_hellinger_distance.py
Outdated
Show resolved
Hide resolved
| """ | ||
| Compute the empirical Hellinger distance between two discrete probability distributions. Hellinger distance for | ||
| discrete probability distributions $p$ and $q$ is expressed as | ||
| $$\\frac{1}{2} \\cdot \\Vert \\sqrt{p} - \\sqrt{q} \\Vert_2$$. |
There was a problem hiding this comment.
Minor:
| ``do_preprocess`` is True, the default Syntheval pipeline is used, which does NOT one-hot encode the | ||
| categoricals. |
There was a problem hiding this comment.
Maybe we can say "which performs "OrdinalEncoding" for categorical columns and "MinMaxScaling" for numerical columns." to clarify what exactly happens.
| The mean of the individual Hellinger distances between each of the corresponding columns of the real and | ||
| synthetic dataframes. This mean is keyed by 'mean_hellinger_distance' and is reported along with the | ||
| "standard error" associated with that mean keyed under 'hellinger_standard_error'. | ||
| """ |
There was a problem hiding this comment.
Probably not for this PR, and it might already be covered, but at some point it would be nice to check that the categorical and numerical columns actually exist in the provided real and synthetic data.
| distance = hellinger_distance(real_discrete_pdf, synthetic_discrete_pdf) | ||
| hellinger_distances.append(distance) | ||
|
|
||
| mean_hellinger_distance = np.mean(hellinger_distances).item() |
There was a problem hiding this comment.
If there are no categorical columns and include_numerical_columns is False, it will return nan. So, we might want to have a warning message.
PR Type
Feature
Short Description
Clickup Ticket(s): https://app.clickup.com/t/868fge8mt
Adding in two additional quality metrics from SynthEval to the library. These are the penultimate quality metrics to be added (just need an F1 measure in the next ticket to close off the quality measures).
NOTE: The SynthEval implementation of the Hellinger distance for numerical columns was flawed. So I brought the implementation in-house and fixed the issue.
Tests Added
A number of tests have been created to make sure the measures work as expected and produce the correct values across a number of situations.