Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

evaluation_metrics_plot fails when IRA fields are in the input EvaluationRow for multiclass result #75

Closed
alexmchan opened this issue May 19, 2022 · 3 comments · Fixed by #76

Comments

@alexmchan
Copy link
Member

alexmchan commented May 19, 2022

Currently, if votes is not provided to Lighthouse.evaluation_metrics_row, it outputs a row with per_class_IRA_kappas and multiclass_IRA_kappas fields defined as missing. If this is a multiclass result, then passing in such a row to evaluation_metrics_plot fails because it doesn't expect these fields to exist when not dealing with multi-rater metrics.

For example, for a 5 class EvaluationRow, it throws:

ERROR: DimensionMismatch("arrays could not be broadcast to a common size; got a dimension with lengths 12 and 8")
@alexmchan alexmchan changed the title evaluation_metrics_plot fails when IRA fields are in the input EvaluationRow evaluation_metrics_plot fails when IRA fields are in the input EvaluationRow for multiclass result May 19, 2022
@hannahilea
Copy link
Contributor

oh geez. this will be an easy fix---I can make it for you asap, @alexmchan, unless you're already on it?

@hannahilea
Copy link
Contributor

yet another reason that it will be exciting to do #71---plots shouldn't ask for more than they truly need/expect to plot. none of this evaluation_metrics_plot behemoth nonsense.

@alexmchan
Copy link
Member Author

I'm already mostly through this one. Thanks for the offer though!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants