Refactor evaluation metrics to output Legolas table rows

[ericphanson]

I wanted something like this for an internal project but ended up just processing the output myself. But maybe something like this would be useful here.

The idea is to return a Legolas.@row("lighthouse.evaluation@1",...) row instead of a Dict from evaluation_metrics. Then if you run Lighthouse evaluation on e.g. several models with the same dataset or one model with several datasets, you can easily collect the results as rows in a structured table.

See also #42 / #41 for other comparison stuff.

Status: I think the tests are failing and this started to feel like a waste of time so I switched focus, but could be picked up.

[hannahilea]

Great---this would be a huge improvement! As discussed in person, we'll try to get to this at some point in the not too distant future, and in the interim count not having this implementation yet as tech debt.

[hannahilea]

working on this now

[hannahilea]

• TODO: file issue to make Matrix{Int64} in EvaluationRow in future
  EDIT: In EvaluationRow, use Matrix{Int64} instead of Array{Int64} #51

[hannahilea]

Bumping the version b/c we've changed the expected output for "failure" cases from missing to NaN for any metrics output types that show up in vectors (to aid in Arrow serialization).

[hannahilea]

Since we're making a breaking version bump, taking the opportunity to additionally bump the lowest supported Julia version.

[hannahilea]

Should we export EvaluationRow?

Suggested change

	# TODO: export EvaluationRow ?
	export EvaluationRow

[ericphanson]

I think so, because it’s needed to fix Arrow serialization issues - maybe we should add a comment about that too. Since normally it's not important to pass through the Row type after reading from Arrow, but in this case it is, since we're using it to work around serialization issues.

[hannahilea]

Will do, although I'm not sure I understand the rationale?

[ericphanson]

so normally, if you do

table = Legolas.read(path_to_table)

then you get a table that serves your needs; you don't need to do any further processing to it (though you could pass to a DataFrame for convenience, for example). But for a table of evaluation rows, your confusion matrices will be vectors in this case! So to get a "good" table, you should do

table = (EvaluationRow(row) for row in Tables.rows(Legolas.read(path_to_table)))

Then you've got a (row-oriented) table whose confusion matrix column has actual matrices. Then you can pass that to a DataFrame if you want.

Usually you don't need to pass through the Row type, but since we're using it to fix stuff, you do in this case.

[hannahilea]

Ah, gotcha! Thanks for the explanation.

[hannahilea]

in this case, still okay to return missing not NaN b/c we aren't serializing these as part of a vector---the schema accounts for them

[hannahilea]

...juliaformatter got hold of these matrices, and I didn't have the heart to undo the updates.

[hannahilea]

@ericphanson, I can't officially request your review here (since you were the originating author!), but want to take a review anyway?

[ericphanson]

I'm very excited for this! I think this will make things much nicer. (Cannot approve bc it's technically my own PR, but ✅ )

[ericphanson]

I think so, because it’s needed to fix Arrow serialization issues - maybe we should add a comment about that too. Since normally it's not important to pass through the Row type after reading from Arrow, but in this case it is, since we're using it to work around serialization issues.

[hannahilea]

#45 (review)