NIH explorer matching needs refining

[tompollard]

@briangow we need to spend a bit of time refining the matching process in:
https://github.com/MIT-LCP/physionet-25-years/blob/main/notebooks/nih_explorer.ipynb

e.g.

• we need to do some manipulation of name structure before computing distance. e.g. for authors:
  • physionet_name needs converting into the same initial, surname form.
  • middle names in the physionet_name should be converted to an initial.
  • there should be consistency in use of punctuation (e.g. after middle initial).
  • we might want to try multiple matches for a given name (e.g. with and without middle initials)

It would be good to add some tests to confirm our approach is working (e.g. running the matching process on duplicate name lists).

[briangow]

@tompollard , thanks for the suggestions! I agree it makes sense to do these things.

The run I started ~1 day ago just finished (based on #19 ). The results are already showing a good number of matches: ~30,000 (27%) of PN users have a match of 1, ~56,000 (50%) of PN users have a match > 0.95.

[tompollard]

It would be incredible to be able to report these kind of numbers 🤞

[briangow]

@tompollard , I've been looking at some challenging examples from the PhysioNet users and NIH explorer data in regards to being consistent with middle names. I'll email you some examples but at this point I would suggest:

• Identify first and last names (PN users by using spaces b/w names [will result in improperly split last names for the cases where they don't have a "-" but multiple strings be the last name], NIH Explorer by splitting on commas)
  • Throw out suffixes prior to selecting last name (for PN we will probably have to check to see if the text matches a common suffix. For NIH Explorer projects, we can throw out anything after a second comma but for publications we'll have to check for a common suffix)
• Take anything remaining that isn't a first or last name and use it as a middle name. Convert all middle names to the first letter in a given string without any punctuation. (ex: John S. Williams Doe -> S W for a middle name)

[tompollard]

Sounds like a good start to me. It might be helpful to create a test set of synthetic names and add this to the repo. We can then apply different approaches to see which works best.

[tompollard]

(Oh, and it is probably helpful to work with the first_names and last_name fields in the PhysioNet data, rather than full_name, which concatenates the two).

[briangow]

(Oh, and it is probably helpful to work with the first_names and last_name fields in the PhysioNet data, rather than full_name, which concatenates the two).

Since we don't give guidance on where to put middle names or suffixes (into first_names or last_names) I don't think there will be any advantage over using full_name.

[tompollard]

Since we don't give guidance on where to put middle names or suffixes (into first_names or last_names) I don't think there will be any advantage over using full_name.

If you were looking at a single field containing "Trudy Caljé-van der Klei", it might not be clear whether "Caljé-van" was a middle name or a last name: https://physionet.org/content/respiratory-heartrate-dataset/1.0.0/

From the suggested citation, we can see that all of "Caljé-van der Klei" appears to be the last name (which aligns with what we see in other publications: https://www.sciencedirect.com/science/article/abs/pii/S0169260723006545.

(whoops, somehow deleted most of my comment by accident when posting, but you get the idea!)

[briangow]

@tompollard , yeah that makes sense but I'm not sure how to code this. Do you want to say we won't split a last_names by spaces if it contains de or der or other connectors? As stated above I was not originally planning to do this. We have to be able to pick out a last name in cases like John Simon-Cortez Simpson Doe, where Doe should be the last name. This may get entered into PhysioNet as first_names = John Simon-Cortez and last_names = Simpson Doe.

[tompollard]

I'd experiment with different approaches and see what works best. Do what you think works best for now, and we can tweak later if time.

[tompollard]

We have to be able to pick out a last name in cases like John Simon-Cortez Simpson Doe, where Doe should be the last name

I'd expect people to be fairly consistent about how they report their first and last names. Perhaps wrong though, I've not explored.