NITS - how we handle things when we have Nothing Interesting To Say

[Jegelewicz]

A convergence of things has made me consider the data we "require" in order to be able to create a catalog record in Arctos (and how it is sometimes non-nonsensical and also biological-collection centric).

First there is this: [The vexed question of missing data in Darwin Core](https://discourse.gbif.org/t/the-vexed-question-of-missing-data-in-darwin-core/3374)

Then, there is the set of geological specimen data that I am working on for the Natural History Society of Maryland and @mschmidtjr who has pointed out that:

1. Dates of collection are less important and potentially less-often recorded in geological collections and using 1800-01-01 when dates aren't known is VERY MISLEADING.
2. There is difference between "unknown" when we really don't know and "unknown" when all we have to designate a person is "T.J.M."

So, I would like to reiterate a proposal made previously - all that should be required to create a record in Arctos is an identifier - the catalog number. Perhaps individual collections can require other information, but imposing the following is not useful to everyone!

"required" from bulkloader builder

collection_object_id
enteredby
accn
guid_prefix
collector_agent_1
collector_role_1
taxon_name
nature_of_id
id_made_by_agent
made_date
higher_geog
spec_locality
verbatim_locality
verbatim_date
began_date
ended_date
specimen_event_type
event_assigned_by_agent
event_assigned_date
verificationstatus
part_name_1
part_condition_1
part_lot_count_1
part_disposition_1

but I am also not really sure WHAT is required, because I KNOW you don't have to include a part to load stuff.

Next, I think we do need more than one version of "unknown" for agents, and I think that we have been sort of given the terms we need:

unknown = we really don't know (but NULL is better if we can allow it)
unknown:undigitized = we have, or are pretty sure we have this data, we just can't find the time to add it right now
unknown:indecipherable =An indication made by a transcriber that they failed to transcribe the information (this could also cover initials that cannot be associated to a particular agent name?)
known:withheld = we have this information, but it is being with held for conservation or privacy reasons, contact us to get it

[Jegelewicz]

I'll also add to this our choice of what to put in specific locality.

This isn't a criticism, it's a question. Why did the Arctos team decide to give this advice in the handbook (https://handbook.arctosdb.org/documentation/locality.html)?

#In these examples, as well as in records for which appropriate data are missing, the correct value for Specific Locality is, “No specific locality recorded.”#

I ask because it goes against the databasing principle that a field should contain only 1 kind of information, in this case a locality string. You would raise an eyebrow if the Specific Locality entry was "They sell cherry ice cream here", because that's not a locality, and neither is "No specific locality recorded."

I'm arguing in that GBIF forum post (https://discourse.gbif.org/t/the-vexed-question-of-missing-data-in-darwin-core/3374) that you should use a separate field to explain missing data. Did the Arctos community decide that an explanatory field wouldn't be used properly?

Here's a real-world example, from the DMNS Marine Invertebrates dataset is DwC-ed in GBIF. Among the DwC "locality" entries are

No. of records | locality
19 | [blank]
2 | no specfic locality recorded
1 | no specific
2 | no specific date recorded
1 | no specific island recorded
2 | no specific lacality recorded
2 | no specific locaility recorded
2 | no specific localilty recorded
1 | no specific localitly
1 | no specific locality
303 | no specific locality
13 | No specific locality
1 | no specific locality entered
17 | no specific locality given
1 | No specific locality given
1 | no specific locality located
1 | no specific locality recoded
2 | no specific locality recorded
3 | no specific locality recorded
8695 | no specific locality recorded
16 | no specific locality recorded.
247 | No specific locality recorded
2733 | No specific locality recorded.
4 | no specific locality recorded. Specific Gulf inferred from context.
5 | no specific locality recorded (terrestrial)
3 | no specific locality recorded (terrestrial species)
1 | no specific locality redorded
1 | no specific locality reorded
3 | no specific locality reported
2 | no specific locality reported.
2 | no specific locality specified
6 | no specific locallity recorded
1 | no specific localtity recorded
22 | no specific localtiy recorded
4 | no specific location
3 | no specific location recorded
1 | no specific loclality recorded
2 | unknown
4 | unspecified

What happened and why? (I appreciate that a partial answer is given by DMNS use/non-use of lookup tables and validation rules; there are worse issues in this dataset.) From the data user's point of view, wouldn't it have been better if all these entries were blank?

Best wishes,

Bob

Dr Robert Mesibov
West Ulverstone, Tasmania, Australia 7315
Home phone: (03) 6425 2630
https://www.datafix.com.au/ (data auditing and cleaning)

[Jegelewicz]

Hi All,

Fascinating and wonderful conversation. Back in Morphbank heyday (mid to late 2000s), lots of discussion about using terms like
undetermined
unknown
unspecified
not available

etc. And just how useless most of those terms are.

Does undetermined mean no one has tried? Does unknown mean it's unknowable or that no one has tried? Does unspecified mean someone isn't sure what to pick or hasn't bothered yet? Does not available mean it's available to some? Or that there's no more information? And IF I'm trying to see / note / figure out the "completeness" of my data records and what's left to be done, these values do nothing to help me, and in fact they likely make it harder. Could type lots more ... but bravo Bob and Teresa for helping to nudge the community forward on this topic. It's important downstream for data use, but also upstream for those producing / tracking / managing data.

Onward!

Debbie

• Deborah Paul, Biodiversity Informatics Community Liaison

[sharpphyl]

Dates of collection are less important and potentially less-often recorded in geological collections and using 1800-01-01 when dates aren't known is VERY MISLEADING.

Yes! If there is no verbatim date recorded, you should be able to leave the began date and end date blank! Otherwise, you're creating fake dates. You can delete them after the record is uploaded but shouldn't have to enter a totally misleading date to create the record.

[Jegelewicz]

Apparently you CAN do that now - I feel like a lot of things I have migrated now need to be corrected...also documentation is needed.

[dustymc]

Generally agree, and #4384 provides a (potential) way for collections to "require" stuff. (Or I'd be happy to argue that's something that a collection manager need to do when reviewing data on a case-by-case basis.)

more than one version of "unknown" for agents,

Evil. In almost all cases I'll continue to argue that whatever information you might cram in there shouldn't be Agents at all, it's attributes (until they do something that can't be smooshed into a string). Neither the new types nor the agents intended to use them are necessary; both result in worse, not better, data.

If there is no verbatim date recorded, you should be able to leave the began date and end date blank!

That's just willfully tossing out information. You can make educated guesses regarding earliest possible - you almost certainly know more about the history of your collection that your successor will. End is even more clearcut - it wasn't collected tomorrow, and the current tomorrow is more precise than any future tomorrow possibly can be.

CAN do that now

Yes, apparently - I hope that was part of some discussion, seems opposite of Research Grade to me....

Does undetermined mean no one has tried? Does unknown mean....

We had that idea in sex, I think it was seldom used properly. I like the idea, but I don't think it survives reality so I've become a fan of one "we don't know" value, whatever form it might take, supported by appropriate metadata.

[dustymc]

Also important to consider the opposite of most of the above, not saying anything when something is in fact known (or can be inferred).

#4839 for example seems to come from the blank begin dates, specifically https://arctos.database.museum/guid/ASNHC:Mamm:15396. The verbatim date is ambiguous, but getting an earliest year out of it doesn't seem like much of a stretch. Even allowing for the least-expected interpretation (that I can imagine, which isn't always the full scope of the problem) narrows things down to 60 years, which is a whole lot better than someone might be able to do in a couple hundred years. Not recording anything there still seems like a needless loss to me (and I therefore think we should probably be requiring began and ended dates).

Taxonomy is another good example, there are thousands of unidentifiable records in Arctos, most of them are very much identifiable, but perhaps not to the "normal" resolution. https://arctos.database.museum/guid/UAM:ES:9016 (random example) has bone-like doohickeys, it's not a redwood or motorcycle or virus, 'Chordata' seems entirely safe and carries a LOT more information than the current taxon.

That said, the 'you're cataloging it you must know SOMETHING!' view lead to the existence and forced use of https://arctos.database.museum/agent/0 (not to mention https://arctos.database.museum/agent/1015585 and probably more), maybe my view is too focused.

@Jegelewicz you should work this up as a best practice or something; maybe that'll stimulate more discussion if nothing else.