Incorporate Rekognition data into the catalog

[obulat]

Summary

Rekognition data in the form of object labels was collected for roughly 100m records in the Openverse catalog.

These labels should be sanitized for suitability in the Openverse project and applied to records in the Openverse Catalog as tags.

Description

Some exploratory work was done to assess the quality of these labels. The team generally felt positive about them, given we would blanket remove a subset of them (e.g. ones that assume a gender). We will need to do a broader analysis to determine if there are more labels we would want to exclude, and then incorporate them into the existing tags for each record in the catalog. The automated tags include a confidence score associated with the tag value, and we should also incorporate those values into the overall document score for relevant searches.

Best guess at list of implementation plans:

• Strategy for filtering then upserting the tags into their associated records.
• Determining whether/how to surface these tags in the frontend and differentiate them from provider-supplied tags

Documents

• Project Proposal
• Implementation Plan: Determine and design how machine-generated tags will be displayed/conveyed in the API
• Implementation Plan: Determine and design how machine-generated tags will be displayed/conveyed in the Frontend
• Implementation Plan: Augment the catalog database with suitable Rekognition tags
• Reviewed Rekognition label list

Issues

• Project Proposal: Incorporate Rekognition data into the catalog #3896
• Implementation Plan: Determine and design how machine-generated tags will be displayed/conveyed in the API #4038
• Implementation Plan: Determine and design how machine-generated tags will be displayed/conveyed in the Frontend #4039
• Implementation Plan: Augment the catalog database with suitable Rekognition tags #4040

Milestone

Incorporate Rekognition Data

[zackkrida]

Early Testing

Back in April I ran a simple script to do some basic analysis of the Rekognition labels. I mostly wanted to test the speed of reading all of the data.

Here's the script I used: https://gist.github.com/zackkrida/cb125155e87aa1c296887e5c27ea33ff

Infra setup

The script was run on a manually-provisioned EC2 instance. The instance was configured with permissions to access our S3 bucket. I also used an instance with Enhanced Networking support so the script would theoretically stream the rekognigtion data as fast as possible.

Unfortunately I only loosely recall how long it took, and am struggling to find my notes. I believe it was around 4-5 hrs. I do remember being happy with the speed.

General recommendations

For this project I would strongly recommend we download the full list of Rekognition labels from this page: https://docs.aws.amazon.com/rekognition/latest/dg/labels.html and filter out anything related to gender prediction.

As far as the approach we take to importing the rekognition data, we could probably use a script much like the one I wrote to stream the rekognition data and then perform sql updates in batches, adding the new tags to the existing array with a provider value of "Rekognition". We may also want to store the confidence of each tag in the Catalog DB. This would give us more flexibility in the future. We could fine tune tags in Elasticsearch, for example, and only choose to show those with a certain confidence level.

[AetherUnbound]

The project proposal has recently been merged, and issues for the 3 implementation plans have been created (linked above). I plan on starting the API-related IP soon.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

No change since the previous update - IPs still need to be drafted.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

The IP for the API-side of things has been merged (#4189) and can be seen here. The only issues necessary for this work has been created and will be worked on in the next week or so: #4273.

@fcoveram has also established mock-ups for how the machine-generated tags will be displayed in the frontend in #4192. This was a necessary prerequisite for the frontend IP, #4039, which @obulat will be working on.

Work can also begin on the final IP, #4040, which will be a more subjective dive into the tags themselves and what policy Openverse will take for machine-generated labels.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

The final IP for the incorporation of the tags into the catalog has been opened, and raised a number of good discussion points: #4417. We have a few things to iron out there as it relates to the data normalization project (#430) and the removal of the ingestion server (#3925).

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

We've taken a pause on the Rekognition work since the discussion on it has prompted questions on the above. We've got some good clarification by way of #4465 and #4524, but for now I'm going to move this project to On Hold in the short-term while we resolve those discussions!

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

This project was recently moved into In Progress (from On Hold) now that the project lead (myself) is back from AFK. Based on the merging of #4417, I've added a number of issues to the project milestone:

• Clarify policy around which demographics to exclude in Rekognition IP #4642
• Determine which labels to exclude from Rekognition’s label set #4643
• Preemptively filter the Rekognition tags #4644
• Create a DAG to generate and insert the new Rekognition tags #4645
• Filter the existing Clarifai tags #4646
• Assess the existing Clarifai tags #4647

Work on many of these can begin immediately! Particularly #4642, which I may start on this week.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

Discussion around the label set to use (#4643) is ongoing, and the DAG work (#4645) will be picked back up this week.

[sarayourfriend]

@AetherUnbound when working on #4643, it occurred to me that the bounding box information in the tags (and potentially the categories) might also be useful in the future. I just checked the project proposal and the ingestion IP, but I didn't see any clear determination about whether the bucket would be kept around after this work. Mostly just wanted to make sure that it would be, and that we aren't treating ingestion of the tags in the current mode to be the definitive end-all-be-all of that dataset's usefulness to us. Just asking for clarification that we won't delete that bucket, basically.

[AetherUnbound]

Oh definitely not - my intention was to keep the source data in perpetuity regardless! That was implicit in the lack of mentioning what would happen to the bucket, but it can be made explicit in the final IP if you would like me to make it so.

[sarayourfriend]

Maybe worth adding into the project proposal as a clarification about the outcomes of the project, but so long as it's recorded somewhere, I'm happy about that! We could also add an issue to add it to Terraform and move it to infrequent access, as we discussed in the issue related to #3810, to save money on storage of it for the foreseeable future, as well as documenting its existence and our long-term intentions with keeping it around 🙂 That issue wouldn't be part of the project's "shipped" status, though, to clarify.

[AetherUnbound]

Great points! I'll go ahead and make that issue and the adjustment to the project proposal.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

The reviewed Rekognition label list has been added to our documentation, and we now have an issue for filtering the tags during the data refresh which can be worked on: #4813

The add_rekognition_labels DAG is complete as well 🎉 I intend to run it as soon as I can.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

The add_rekognition_labels DAG has been run and the Rekognition labels have now successfully been inserted into the catalog database!

The next step will be to implement the selected filtering for the labels so that we can remove the global filtering we're doing across the provider as a whole.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

No progress has been made on this since the last update, I'm going to move this into on hold for now while the maintainers adjust to new availability levels and priorities.