PFB Handover fails in LungMap catalog

[nadove-ucsc]

• Security design review completed; the Resolution of this issue does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[achave11-ucsc]

Assignee to acquire a reproduction.

[nadove-ucsc]

The user who reported the error provided this workspace URL as an example of a failing import:

https://app.terra.bio/#import-data?format=PFB&url=https%3A%2F%2Fservice.azul.data.humancellatlas.org%2Fmanifest%2Ffiles%3Fcatalog%3Dlm3%26format%3Dterra.pfb%26filters%3D%257B%2522projectId%2522%253A%2B%257B%2522is%2522%253A%2B%255B%252200f056f2-73ff-43ac-97ff-69ca10e38c89%2522%255D%257D%252C%2B%2522genusSpecies%2522%253A%2B%257B%2522is%2522%253A%2B%255B%2522Mus%2Bmusculus%2522%255D%257D%252C%2B%2522fileFormat%2522%253A%2B%257B%2522is%2522%253A%2B%255B%2522h5%2522%255D%257D%257D%26objectKey%3Dmanifests%252F7bc50d88-3765-5909-b31c-824554e66938.1a5242f3-aea4-56c8-83ec-c912a74fe757.avro

Decoding the url query parameter yields

https://service.azul.data.humancellatlas.org/manifest/files?catalog=lm3&format=terra.pfb&filters=%7B%22projectId%22%3A+%7B%22is%22%3A+%5B%2200f056f2-73ff-43ac-97ff-69ca10e38c89%22%5D%7D%2C+%22genusSpecies%22%3A+%7B%22is%22%3A+%5B%22Mus+musculus%22%5D%7D%2C+%22fileFormat%22%3A+%7B%22is%22%3A+%5B%22h5%22%5D%7D%7D&objectKey=manifests%2F7bc50d88-3765-5909-b31c-824554e66938.1a5242f3-aea4-56c8-83ec-c912a74fe757.avro

Which we should be able to use to debug the manifest.

[nadove-ucsc]

The manifest from the above link contains a donor aggregate where the age_range field contains a mix of value/unit object and empty strings.

$ pfb show --input hca-manifest-7bc50d88-3765-5909-b31c-824554e66938.1a5242f3-aea4-56c8-83ec-c912a74fe757.avro | jq 'select(.name=="donors")' | jq -s '.[2].object | {document_id, organism_age}'
{
  "document_id": [
    "07a620ff-1ac8-45bb-85c1-2b7af107d68d",
    "17c9743a-bad9-417d-a56a-93a7b3e9ea71",
    "3886b8ef-fe67-480c-8496-7799ceabe6de",
    "42194a89-2b58-4bb0-9d70-e403db53bbf9",
    "90d58445-99d0-46e8-a882-5beac347a7a2",
    "913f8cad-2080-4ec7-a51f-aba0d4f09b66",
    "b122c921-e432-42c7-a5c6-f245f106970b"
  ],
  "organism_age": [
    {
      "value": "15",
      "unit": "day"
    },
    {
      "value": "3",
      "unit": "day"
    },
    {
      "value": "42",
      "unit": "day"
    },
    {
      "value": "7",
      "unit": "day"
    },
    ""
  ]
}

Based on the error message, it is the type mismatch between the string and the objects that is being rejected by Terra.

[nadove-ucsc]

This file is linked to the same set of donors shown above. The organism_age field matches that observed in the manifest except that the last element is null instead of "".

$ http 'https://service.azul.data.humancellatlas.org/index/files/0aec111a-fbb0-4063-bba2-b73c0666f17d?catalog=lm3' | jq '.donorOrganisms[] | {id, organismAge}'
{
  "id": [
    "07a620ff-1ac8-45bb-85c1-2b7af107d68d",
    "17c9743a-bad9-417d-a56a-93a7b3e9ea71",
    "3886b8ef-fe67-480c-8496-7799ceabe6de",
    "42194a89-2b58-4bb0-9d70-e403db53bbf9",
    "90d58445-99d0-46e8-a882-5beac347a7a2",
    "913f8cad-2080-4ec7-a51f-aba0d4f09b66",
    "b122c921-e432-42c7-a5c6-f245f106970b"
  ],
  "organismAge": [
    {
      "value": "15",
      "unit": "day"
    },
    {
      "value": "3",
      "unit": "day"
    },
    {
      "value": "42",
      "unit": "day"
    },
    {
      "value": "7",
      "unit": "day"
    },
    null
  ]
}

On prod, we replace nulls with the empty string when creating the Avro PFB manifest source. This is no longer the case on dev (#5444). I hypothesize that if this change were promoted to prod, and the null value was restored, the error would be resolved, because it seems logical that Terra should accept null where it expects an object. I am struggling to think of a way to test this hypothesis.

[nadove-ucsc]

As luck would have it, the exact same file is also present on dev in the lm2 catalog. I created a single-file manifest on Azul dev and Azul prod containing this file, and confirmed that the only difference between their contents was that the manifest created on prod has the nulls replaced with empty strings.

I imported the manifest created on prod into a new workspace and reproduced the original error.

I then imported the manifest created on dev into a new workspace. The import was successful.

I believe that this proves my hypothesis correct, meaning that this is a freebie or duplicate of #2462.

[achave11-ucsc]

Assignee to estimate prevalence of the problem per @bvizzier-ucsc.

[achave11-ucsc]

Fixed in #5444.

[bvizzier-ucsc]

The goal is to be able to loosely quantify the problem so we can notify LungMAP users and ask anyone who encounters it to let us know so we are aware of the impact. Knowing the breadth of impact will inform the relative priority of this issue.

[nadove-ucsc]

For demo, attempt to reproduce on prod.

[nadove-ucsc]

This problem affects:

• 9/600 files in the entirety of catalog lm3 (1.5%)
• 4/999 and 7/999 files in two random samples of catalog dcp29 (0.55%)