O+M 2025-02-10

[FuhuXia]

As part of day-to-day operation of Data.gov, there are many Operation and Maintenance (O&M) responsibilities. Instead of having the entire team watching notifications and risking some notifications slipping through the cracks, we have created an O&M Triage role. One person on the team is assigned the Triage role which rotates each sprint. This is not meant to be a 24/7 responsibility, only East Coast business hours. If you are unavailable, please note when you will be unavailable in Slack and ask for someone to take on the role for that time.

Check the O&M Rotation Schedule for future planning.

Acceptance criteria

You are responsible for all O&M responsibilities this week. We've highlighted a few so they're not forgotten. You can copy each checklist into your daily report.

Daily Checklist

• Check Production State/Actions

Note: Catalog Auto Tasks
You will need to update the chart values manually. Click the Action link in each issue and grab the values from monitor task output and check runtime.

• Check auto generated O&M tickets from no status column
• Check Harvesting Emails
• New Relic Alerts Triaged
• Triage DMARC Report from Google

Weekly Checklist

• DB-Solr Sync
• Audit Log (more info on AU-3 and AU-6 Log auditing)
• Tracking Update
  • NOTE: This job will consistently timeout, but it is processing results ((more details)[https://github.com/change tracking update from nightly job to weekly #4345])
• Check Catalog Solr
• Catalog Dupe Check
• Check user management requests
• Check Snyk Dashboard and cleanup

Monthly Checklist

• Invicti Scan

ad-hoc checklist

• audit/review applications on cloud foundry and determine what can be stopped and/or deleted.

Reference

• Watch for user email requests
• Watch in #datagov-alerts and Vulnerable dependency notifications (daily email reports) for critical alerts.
• Monitor and improve Data.gov O&M Dashboard
• Update and revise Data.gov O&M Tasks

[FuhuXia]

Inventory gave false site down alarm. Need to update ckan.ini and add apitoken_header_name = Authorization

condition
[inventory.data.gov](http://inventory.data.gov/) unavailable

policy
Site alarm (production)

[FuhuXia]

Things noticed for Inventory

• false site down alarm (as described above) (fixed by apitoken_header_name = Authorization inventory-app#812)
• NewRelic synthetic alert comes late (one day delay)
• Logged-in user's session does not end (fixed by keep session live for active user inventory-app#811)
• Editor could not edit a dataset. (fixed by add id to check_access, bump to 0.6.1 ckanext-dcat_usmetadata#336)

[FuhuXia]

All of a sudden PR created by snyk/GH does not run the checks any more. I have to add an empty commit to move it forward.

[FuhuXia]

harvest source /harvest/admin/connecticut-data-json generated 575 (1838 - 1263) duplicates.
need to run dedupe on it and reharvest to see if it comes back.

[FuhuXia]

Some records in harvest source /harvest/admin/connecticut-data-json appears to have identical content except the modified, causing extra workload on the harvester.

Take the following record for example, it is "modified": "2025-02-12" on day 1 then "modified": "2025-02-13" on the next day.

Also the order of keywords are changing everyday but that is not an issue for us since we sort our json before hashing.

{
  "accessLevel": "public",
  "landingPage": "https://data.ct.gov/d/yhjn-x67g",
  "issued": "2018-11-26",
  "@type": "dcat:Dataset",
  "modified": "2025-02-12",
  "keyword": [
    "child care",
    "enforcement actions",
    "licensing"
  ],
  "contactPoint": {
    "@type": "vcard:Contact",
    "fn": "Open Data Hub",
    "hasEmail": "mailto:no-reply@data.ct.gov"
  },
  "publisher": {
    "@type": "org:Organization",
    "name": "data.ct.gov"
  },
  "identifier": "https://data.ct.gov/api/views/yhjn-x67g",
  "description": "Child Care & Youth Camp Licensing Enforcement Actions New",
  "title": "Child Care & Youth Camp Licensing Enforcement Actions New",
  "distribution": [
    {
      "@type": "dcat:Distribution",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.csv?accessType=DOWNLOAD",
      "mediaType": "text/csv"
    },
    {
      "mediaType": "application/rdf+xml",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.rdf?accessType=DOWNLOAD",
      "describedBy": "https://data.ct.gov/api/views/yhjn-x67g/columns.rdf",
      "describedByType": "application/rdf+xml",
      "@type": "dcat:Distribution"
    },
    {
      "mediaType": "application/json",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.json?accessType=DOWNLOAD",
      "describedBy": "https://data.ct.gov/api/views/yhjn-x67g/columns.json",
      "describedByType": "application/json",
      "@type": "dcat:Distribution"
    },
    {
      "mediaType": "application/xml",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.xml?accessType=DOWNLOAD",
      "describedBy": "https://data.ct.gov/api/views/yhjn-x67g/columns.xml",
      "describedByType": "application/xml",
      "@type": "dcat:Distribution"
    }
  ],
  "theme": [
    "Business"
  ]
}

{
  "accessLevel": "public",
  "landingPage": "https://data.ct.gov/d/yhjn-x67g",
  "issued": "2018-11-26",
  "@type": "dcat:Dataset",
  "modified": "2025-02-13",
  "keyword": [
    "child care",
    "enforcement actions",
    "licensing"
  ],
  "contactPoint": {
    "@type": "vcard:Contact",
    "fn": "Open Data Hub",
    "hasEmail": "mailto:no-reply@data.ct.gov"
  },
  "publisher": {
    "@type": "org:Organization",
    "name": "data.ct.gov"
  },
  "identifier": "https://data.ct.gov/api/views/yhjn-x67g",
  "description": "Child Care & Youth Camp Licensing Enforcement Actions New",
  "title": "Child Care & Youth Camp Licensing Enforcement Actions New",
  "distribution": [
    {
      "@type": "dcat:Distribution",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.csv?accessType=DOWNLOAD",
      "mediaType": "text/csv"
    },
    {
      "mediaType": "application/rdf+xml",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.rdf?accessType=DOWNLOAD",
      "describedBy": "https://data.ct.gov/api/views/yhjn-x67g/columns.rdf",
      "describedByType": "application/rdf+xml",
      "@type": "dcat:Distribution"
    },
    {
      "mediaType": "application/json",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.json?accessType=DOWNLOAD",
      "describedBy": "https://data.ct.gov/api/views/yhjn-x67g/columns.json",
      "describedByType": "application/json",
      "@type": "dcat:Distribution"
    },
    {
      "mediaType": "application/xml",
      "downloadURL": "https://data.ct.gov/api/views/yhjn-x67g/rows.xml?accessType=DOWNLOAD",
      "describedBy": "https://data.ct.gov/api/views/yhjn-x67g/columns.xml",
      "describedByType": "application/xml",
      "@type": "dcat:Distribution"
    }
  ],
  "theme": [
    "Business"
  ]
}

[FuhuXia]

Due to #5085, NewRelic Synthetic monitoring for Inventory prod changed its endpoint url from /dataset/ to /, until the issue is fixed.

UPDATE:
issue fixed, NewRelic is monitoring '/dataset/' with X-CKAN-API-Key.