Begin UC3 admin functionality

[terrywbrady]

Admin Tool Infrastructure

• sceptre to create lambda and load balancer
• code build for new lambda
• allow lambda to run inside/outside of VPC (for mysql)
• allow the code base to be used to create a UC3 Dev Ops admin tool
• code base can support multiple admin tool instances
• enable domain specific constraints for the admin tool (dev, stg, prod)
• enable yaml hierarchy selection for the admin tool (dev vs docker)
• unit test framework for the admin tool

Admin Tool UI

• Build UI based on Sinatra - a known UI framework
• Logical Navigation Menu
• Accessible Navigation Menu
  • consult with UX team
• Ability to define breadcrumbs
• Add meaningful breadcrumb pages
• Generate meaningful breadcrumb pages from menu hierarchy
• markdown rendering
• responsive UI adjustments (maybe)
• trigger smart reload of CSS and JS

Table display

• identify known int and float values that require number formatting
• compute table totals in javascript
• generate filters on specific column names
• generate calculated filters based on computed values
• meaningful navigation for key primary key values
  • object id
  • collection id
  • mnemonic
  • job id
  • batch id
• ability to customize table headers esp for query columns
• ability to customize css for query columns
• ability to customize css for columns within a specific query/query class
• CSV export
• JSON export
• Ability to document table displays using markdown
• pagination controls

Code Tagging and Deployment

• List tags by repo
• Allow for tag documentation by repo (may need github api permissions)
  • Use GitHub releases
    • Commits and tags are static and must be authored at the time of creation
    • Releases can be created as drafts, created as releases and deleted without modifying the underlying structure of the repository
• Allow for tag deletion
  • If releases are used, there may be no need to delete tags
  • Release can be active/inactive
• Delete artifacts based on tags
• Delete ECR images based on tags
• Eventually deploy based on tags/releases

Query Tool

• Create Yaml-based query solution
• Utilize templates in order to re-use query fragments
• Display generated SQL with pretty formatting
• Sanitize any url parameters passed to a query using prepared statements
• Port existing queries
• gracefully handle data too large queries
• consistent pagination

Consistency report

• Establish consistency report framework
• Extend consistency report concept to the UC3 account: daily checks of system state that result in PASS, FAIL, WARN, INFO
• Implement PASS, INFO, WARN, FAIL, SKIP logic

Sub Applications

• Generic ZK Query Tools
• Merritt Queue Rendering Tools
• LDAP Query Tools
• Ingest Service Interaction
• Collection Configuration Service
• Inventory Service Interaction
• Storage Service Interaction
• Replication Service Interaction
• Storage Scan Application
• OpenSearch Interaction
  • Perform consistency checks agains error logs
  • Replace mrt-cron data build tasks with admin function calls

AWS resources

SSM

• pull SSM values
• compare SSM to SSM registry

EC2

• List EC2
• Flag EC2 running more than 30 days

Lambda

• List lambda
• list lambda update
• list lambda properties
• deploy lamda

Builds and Pipelines

• Trigger pipelines
• Look for failed pipelines/builds

Tags

• scan for resources by AWS tag
• enforce tagging standards (unless AWS already has a good approach for this)

Buckets

• list buckets and bucket sizes