ODC SC Meeting 2019 06 27

Attendees

• Rob Woodcock, Chair (CSIRO)
• Andrew Cherry (AMA)
• Kirill Kouzoubov (GA)
• Alex Leith (GA)
• Luigi Di Fraia (Catapult)
• Peter Wang (Data61)
• Syed Rizvi (AMA)
• Tony Butzer (USGS)
• George Dyke (Symbios)

Apologies

• Alex Leith (GA)
• Syed Rizvi (AMA)

Minutes

See Google Drive document.

Agenda

• Welcome
• Previous Actions
  • See below
• ODC Project Code of Conduct (Alex L)
• ODC AWS EKS k8s cluster release v0.5 milestone
  • SC confirm ODC supported feature set
  • Project board for 0.5
• Enhancement Proposals
  • Clean up:
    • Duplicate process - delete one or both of:
      • enhancement-proposal repo
      • datacube-core Wiki Enhancement Proposals
      • and alternate just use Issues flagged as enhancement proposal against each repository (this can also be supported by automation to remove stale issues not seeing any action).
  • New Proposals
    • ODC-EP01 Add Support for 3D Datasets
    • ODC-EP02 Support for multiple databases
    • ODC datacube-core v2.0
  • Committed Proposals
• State of the Cube Highlights
  • CSIRO
    • Working with GA and FrontierSI on the Jupyterhub/K8s/EKS implementation on github and CSIRO deployment.
      • Now available for authorised users at https://hub.easi-eo.solutions.
      • Dask is supported. Hit a few issues with parallel IO at scale swamping coredns. Manifests as connection resets in the Dask worker logs. dns-autoscaler resolves the issue by increasing the number of coredns replicas in proportion to worker node/core count (not CPU load Since that isn’t the issue). A PR is being developed to add this option to the opendatacube/datacube-k8s-eks repository.
      • Borrowed quite a few ideas from Pangeo. Not surprisingly the components add together well though we don’t have support for Pangeo binder (yet).
      • Next steps: integrate with CSIRO Live Ingester and ODC/GA indexing components, add datacube-ows.
    • Continuing work on CSIRO Live Ingester:
      • Supporting Appeears (MODIS), Espa (LS 5/7/8), Sara (S 1/2)
      • Working through supporting ingest of Geotiffs with GCP
      • Three Workflow types (Order & Ingest, Ingest only, Order only)
  • Catapult
    • IPP CommonSensing: Working on the JupyterHub setup with Kubernetes for on-premise deployment
      • Goals:
        • Clear provenance of the setup material
        • Computing environment and storage agnostic: dev laptop (microk8s/minikube), on-premise, or public cloud
        • Simple setup: all Helm/kubectl YAML blueprints in one place on GitHub
        • Optionally offload computing to a distributed Dask cluster, part of the setup
      • Documentation:
        • Setup material available here
        • Deployment instructions here
      • Blueprints/architecture:
        • Catapult internal environment setup blueprints shared here
      • Ongoing:
        • Testing of DCAL applications with Dask
          • S3 bucket access available from Dask workers, both public and non-public
          • Cloud masking working, as per execution example
          • Unfortunately median calculation is not yet supported for dask arrays: AttributeError: module 'dask.array' has no attribute 'nanmedian'
      • Next steps: End-user validation of the setup
  • GA
    • DE Africa
      • Working with NASA and AMA to acquire initial data for DE Africa, including Landsat and ALOS
      • Developing a Technical Roadmap for internal purposes at first, hopefully can share
    • DE Australia
      • Working on collection upgrade, which will involve freshly indexing
      • Collection upgrade will also involve re-running downstream products (WOfS and FC etc.), which will be a datacube-driven process
      • Work underway to define what an ODC 2.0 might look like...
  • USGS
    • Working with the USGS Pangeo Team to stand up a JupyterHub/K8s/EKS implementation in the USGS Secure AWS Space
      • pangeo.chs.usgs.gov
      • https://pangeo.io/architecture.html#where-we-began
      • Richard Signell - https://www.usgs.gov/staff-profiles/richard-p-signell?qt-staff_profile_science_products=0#qt-staff_profile_science_products
      • Scott Henderson - https://github.com/scottyhq/esip-tech-dive/blob/master/notebooks/0-demo-aws.ipynb
        • geophysicist and python enthusiast University of Washington, @uwescience
      • Ryan Abernathey (Columbia University) - The Future of Data-driven Discovery in the Cloud
    • Planning on attending the ARD19 - Planet Radiant.Earth Second ARD Workshop on Remote Sensing Data Interoperability - AUGUST 5-7th
      • Labahn, Kline, Renee, Covington, etc to attend also from USGS as well as others from all over the USA
    • Digital Earth Africa
      • We should decide on some target test sites for C2L2 Data - for example: South Africa
      • Developing a conops with Randy Sunne and Ron Dilley at EROS
      • L0Ra and L0Rp - will be no problem - all the way to L2 - still some schedule risk for August - mitigations?
  • AMA
    • Working with GA to get Landsat 5, 7, and 8 data for ARDC countries as well as continental ALOS available as CoG on S3
    • Ongoing notebook development, including SDG & DCAL
    • Working with ESRI to integrate their UI tools with the ODC. First pass is currently reading ingested data - looking toward a real OGC interface!
    • Testing latest UI/notebook release scheduled for end of month. Ensuring it gets properly pushed upstream as well
    • Testing UI code & deployments with 1.7
  • FrontierSI
• Other Business
  • ODC Conference 2020
  • Mailing lists
• Next Meeting and Close
  • Date

Actions

See the action tracking project.

Outstanding Actions

• Close out 1.7 Release
• Next ODC Conference structure
• Architecture document
• Licenses in ODC organisation on GitHub
• PR template text in datacube-core
• Copyright licensing of code contributions
• CSIRO to merge develop branch in to their code -> datacube-core 0.7

New Actions

• Update EP002
• Update EP001
• Tidy Enhancement Proposals Wiki
• Suggestion for Generic setup for k8s repo
• Draft Code of Conduct