v1.0.0

This is a reference release created for testing version support. It is based on Science Pipelines d_2023_12_19, which uses rubin-env 7.0.1, to avoid a crash introduced with rubin-env 8.0.0 (see DM-42543).

Compatible with Butler repositories with dimensions-config daf_butler 3-4.

What's Changed

• DM-33918: Create code repository for prompt processing by @kfindeisen in #1
• DM-33938: Rename prompt-prototype by @kfindeisen in #2
• Add a playbook section on databases. by @ktlim in #3
• add Cloud Shell instructions by @ebellm in #5
• DM-33935: Create basic test framework for prompt-prototype by @parejkoj in #6
• DM-33936: Create GHAs to build containers and run unit tests. by @ktlim in #4
• DM-33939: Set up an APDB instance on GCP by @ebellm in #7
• DM-34018: Move activator service one level down. by @ktlim in #10
• DM-33931: Improve logging by @kfindeisen in #9
• DM-33974: Create a "Butler worker" VM on Google by @kfindeisen in #11
• DM-34031: Add script for uploading raws. by @kfindeisen in #12
• DM-33996: Use the butler API and mock data for ingest tests by @parejkoj in #8
• DM-33937: Setup a butler with real data on google cloud by @kfindeisen in #13
• DM-33970: Use real images in upload.py by @kfindeisen in #15
• DM-34068: Implement Prompt Prototype's data transfer to workers by @parejkoj in #14
• DM-34123: Use hashable Visit in upload.py by @ebellm in #16
• DM-34132: Use shared visit.py with pure python types by @ebellm in #17
• DM-34134: Reintegrate activator with upload.py by @kfindeisen in #18
• DM-34017: Use the butler API and mock data for run_pipeline tests by @ebellm in #19
• DM-34135: Reintegrate activator with MiddlewareInterface by @kfindeisen in #20
• Update instructions for creating a Cloud Run service by @kfindeisen in #21
• DM-35052: Fix single visit restriction for Prompt Processing by @kfindeisen in #22
• DM-35051: Allow Prompt Processing to set up a local repo with existing files by @kfindeisen in #23
• DM-35941: Remove use of implicit dimensions in Prompt Prototype by @kfindeisen in #25
• DM-34020: Get Google Cloud to detect logging levels in prompt_prototype by @kfindeisen in #26
• DM-35053: Implement syncing of output products for Prompt Processing by @kfindeisen in #24
• DM-36485: Update playbook to give accurate description of upload.py by @kfindeisen in #28
• DM-36080: Separate GCP-specific code in Prompt Processing prototype by @kfindeisen in #27
• Fix oversights in envvar descriptions by @kfindeisen in #29
• Update GHA workflows by @kfindeisen in #30
• DM-36509: Set up test Prompt Processing central repository at USDF by @kfindeisen in #31
• Include S3 user profile in bucket examples. by @kfindeisen in #32
• DM-36602: Move from NCSA+GCP to USDF by @hsinfang in #33
• DM-36712: Migrate activator.py to use Kafka/Knative by @kfindeisen in #34
• DM-36720: migrate upload.py to USDF by @hsinfang in #35
• DM-36995: Remove test_upload.py temporarily by @hsinfang in #36
• DM-36999: Complete integration test at USDF by @kfindeisen in #38
• DM-36791: Use unique IDs in upload.py output by @kfindeisen in #39
• DM-37258: Logs getting dropped in prompt processing by @kfindeisen in #42
• DM-37068: Clear out conflicting datasets before prompt processing runs by @kfindeisen in #41
• DM-37070: tester script using HSC-RC2 data at /repo/main by @hsinfang in #40
• DM-37382: fix repeating raw image by @hsinfang in #43
• DM-37547: HSC exposure ID generator produces invalid IDs in 2023 by @kfindeisen in #46
• DM-37072: Expand HSC calibs and templates in central repo by @hsinfang in #44
• Document the kubectl describe command in the playbook by @kfindeisen in #48
• Add troubleshooting section to playbook by @kfindeisen in #50
• DM-36771: Bring Visit object in line with SAL spec by @kfindeisen in #49
• DM-37759: Rework container build workflows. by @ktlim in #51
• DM-37072: Expand HSC calibs and templates in central repo by @hsinfang in #47
• DM-37751: Add a script to make the export file for templates by @hsinfang in #52
• DM-36586: Use a single output run in Prompt Prototype by @kfindeisen in #45
• DM-37745: Prompt processing preload fails silently if central repo doesn't have prerequisites by @kfindeisen in #53
• DM-38066: Reconcile /repo/embargo as the central repo by @hsinfang in #54
• DM-36162: Upgrade from SimplePipelineExecutor in prompt prototype by @kfindeisen in #56
• DM-38405: Update the nextVisit events produced by the tester upload scripts by @hsinfang in #57
• DM-38752: Output products can go to the wrong collection by @kfindeisen in #58
• DM-38763: ValueError: Requested 'N921' not in DiaPipelineConfig.validBands by @hsinfang in #59
• DM-38269: For LSST cameras, get group/snap from sidecar JSON. by @ktlim in #55
• DM-38597: Add visit_system to WHERE clause. by @ktlim in #62
• DM-38594: Add image timeout heuristic using script duration. by @ktlim in #60
• DM-38596: Get bucket_topic from env var. by @ktlim in #61
• DM-39030: misleading ARGs in Dockerfile.activator by @hsinfang in #63
• DM-39282: Filter out JSON notifications. by @ktlim in #65
• DM-39331: Reduce number of messages consumed. by @ktlim in #66
• DM-39364: Reduce notification messages consumed per call to 1. by @ktlim in #67
• DM-39395: Use day_obs in run names by @kfindeisen in #68
• DM-38225: Intermittent init-output collision in Prompt Processing by @kfindeisen in #69
• DM-39653: Prompt processing unit tests have bitrotted by @kfindeisen in #70
• DM-39188: Export raws only if they are not already in the central repo by @hsinfang in #64
• DM-37253: Make Prompt Processing service configurable by @kfindeisen in #71
• DM-39678: Allow Prompt Prototype to be configured with no pipeline by @kfindeisen in #72
• DM-39681: Add LATISS pipelines to ap_pipe to be used by prompt processing by @hsinfang in #73
• DM-39790: Try to consume right after Kafka subscription. by @ktlim in #74
• DM-39655: Use a list of "fallback" pipelines in Prompt Processing by @kfindeisen in #75
• DM-40225: Update testers and dev env to use the new bucket credential by @hsinfang in #76
• DM-40210: Guard against future ap_pipe changes by @kfindeisen in #78
• DM-40296: Add a timestamp to FannedOutVisit by @kfindeisen in #79
• DM-39514: Add validity-range workaround to prep_butler by @kfindeisen in #77
• DM-40511: Refresh the unit test data repo by @hsinfang in #80
• DM-35828: Use MDC or equivalent in prompt_prototype logging by @kfindeisen in #81
• DM-38215: Parallelize upload_hsc_rc2.py by @kfindeisen in #82
• DM-37387: A tester script using LATISS data in /repo/embargo by @hsinfang in #84
• DM-36772: Migrate PostgreSQL credentials to mounted file by @kfindeisen in #83
• DM-40163: Rework dataset transfer code in MiddlewareInterface by @kfindeisen in #85
• DM-40986: error on gaia_dr3_20230707 in prompt processing by @hsinfang in #87
• DM-41084: AttributeError: '_thread._local' object has no attribute 'context' by @kfindeisen in #90
• DM-41165: Prompt processing can't export outputs to /repo/embargo by @kfindeisen in #89
• DM-37387: A tester script using LATISS data by @hsinfang in #88
• DM-41498: Reorganize the pipeline config Prompt Processing uses by @hsinfang in #92
• DM-40822: Update Prompt Processing docs to remove references to the IDF by @kfindeisen in #93
• DM-41658: Prompt Processing can't configure surveys with "-" by @kfindeisen in #94
• DM-41664: Rename prompt_prototype package and containers by @kfindeisen in #96
• DM-41717: Prompt Processing observing support for AuxTel week 46 by @kfindeisen in #98
• DM-41836: Prompt Processing may wait too long for raws by @kfindeisen in #99
• DM-34141: Distinguish retriable and non-retriable failures in Prompt Processing by @kfindeisen in #100
• DM-42027: new stack segfault with libpq by @hsinfang in #102
• DM-42038: Pipeline execution breaks on dimension version mismatch by @kfindeisen in #104
• DM-41730: Exception logs don't have metadata by @kfindeisen in #103
• DM-42063: No task logs in Prompt Processing output by @kfindeisen in #106
• DM-42071: Investigate HSC visit=19710 failure in dev test by @hsinfang in #109
• DM-41591: Include a visit with no template coverage in the upload.py test by @hsinfang in #108
• Document repo migration process. by @kfindeisen in #111
• DM-41713: Many calibs (bias/flat/dark) preloaded in LATISS testing by @kfindeisen in #112
• DM-42035: Remove the DM-42027 workaround on openssl pin by @hsinfang in #110
• DM-42357: Use the new postgres users in dev by @hsinfang in #113

New Contributors

• @kfindeisen made their first contribution in #1
• @ktlim made their first contribution in #3
• @ebellm made their first contribution in #5
• @parejkoj made their first contribution in #6
• @hsinfang made their first contribution in #33

Full Changelog: https://github.com/lsst-dm/prompt_processing/commits/v1.0.0