Releases: lsst-dm/prompt_processing
2.4.0
This release is based on Science Pipelines d_2024_03_29
and rubin-env 8.0.0, and includes fixes for several critical bugs in w_2024_13
. It was published for use in Ops Rehearsal 3.
This release is compatible with:
- Next Visit Fan Out v2.x
- APDB schema 0.1.x and ApdbSql 0.1.x
- Butler repositories with dimensions-config versions
daf_butler
3 to 7.
This release distributes alerts with schema 6.1.
What's Changed
- DM-43584: Add an option to turn off caching entirely by @kfindeisen in #149
Full Changelog: 2.3.0...2.4.0
Service Image
2.3.0
This release is based on Science Pipelines d_2024_03_26
and rubin-env 8.0.0. Its primary purpose is to allow testing of alert distribution ahead of Ops Rehearsal 3.
This release is compatible with:
- Next Visit Fan Out v2.x
- APDB schema 0.1.x and ApdbSql 0.1.x
- Butler repositories with dimensions-config versions
daf_butler
3 to 6.
This release distributes alerts with schema 6.1.
What's Changed
- DM-43445: Camera load in Prompt Processing assumes a single unbounded collection by @kfindeisen in #141
- DM-43502: Prompt Processing should get expId from ingest, not file path by @kfindeisen in #145
- DM-42739: Configure Prompt Processing production service for Sasquatch upload by @kfindeisen in #142
- DM-43488: Concurrency errors in chain handling interrupt processing by @kfindeisen in #146
- Tickets DM-40415: Add doPackageAlerts and doProduceAlerts to ApPipe.yaml by @bsmartradio in #137
- DM-43557: Export dimensions from Prompt Processing only if they are undefined by @kfindeisen in #147
New Contributors
- @bsmartradio made their first contribution in #137
Full Changelog: 2.2.2...2.3.0
2.2.2
This release is based on Science Pipelines w_2024_12
and rubin-env 8.0.0. It fixes a compatibility bug with version 6 Butler repositories in versions 2.2.0 and 2.1.0.
This release is compatible with:
- Next Visit Fan Out v2.x
- APDB schema 0.1.x and ApdbSql 0.1.x
- Butler repositories with dimensions-config versions
daf_butler
3 to 6.
What's Changed
- DM-43472: Prompt Processing version 2 not compatible with Butler dimensions-config 6 by @kfindeisen in #144
Full Changelog: 2.2.1...2.2.2
Service Image
2.2.1
This release is based on Science Pipelines w_2024_12
and rubin-env 8.0.0. It attempts to fix a compatibility bug with version 6 Butler repositories in versions 2.2.0 and 2.1.0, but the fix is incorrect and should not be used.
This release is compatible with:
- Next Visit Fan Out v2.x
- APDB schema 0.1.x and ApdbSql 0.1.x
- Butler repositories with dimensions-config versions
daf_butler
3 to 6.
What's Changed
- DM-43472: Prompt Processing version 2 not compatible with Butler dimensions-config 6 by @kfindeisen in #143
Full Changelog: 2.2.0...2.2.1
Service Image
2.2.0
This release is based on Science Pipelines w_2024_12 stack and rubin-env 8.0.0. New instrument LSSTComCamSim
is now supported. This release is intended for integration testing ahead of Ops Rehearsal 3.
This release is compatible with:
- Next Visit Fan Out v2.x
- APDB schema 0.1.x and ApdbSql 0.1.x
- Butler repositories with dimensions-config versions daf_butler 3 to 6. [Known issue when working with version 6; fixed on 2.2.2]
What's Changed
- DM-43205: Calibs fail to locally certify in Prompt Processing by @kfindeisen in #135
- DM-42726: Implement upload.py LSSTComCamSim for dev test by @hsinfang in #131
- DM-43317: Use prompt-service as sole package for Prompt Processing by @kfindeisen in #138
- DM-43188: Add a nightly chained collection to Prompt Processing output by @kfindeisen in #136
- DM-43394: Set up AP pipeline variant that omits forced-source loading by @kfindeisen in #139
Full Changelog: 2.1.0...2.2.0
2.1.0
This release is based on Science Pipelines w_2024_10
stack and rubin-env 8.0.0. The Science Pipelines changes since d_2024_02_23
include switching to the CalibrateImage
task and butler dimensions-config version 6 that supports group and day_obs as dimensions. The prompt-service has not used the butler dimensions-config version 6 features yet.
This release is compatible with:
- Next Visit Fan Out v2.x
- APDB schema 0.1.x and ApdbSql 0.1.x
- Butler repositories with dimensions-config versions daf_butler 3 to 6. [Known issue when working with version 6; fixed on 2.2.2]
What's Changed
- DM-41915: Preload calls to queryDatasetAssociations take almost a minute each by @kfindeisen in #129
- DM-42997: Separate APDBs for different instruments in Prompt Processing by @kfindeisen in #130
- DM-40388: Switch AP pipelines to use new CalibrateImageTask by @parejkoj in #127
- DM-43200: Prompt processing unit tests don't work with group dimensions by @kfindeisen in #133
- DM-43182: Prompt Processing GitHub Actions don't account for test-only PRs by @kfindeisen in #134
Full Changelog: 2.0.0...2.1.0
2.0.0
This is the first Prompt Processing release created with the mature release management GitHub action; it consolidates the many changes since v1.0.0. The most important change is requiring instrument
and startTime
in incoming nextVisit
messages, which breaks compatibility with version 1 of Next Visit Fan Out.
This release is based on Science Pipelines d_2024_02_23
and rubin-env 8.0.0. The Science Pipelines changes since d_2023_12_19
include real/bogus analysis, APDB versioning, and efficiency improvements.
This release is compatible with:
- Next Visit Fan Out v2.x
- APDB schema 0.1.x and
ApdbSql
0.1.x - Butler repositories with dimensions-config versions
daf_butler
3 to 5
What's Changed Since prompt-service:w_2024_08
- DM-42725: Make ApPipe run with LSSTComCamSim by @hsinfang in #124
- Add compatibility guidelines to release process by @kfindeisen in #128
What Else Has Changed Since v1.0.0
- DM-42469: Update testers and fan-out for the nextVisit schema change by @hsinfang in #114
- DM-42227: Switch from datetime to astropy.time in MiddlewareInterface by @kfindeisen in #116
- DM-33453: Update pipeline references in response to RFC-775. by @erinleighh in #117
- DM-40823: Explore methods for calculating Prompt Processing preload timing by @kfindeisen in #107
- DM-42710: Change to use goodSeeingCoadd as the LATISS templates by @hsinfang in #118
- DM-42809: Add more detailed logging of calib preload by @kfindeisen in #121
- DM-41154: Do not skip missing files silently in Butler transfers by @kfindeisen in #122
- DM-40418: Release Management GitHub Action by @dspeck1 in #115
- Formatting fixes to Playbook by @kfindeisen in #123
- DM-41241: Create a new dev butler central repo in s3://rubin-pp-dev-users and move there by @hsinfang in #120
- DM-42220: Incorporate ModelPackage Butler datasets into Prompt Processing by @kfindeisen in #125
- DM-42618: Add startTime to the nextVisit schema in prompt processing by @hsinfang in #126
New Contributors
- @erinleighh made their first contribution in #117
- @dspeck1 made their first contribution in #115
Full Changelog: v1.0.0...2.0.0
Service Image
v1.0.0
This is a reference release created for testing version support. It is based on Science Pipelines d_2023_12_19
, which uses rubin-env 7.0.1, to avoid a crash introduced with rubin-env 8.0.0 (see DM-42543).
Compatible with Butler repositories with dimensions-config daf_butler
3-4.
What's Changed
- DM-33918: Create code repository for prompt processing by @kfindeisen in #1
- DM-33938: Rename prompt-prototype by @kfindeisen in #2
- Add a playbook section on databases. by @ktlim in #3
- add Cloud Shell instructions by @ebellm in #5
- DM-33935: Create basic test framework for prompt-prototype by @parejkoj in #6
- DM-33936: Create GHAs to build containers and run unit tests. by @ktlim in #4
- DM-33939: Set up an APDB instance on GCP by @ebellm in #7
- DM-34018: Move activator service one level down. by @ktlim in #10
- DM-33931: Improve logging by @kfindeisen in #9
- DM-33974: Create a "Butler worker" VM on Google by @kfindeisen in #11
- DM-34031: Add script for uploading raws. by @kfindeisen in #12
- DM-33996: Use the butler API and mock data for ingest tests by @parejkoj in #8
- DM-33937: Setup a butler with real data on google cloud by @kfindeisen in #13
- DM-33970: Use real images in upload.py by @kfindeisen in #15
- DM-34068: Implement Prompt Prototype's data transfer to workers by @parejkoj in #14
- DM-34123: Use hashable Visit in upload.py by @ebellm in #16
- DM-34132: Use shared visit.py with pure python types by @ebellm in #17
- DM-34134: Reintegrate activator with upload.py by @kfindeisen in #18
- DM-34017: Use the butler API and mock data for run_pipeline tests by @ebellm in #19
- DM-34135: Reintegrate activator with MiddlewareInterface by @kfindeisen in #20
- Update instructions for creating a Cloud Run service by @kfindeisen in #21
- DM-35052: Fix single visit restriction for Prompt Processing by @kfindeisen in #22
- DM-35051: Allow Prompt Processing to set up a local repo with existing files by @kfindeisen in #23
- DM-35941: Remove use of implicit dimensions in Prompt Prototype by @kfindeisen in #25
- DM-34020: Get Google Cloud to detect logging levels in prompt_prototype by @kfindeisen in #26
- DM-35053: Implement syncing of output products for Prompt Processing by @kfindeisen in #24
- DM-36485: Update playbook to give accurate description of upload.py by @kfindeisen in #28
- DM-36080: Separate GCP-specific code in Prompt Processing prototype by @kfindeisen in #27
- Fix oversights in envvar descriptions by @kfindeisen in #29
- Update GHA workflows by @kfindeisen in #30
- DM-36509: Set up test Prompt Processing central repository at USDF by @kfindeisen in #31
- Include S3 user profile in bucket examples. by @kfindeisen in #32
- DM-36602: Move from NCSA+GCP to USDF by @hsinfang in #33
- DM-36712: Migrate activator.py to use Kafka/Knative by @kfindeisen in #34
- DM-36720: migrate upload.py to USDF by @hsinfang in #35
- DM-36995: Remove test_upload.py temporarily by @hsinfang in #36
- DM-36999: Complete integration test at USDF by @kfindeisen in #38
- DM-36791: Use unique IDs in upload.py output by @kfindeisen in #39
- DM-37258: Logs getting dropped in prompt processing by @kfindeisen in #42
- DM-37068: Clear out conflicting datasets before prompt processing runs by @kfindeisen in #41
- DM-37070: tester script using HSC-RC2 data at /repo/main by @hsinfang in #40
- DM-37382: fix repeating raw image by @hsinfang in #43
- DM-37547: HSC exposure ID generator produces invalid IDs in 2023 by @kfindeisen in #46
- DM-37072: Expand HSC calibs and templates in central repo by @hsinfang in #44
- Document the kubectl describe command in the playbook by @kfindeisen in #48
- Add troubleshooting section to playbook by @kfindeisen in #50
- DM-36771: Bring Visit object in line with SAL spec by @kfindeisen in #49
- DM-37759: Rework container build workflows. by @ktlim in #51
- DM-37072: Expand HSC calibs and templates in central repo by @hsinfang in #47
- DM-37751: Add a script to make the export file for templates by @hsinfang in #52
- DM-36586: Use a single output run in Prompt Prototype by @kfindeisen in #45
- DM-37745: Prompt processing preload fails silently if central repo doesn't have prerequisites by @kfindeisen in #53
- DM-38066: Reconcile /repo/embargo as the central repo by @hsinfang in #54
- DM-36162: Upgrade from SimplePipelineExecutor in prompt prototype by @kfindeisen in #56
- DM-38405: Update the nextVisit events produced by the tester upload scripts by @hsinfang in #57
- DM-38752: Output products can go to the wrong collection by @kfindeisen in #58
- DM-38763: ValueError: Requested 'N921' not in DiaPipelineConfig.validBands by @hsinfang in #59
- DM-38269: For LSST cameras, get group/snap from sidecar JSON. by @ktlim in #55
- DM-38597: Add visit_system to WHERE clause. by @ktlim in #62
- DM-38594: Add image timeout heuristic using script duration. by @ktlim in #60
- DM-38596: Get bucket_topic from env var. by @ktlim in #61
- DM-39030: misleading ARGs in Dockerfile.activator by @hsinfang in #63
- DM-39282: Filter out JSON notifications. by @ktlim in #65
- DM-39331: Reduce number of messages consumed. by @ktlim in #66
- DM-39364: Reduce notification messages consumed per call to 1. by @ktlim in #67
- DM-39395: Use day_obs in run names by @kfindeisen in #68
- DM-38225: Intermittent init-output collision in Prompt Processing by @kfindeisen in #69
- DM-39653: Prompt processing unit tests have bitrotted by @kfindeisen in #70
- DM-39188: Export raws only if they are not already in the central repo by @hsinfang in #64
- DM-37253: Make Prompt Processing service configurable by @kfindeisen in #71
- DM-39678: Allow Prompt Prototype to be configured with no pipeline by @kfindeisen in #72
- DM-39681: Add LATISS pipelines to ap_pipe to be used by prompt processing by @hsinfang in #73
- DM-39790: Try to consume right after Kafka subscription. by @ktlim in #74
- DM-39655: Use a list of "fallback" pipelines in Prompt Processing by @kfindeisen in #75
- DM-40225: Update testers and dev env to use the new bucket credential by @hsinfang in #76
- DM-40210: Guard against future ap_pipe changes by @kfindeisen in #78
- DM-40296: Add a timestamp ...