Skip to content

Releases: lsst-dm/prompt_processing

2.4.0

29 Mar 19:15
Compare
Choose a tag to compare

This release is based on Science Pipelines d_2024_03_29 and rubin-env 8.0.0, and includes fixes for several critical bugs in w_2024_13. It was published for use in Ops Rehearsal 3.

This release is compatible with:

  • Next Visit Fan Out v2.x
  • APDB schema 0.1.x and ApdbSql 0.1.x
  • Butler repositories with dimensions-config versions daf_butler 3 to 7.

This release distributes alerts with schema 6.1.

What's Changed

  • DM-43584: Add an option to turn off caching entirely by @kfindeisen in #149

Full Changelog: 2.3.0...2.4.0

Service Image

prompt-service@454de5c

2.3.0

28 Mar 23:37
Compare
Choose a tag to compare

This release is based on Science Pipelines d_2024_03_26 and rubin-env 8.0.0. Its primary purpose is to allow testing of alert distribution ahead of Ops Rehearsal 3.

This release is compatible with:

  • Next Visit Fan Out v2.x
  • APDB schema 0.1.x and ApdbSql 0.1.x
  • Butler repositories with dimensions-config versions daf_butler 3 to 6.

This release distributes alerts with schema 6.1.

What's Changed

  • DM-43445: Camera load in Prompt Processing assumes a single unbounded collection by @kfindeisen in #141
  • DM-43502: Prompt Processing should get expId from ingest, not file path by @kfindeisen in #145
  • DM-42739: Configure Prompt Processing production service for Sasquatch upload by @kfindeisen in #142
  • DM-43488: Concurrency errors in chain handling interrupt processing by @kfindeisen in #146
  • Tickets DM-40415: Add doPackageAlerts and doProduceAlerts to ApPipe.yaml by @bsmartradio in #137
  • DM-43557: Export dimensions from Prompt Processing only if they are undefined by @kfindeisen in #147

New Contributors

Full Changelog: 2.2.2...2.3.0

2.2.2

24 Mar 16:02
Compare
Choose a tag to compare

This release is based on Science Pipelines w_2024_12 and rubin-env 8.0.0. It fixes a compatibility bug with version 6 Butler repositories in versions 2.2.0 and 2.1.0.

This release is compatible with:

  • Next Visit Fan Out v2.x
  • APDB schema 0.1.x and ApdbSql 0.1.x
  • Butler repositories with dimensions-config versions daf_butler 3 to 6.

What's Changed

  • DM-43472: Prompt Processing version 2 not compatible with Butler dimensions-config 6 by @kfindeisen in #144

Full Changelog: 2.2.1...2.2.2

Service Image

prompt-service@87ceee2

2.2.1

23 Mar 17:33
Compare
Choose a tag to compare

This release is based on Science Pipelines w_2024_12 and rubin-env 8.0.0. It attempts to fix a compatibility bug with version 6 Butler repositories in versions 2.2.0 and 2.1.0, but the fix is incorrect and should not be used.

This release is compatible with:

  • Next Visit Fan Out v2.x
  • APDB schema 0.1.x and ApdbSql 0.1.x
  • Butler repositories with dimensions-config versions daf_butler 3 to 6.

What's Changed

  • DM-43472: Prompt Processing version 2 not compatible with Butler dimensions-config 6 by @kfindeisen in #143

Full Changelog: 2.2.0...2.2.1

Service Image

prompt-service@81dabd2

2.2.0

21 Mar 21:23
Compare
Choose a tag to compare

This release is based on Science Pipelines w_2024_12 stack and rubin-env 8.0.0. New instrument LSSTComCamSim is now supported. This release is intended for integration testing ahead of Ops Rehearsal 3.

This release is compatible with:

  • Next Visit Fan Out v2.x
  • APDB schema 0.1.x and ApdbSql 0.1.x
  • Butler repositories with dimensions-config versions daf_butler 3 to 6. [Known issue when working with version 6; fixed on 2.2.2]

What's Changed

  • DM-43205: Calibs fail to locally certify in Prompt Processing by @kfindeisen in #135
  • DM-42726: Implement upload.py LSSTComCamSim for dev test by @hsinfang in #131
  • DM-43317: Use prompt-service as sole package for Prompt Processing by @kfindeisen in #138
  • DM-43188: Add a nightly chained collection to Prompt Processing output by @kfindeisen in #136
  • DM-43394: Set up AP pipeline variant that omits forced-source loading by @kfindeisen in #139

Full Changelog: 2.1.0...2.2.0

2.1.0

07 Mar 17:53
Compare
Choose a tag to compare

This release is based on Science Pipelines w_2024_10 stack and rubin-env 8.0.0. The Science Pipelines changes since d_2024_02_23 include switching to the CalibrateImage task and butler dimensions-config version 6 that supports group and day_obs as dimensions. The prompt-service has not used the butler dimensions-config version 6 features yet.

This release is compatible with:

  • Next Visit Fan Out v2.x
  • APDB schema 0.1.x and ApdbSql 0.1.x
  • Butler repositories with dimensions-config versions daf_butler 3 to 6. [Known issue when working with version 6; fixed on 2.2.2]

What's Changed

  • DM-41915: Preload calls to queryDatasetAssociations take almost a minute each by @kfindeisen in #129
  • DM-42997: Separate APDBs for different instruments in Prompt Processing by @kfindeisen in #130
  • DM-40388: Switch AP pipelines to use new CalibrateImageTask by @parejkoj in #127
  • DM-43200: Prompt processing unit tests don't work with group dimensions by @kfindeisen in #133
  • DM-43182: Prompt Processing GitHub Actions don't account for test-only PRs by @kfindeisen in #134

Full Changelog: 2.0.0...2.1.0

2.0.0

01 Mar 00:32
Compare
Choose a tag to compare

This is the first Prompt Processing release created with the mature release management GitHub action; it consolidates the many changes since v1.0.0. The most important change is requiring instrument and startTime in incoming nextVisit messages, which breaks compatibility with version 1 of Next Visit Fan Out.

This release is based on Science Pipelines d_2024_02_23 and rubin-env 8.0.0. The Science Pipelines changes since d_2023_12_19 include real/bogus analysis, APDB versioning, and efficiency improvements.

This release is compatible with:

  • Next Visit Fan Out v2.x
  • APDB schema 0.1.x and ApdbSql 0.1.x
  • Butler repositories with dimensions-config versions daf_butler 3 to 5

What's Changed Since prompt-service:w_2024_08

What Else Has Changed Since v1.0.0

  • DM-42469: Update testers and fan-out for the nextVisit schema change by @hsinfang in #114
  • DM-42227: Switch from datetime to astropy.time in MiddlewareInterface by @kfindeisen in #116
  • DM-33453: Update pipeline references in response to RFC-775. by @erinleighh in #117
  • DM-40823: Explore methods for calculating Prompt Processing preload timing by @kfindeisen in #107
  • DM-42710: Change to use goodSeeingCoadd as the LATISS templates by @hsinfang in #118
  • DM-42809: Add more detailed logging of calib preload by @kfindeisen in #121
  • DM-41154: Do not skip missing files silently in Butler transfers by @kfindeisen in #122
  • DM-40418: Release Management GitHub Action by @dspeck1 in #115
  • Formatting fixes to Playbook by @kfindeisen in #123
  • DM-41241: Create a new dev butler central repo in s3://rubin-pp-dev-users and move there by @hsinfang in #120
  • DM-42220: Incorporate ModelPackage Butler datasets into Prompt Processing by @kfindeisen in #125
  • DM-42618: Add startTime to the nextVisit schema in prompt processing by @hsinfang in #126

New Contributors

Full Changelog: v1.0.0...2.0.0

Service Image

prompt_processing@cf2523a

v1.0.0

18 Jan 23:01
1c6b97a
Compare
Choose a tag to compare

This is a reference release created for testing version support. It is based on Science Pipelines d_2023_12_19, which uses rubin-env 7.0.1, to avoid a crash introduced with rubin-env 8.0.0 (see DM-42543).

Compatible with Butler repositories with dimensions-config daf_butler 3-4.

What's Changed

  • DM-33918: Create code repository for prompt processing by @kfindeisen in #1
  • DM-33938: Rename prompt-prototype by @kfindeisen in #2
  • Add a playbook section on databases. by @ktlim in #3
  • add Cloud Shell instructions by @ebellm in #5
  • DM-33935: Create basic test framework for prompt-prototype by @parejkoj in #6
  • DM-33936: Create GHAs to build containers and run unit tests. by @ktlim in #4
  • DM-33939: Set up an APDB instance on GCP by @ebellm in #7
  • DM-34018: Move activator service one level down. by @ktlim in #10
  • DM-33931: Improve logging by @kfindeisen in #9
  • DM-33974: Create a "Butler worker" VM on Google by @kfindeisen in #11
  • DM-34031: Add script for uploading raws. by @kfindeisen in #12
  • DM-33996: Use the butler API and mock data for ingest tests by @parejkoj in #8
  • DM-33937: Setup a butler with real data on google cloud by @kfindeisen in #13
  • DM-33970: Use real images in upload.py by @kfindeisen in #15
  • DM-34068: Implement Prompt Prototype's data transfer to workers by @parejkoj in #14
  • DM-34123: Use hashable Visit in upload.py by @ebellm in #16
  • DM-34132: Use shared visit.py with pure python types by @ebellm in #17
  • DM-34134: Reintegrate activator with upload.py by @kfindeisen in #18
  • DM-34017: Use the butler API and mock data for run_pipeline tests by @ebellm in #19
  • DM-34135: Reintegrate activator with MiddlewareInterface by @kfindeisen in #20
  • Update instructions for creating a Cloud Run service by @kfindeisen in #21
  • DM-35052: Fix single visit restriction for Prompt Processing by @kfindeisen in #22
  • DM-35051: Allow Prompt Processing to set up a local repo with existing files by @kfindeisen in #23
  • DM-35941: Remove use of implicit dimensions in Prompt Prototype by @kfindeisen in #25
  • DM-34020: Get Google Cloud to detect logging levels in prompt_prototype by @kfindeisen in #26
  • DM-35053: Implement syncing of output products for Prompt Processing by @kfindeisen in #24
  • DM-36485: Update playbook to give accurate description of upload.py by @kfindeisen in #28
  • DM-36080: Separate GCP-specific code in Prompt Processing prototype by @kfindeisen in #27
  • Fix oversights in envvar descriptions by @kfindeisen in #29
  • Update GHA workflows by @kfindeisen in #30
  • DM-36509: Set up test Prompt Processing central repository at USDF by @kfindeisen in #31
  • Include S3 user profile in bucket examples. by @kfindeisen in #32
  • DM-36602: Move from NCSA+GCP to USDF by @hsinfang in #33
  • DM-36712: Migrate activator.py to use Kafka/Knative by @kfindeisen in #34
  • DM-36720: migrate upload.py to USDF by @hsinfang in #35
  • DM-36995: Remove test_upload.py temporarily by @hsinfang in #36
  • DM-36999: Complete integration test at USDF by @kfindeisen in #38
  • DM-36791: Use unique IDs in upload.py output by @kfindeisen in #39
  • DM-37258: Logs getting dropped in prompt processing by @kfindeisen in #42
  • DM-37068: Clear out conflicting datasets before prompt processing runs by @kfindeisen in #41
  • DM-37070: tester script using HSC-RC2 data at /repo/main by @hsinfang in #40
  • DM-37382: fix repeating raw image by @hsinfang in #43
  • DM-37547: HSC exposure ID generator produces invalid IDs in 2023 by @kfindeisen in #46
  • DM-37072: Expand HSC calibs and templates in central repo by @hsinfang in #44
  • Document the kubectl describe command in the playbook by @kfindeisen in #48
  • Add troubleshooting section to playbook by @kfindeisen in #50
  • DM-36771: Bring Visit object in line with SAL spec by @kfindeisen in #49
  • DM-37759: Rework container build workflows. by @ktlim in #51
  • DM-37072: Expand HSC calibs and templates in central repo by @hsinfang in #47
  • DM-37751: Add a script to make the export file for templates by @hsinfang in #52
  • DM-36586: Use a single output run in Prompt Prototype by @kfindeisen in #45
  • DM-37745: Prompt processing preload fails silently if central repo doesn't have prerequisites by @kfindeisen in #53
  • DM-38066: Reconcile /repo/embargo as the central repo by @hsinfang in #54
  • DM-36162: Upgrade from SimplePipelineExecutor in prompt prototype by @kfindeisen in #56
  • DM-38405: Update the nextVisit events produced by the tester upload scripts by @hsinfang in #57
  • DM-38752: Output products can go to the wrong collection by @kfindeisen in #58
  • DM-38763: ValueError: Requested 'N921' not in DiaPipelineConfig.validBands by @hsinfang in #59
  • DM-38269: For LSST cameras, get group/snap from sidecar JSON. by @ktlim in #55
  • DM-38597: Add visit_system to WHERE clause. by @ktlim in #62
  • DM-38594: Add image timeout heuristic using script duration. by @ktlim in #60
  • DM-38596: Get bucket_topic from env var. by @ktlim in #61
  • DM-39030: misleading ARGs in Dockerfile.activator by @hsinfang in #63
  • DM-39282: Filter out JSON notifications. by @ktlim in #65
  • DM-39331: Reduce number of messages consumed. by @ktlim in #66
  • DM-39364: Reduce notification messages consumed per call to 1. by @ktlim in #67
  • DM-39395: Use day_obs in run names by @kfindeisen in #68
  • DM-38225: Intermittent init-output collision in Prompt Processing by @kfindeisen in #69
  • DM-39653: Prompt processing unit tests have bitrotted by @kfindeisen in #70
  • DM-39188: Export raws only if they are not already in the central repo by @hsinfang in #64
  • DM-37253: Make Prompt Processing service configurable by @kfindeisen in #71
  • DM-39678: Allow Prompt Prototype to be configured with no pipeline by @kfindeisen in #72
  • DM-39681: Add LATISS pipelines to ap_pipe to be used by prompt processing by @hsinfang in #73
  • DM-39790: Try to consume right after Kafka subscription. by @ktlim in #74
  • DM-39655: Use a list of "fallback" pipelines in Prompt Processing by @kfindeisen in #75
  • DM-40225: Update testers and dev env to use the new bucket credential by @hsinfang in #76
  • DM-40210: Guard against future ap_pipe changes by @kfindeisen in #78
  • DM-40296: Add a timestamp ...
Read more