v0.42.0

New Features and Major Changes

• Add basic filtering to spk ls when listing packages by @dcookspi in #1005
• Add --no-publish/--no option to spk-convert-pip by @dcookspi in #1015
• Add -v flags to spk-convert-pip and update spk convert to pass verbosity through by @dcookspi in #1019
• Change any hyphens in SPK_OPT_* env var names by @jrray in #1031
• Change license metadata semantics by @rydrman in #1073

Fixes and Minor Changes

• Add installation, setup and full config description/spec by @rydrman in #1004
• Mark NameOrCurrent as an untagged enum by @jrray in #1012
• Add test for graph::Manifest conversion from tracking::Manifest by @dcookspi in #1014
• Bugfix: implements counting hardlinks for directories in spfs' fuse by @dcookspi in #1021
• Do not clean renders when not supported by @rydrman in #1018
• Partly remove the notion of entry sizes for non-blobs by @jrray in #1001
• Expand the templating language used by pinning by @jrray in #1024
• Store the solution's 'requested by' and 'repo name' data in an spfs annotation for spk to use by @dcookspi in #837
• Update to Rust 1.78.0 by @jrray in #1027
• Remove generated code from source control by @jrray in #1028
• Fix dead code warning depending on enabled features by @jrray in #1029
• Add --allow-with-existing-label to publish by @dcookspi in #992
• Add parallelism to searching for builds by @jrray in #970
• Allow requests for "3.10" to be compat with "3.10.10" by @jrray in #1033
• Use MS_SLAVE instead of MS_PRIVATE when starting a new namespace by @rydrman in #1025
• Adds --stop-on-block/--stop option to solver by @dcookspi in #1016
• Add Version::to_storage_string by @jrray in #1032
• Fix improper capturing of mask into package by @jrray in #1037
• Add --step-on-decision to the solver by @dcookspi in #1038
• Use a single http client and pool in rpc repo by @rydrman in #1041
• Disable annotations when using Legacy Encoding by @dcookspi in #1035
• Fix poor performance of to_tracking_manifest by @jrray in #1047
• Fix spfs-fuse task panicking on mask entry by @jrray in #1048
• Minor doc improvements by @rydrman in #1050
• Ensure edits saved into platforms are still present when run later by @dcookspi in #1045
• Spfs fuse and unknown object by @jrray in #1054
• Rework storage tests to test both encoding formats by @jrray in #1052
• Fix install.environment silently ignoring configuration by @jrray in #1053
• Update documentation links to be relative by @jrray in #1051
• Fix install.environment not expanding some vars by @jrray in #1055
• config: expand tilde ~/paths in repository paths by @davvid in #1056
• Require universal filesystem access for spfs-enter by @rydrman in #1058
• Add ability to commit a platform by refs by @paxsonsa in #676
• Fix rpc error when encoding default is legacy by @jrray in #1046
• Walk the path backwards when creating new directories by @rydrman in #1057
• Updates console and indicatif dependencies to fix progress bar clearing issue by @dcookspi in #1068
• Fix "and empty stream" in docs by @jrray in #1076
• Update Repository URL, Copyright Statements by @rydrman in #1071
• Fix deprecation warnings in latest cargo version by @rydrman in #1080
• Use Cow to avoid annotation cloning by @rydrman in #1075
• Stop using .completed file for renders by @rydrman in #1072
• spk-convert-pip updates for handling open3d by @jrray in #1078
• Fix test selector not handling "distro" (or other var opts) by @jrray in #1077
• Remove imageworks-specific sentry configuration by @jrray in #1079
• Stop spfs runtime prune from deleting durable runtimes by @dcookspi in #1063
• Add heading to the "Installed Packages:" list by @dcookspi in #1062
• Fix make install not installing spfs binaries by @rydrman in #1082
• Update for Release by @rydrman in #1083

Full Changelog: v0.40.0...v0.42.0

v0.40.0

New Features and Major Changes

• Adds --keep-runtime to spfs edit for changing the runtime into a durable one by @dcookspi in #859
  Durable runtimes ensure that the changes on a runtime are not lost when the runtime exists, and allows them to be reused later on in a new runtime
• Initial Windows Runtime Support by @rydrman in #880
  Allows for mounting and reading spfs environments on Windows via WinFSP
• Pinned Repositories (At a Time in the Past) by @rydrman in #538
  Adds support for locking a repository to a time in the past and viewing/resolving environments as they were at that time
• Add live layer support to spfs and spk by @dcookspi in #814
  Live layers are runtime layers that are backed by a directory outside of spfs, this is only supported in the overlayfs mount backend on Linux
• Package Validation Revamp (+ Allow Builds To Change Their Own Files) by @rydrman in #911
  Changes the way that spk packages are validated and how that validation is controlled in the spk spec file. Crucially, it ensures that packages in a recursive build environment are allowed to change their own files.
• Platform spec for meta-packages by @jrray in #873
  Adds a v0/platform spec file format that allows more natural creation of platform-like packages
• Replace liquid with tera for templating by @rydrman in #953
• Add tag namespace concept by @jrray in #715
  Tag namespaces can be used to globally segregate tags in a shared spfs repository
• Flatbuffers for Spfs Objects by @rydrman in #938
  Flatbuffers are now used as the internal and default on-disk format for spfs objects, removing the need for deserialization and improving our ability to develop the format over time

Fixes and Minor Changes

• Fixed EnvOp from panicing when unknown config is found. by @nicholy in #864
• Add depth numbers to 'TRY' lines, and put solver time in "Took too long..." warnings by @dcookspi in #881
• Add support for durable upperdirs and durable runtimes (--keep_runtime) by @dcookspi in #754
• spk info fix for missing version recipes in older repos by @dcookspi in #858
• Expose all env ops publically by @rydrman in #888
• Fix pins in components not being rendered by @jrray in #886
• Deny circular dependencies when building packages by @jrray in #850
• Support combining OverlayFsWithFuse and ProxyRepository by @jrray in #891
• Workaround hang when running spfs-fuse by @jrray in #893
• Changes ExcludeAll and IncludeAll prerelease policy merges to result … by @dcookspi in #840
• spk: add a "completion" sub-command for generating shell completions by @davvid in #901
• Bugfix for 'ps_startup_file' field stopping all runtimes loading in sites with existings spfs installs by @dcookspi in #903
• Add ESRCH to ignore list in find_other_processes by @jrray in #899
• Stop using now when fuse returns file attributes by @jrray in #904
• Make RemoteConfig fields public by @rydrman in #905
• Capture metadata from external executable by @nicholy in #855
• Fix missing closing '_' character in documentation by @jrray in #906
• Fix test_validate_build_changeset_collected by @jrray in #907
• Add spfs read breadcrumb showing configured repo by @jrray in #909
• Generalize and document the setcap Makefile target by @davvid in #900
• Align spfs/spk version to 0.38.0 by @jrray in #910
• Remove runtime from spk explain by @jrray in #913
• Rust 1.74.0 by @jrray in #914
• Improve error for spk explain <non-existing file>@stage (reopened) by @jrray in #915
• Environment variables from build env not expanded by @nicholy in #896
• Removed tracing message when metadata is successfully updated by @nicholy in #919
• Clarify directionality of layer stacks by @rydrman in #916
• Support double underscores for env overrides by @jrray in #860
• Improve error message when spfs-render fails by @jrray in #716
• Adding host_compat setting to package BuildSpec by @dcookspi in #922
• sentry-miette update for sentry error handling by @dcookspi in #918
• Remove spfs-join from process tree by @rydrman in #931
• Support variants introducing packages with components by @jrray in #816
• Updates for auto_host_vars docs and spk new template by @dcookspi in #934
• Adds '--compare-solvers' option by @dcookspi in #882
• Further updates the for auto_host_var docs by @dcookspi in #935
• Fix config test failing if $SPFS_STORAGE_ROOT is set by @jrray in #937
• Fix relative path to spfs/configuration doc page by @jrray in #936
• Print useful messages when building variants during spk build by @nicholy in #939
• modified error message when variant fails by @nicholy in #940
• Fix stack overflow in package iterator by @jrray in #944
• Fix check not reporting the correct number of repairs by @jrray in #925
• Add embedded packages to impossible request checks by @dcookspi in #883
• Fix test selectors interaction with components by @jrray in #948
• Display variants in json format by @nicholy in #946
• Updates auto_host_vars field so default value doesn't get serialised by @dcookspi in #951
• Fix not being able to read a recipe that has a host var defined with "choices" but no default value by @jrray in #952
• Update stdfs to use new features by @jrray in #955
• Host options scoped options by @jrray in #957
• Update help message for --solver-to-run by @jrray in #928
• Description field for build vars by @nicholy in #954
• Handle packages with differing amounts of trailing zeros by @jrray in #967
• Fixup platforms PR by @jrray in #977
• Change --variant to act as a variant filter by @jrray in #964
• Adds some metadata to spk convert pip generated packages by @dcookspi in #975
• Use relative paths in documentation links by @jrray in #979
• Adds a license value to the metadata for spk convert pip packages by @dcookspi in #978
• Make some recipe elements case insensitive by @jrray in #983
• Add rpc config values for timeout and message size by @rydrman in #985
• Also check payloads when committing by @rydrman in #988
• Add version normalization by @jrray in #972
• Legacy spk version testing by @jrray in #976
• Add feature to disable version normalization for writes by @jrray in #982
• Add descriptions to strong inheritance vars by @rydrman in #996
• Make spfs diff only check sizes on blobs by @jrray in #999
• spk-convert-pip updates for rocky 9 support by @jrray in #1000
• Bugfix: spk test package.spk.yaml@install was not using host options in solves by @dcookspi in #1002
• Add flag to follow refs in info command by @rydrman in #994
• Update rpm spec files and docker image for el9 by @rydrman in #998
• Add extra/external annotation data to spfs runtimes by @dcookspi in #815
• Use flatbuffers in grpc instead of protobuf fields by @rydrman in #1010

Full Changelog: v0.38.0...v0.40.0

v0.38.0

This public release has been a long-time coming, with a huge number significant changes and improvements being added in the last year!

New Features and Major Changes

• Combine with spfs by @rydrman in #440
  The https://github.com/imageworks/spfs repository has been archived and the code migrated into this one for easier build/test and release processes.

• Fuse Backend (First Pass) by @rydrman in #657
  Spfs and spk can now be run using a FUSE filesystem backend to avoid the use of file localization and OverlayFS. OverlayFS can be combined with FUSE in order to still allow runtime editing and package creation. Use the spfs config setting mount.backend with FuseOnly or FuseWithOverlayFS to enable.

• Spec File Versioning via Traits by @rydrman in #327
  To better support breaking changes in the spec file format, we introduce an api version that should be specified in all spec files moving forward.

• Better Source Package Handing by @rydrman in #428

• Further enhancements to spfs clean by @jrray in #454

• Better Yaml Errors by @rydrman in #499

• Add filter mode for components file filtering by @jrray in #518

• Support pip converting packages that use a zip file by @jrray in #556

• Add filter concept for when collecting files after build by @jrray in #559

• Adjust warning/error behavior of spk rm by @jrray in #585

• Add 'package/version@build' support to more commands by @dcookspi in #574

• Add spfs runtime prune by @jrray in #610

• Add a simple spk lint command by @rydrman in #600
  This will be expanded in the future, but aims to allow local and CI validation as well as provide a place to encode best-practices and warnings for possible gotchas.

• Adding Impossible request checking to the solver by @dcookspi in #506
  Improves the time-to-fail in some common resolves where the requests can be identified as impossible. Spk now runs a solver with and without these checks at the same time and let's them race. In cases where this happens the resolve fails much sooner, but it can also slow down legitimately complex resolves that are possible.

• Refactor check command to allow for repairs of subtrees by @paxsonsa in #602

• Buffered, Compressed Uploads and Downloads by @rydrman in #640

• Use file descriptors in render process, add progress bars by @rydrman in #641

• Spfs ls-tags --recursive by @rydrman in #644

• Add a debug cli command by @jrray in #524
  This command can be used to install source packages for existing packages in an environment when debugging issues with the software.

• Adds search space size calculations with the --show-search-size option by @dcookspi in #620

• Add an Spk Config File by @rydrman in #661

• Adds 'statsd' feature to send statsd metrics from spk by @dcookspi in #565

• Package Templating (va Liquid) by @rydrman in #513

• Extend "@stage" syntax to allow variant to be specified by @jrray in #493

• Add reporting of render statistics by @jrray in #668

• Don't use expect when launching tcsh/csh by @rydrman in #700

• Allow for custom shell messages by @rydrman in #706

• Add --check flag to recover from missing objects by @rydrman in #725

• Add --pid flag to spfs-join by @jrray in #734

• Change --local-repo short flag to -L by @jrray in #751

• Spfs ls upgrade by @nicholy in #689

• Spk du cmd by @nicholy in #774

• Adds 'spfs info /spfs/file/path' support by @dcookspi in #687

• Support conditional install requirements by @jrray in #849

• Make 'spk env package.yaml@build' start an editable /spfs environment by @dcookspi in #842

• Fix illegal env vars by @krishaiquin in #793

Fixes and Minor Changes

• Fix spk test package@build failing by @jrray in #435
• Add "variant-count" alias for num-variants command by @jrray in #438
• Add --tmpdir flag to spfs-enter by @jrray in #448
• Fix failed to read recipe by @rydrman in #453
• Make spk rpm obsolete & replace spfs one by @rydrman in #455
• Change clean_render to clean all users by @jrray in #449
• Sentry context and message tweaks by @dcookspi in #446
• Fix renders having wrong ownership / mode by @jrray in #471
• Improve error messages for IO-related errors by @jrray in #461
• Bugfix for /src package requests with empty component lists by @dcookspi in #474
• Add a way to track the owner of an embedded package by @jrray in #415
• Use --no-runtime flag to avoid recursive runtime creation by @jrray in #483
• Change PkgOpt::validate to use intersects by @jrray in #498
• Add defaults for all metadata fields by @rydrman in #482
• Adding a display limit and additional collation for frequent errors in end of solve report by @dcookspi in #501
• Import Configuration and Efficiency by @rydrman in #487
• Fill out incomplete error message by @rydrman in #521
• Fix bug where static values conflicted with defaults by @rydrman in #522
• Change test additional requirements request order by @jrray in #523
• Experimental solver progress UI by @jrray in #374
• Bugfixes for empty package layers and components in spk bake output by @dcookspi in #510
• Add cap_dac_override to spfs-clean by @jrray in #542
• Fix CompatRange::is_satisfied_by wrong Compatible case by @jrray in #550
• Update spk's spfs repository caches to use DashMap by @dcookspi in #535
• Add retry for copy renders in spfs by @rydrman in #569
• Remove redundant solve in binary build by @rydrman in #557
• Handle "object is not a blob" in spfs clean by @jrray in #578
• Set sync policy when syncing tag in spk-launcher by @jrray in #554
• Improve performance of flattening layers by @jrray in #579
• Fix bug parsing envop with separator field by @rydrman in #555
• Allow components to use undeclared default ones by @rydrman in #558
• Add post rule to rpm to mkdir /spfs by @jrray in #584
• Use daemon when launching spfs-monitor by @jrray in #588
• Fix spfs-clean giving up too early by @jrray in #582
• Track flattened layers in runtime by @jrray in #580
• Stop trying to delete the tags or spfs root directory when deleting tags by @jrray in #597
• search bug-fixes for deleted and deprecated packages by @dcookspi in #572
• Separate Ident Types for Different Specificities by @rydrman in #530
• Add mount namespace to runtime data by @jrray in #593
• Report out of space errors with a custom message by @jrray in #612
• Add contextual info to StorageReadError & StorageWriteError by @jrray in #606
• Pass --python-version when pip converting packages by @jrray in #619
• Bugfix for missing build info for "(requested by..." etc lines in solver output by @dcookspi in #625
• Add logo to website and README by @rydrman in #627
• Use exact version matching for @source by @rydrman in #633
• Simplify Solution Type by @rydrman in #599
• Embedded package RequestedBy improvements by @dcookspi in #634
• F...

v0.36.0

Changes

• Introduce the spk launcher for version rollout workflows (#264)
• update compat-based version requests to not expand all version numbers to three numbers. This means that a request for python/2 is no longer the same as a request for python/2.0.0, and can resolve as binary-compatible with any 2.x version (#286)
• Add an == operator for exact version requests down to the pre and post release tags (#295)
• Enable the local repository by default in all commands (#299)
• Update the output of spk ls --recursive to be sorted (#303)
• Improve wording and clarity in incompatibility messages and some /src related solve error messages (#308)
• Display initial (command line) request details when verbosity > 1 (#309)
• Add solver statistics and improve output when interrupted by the user (#311)
• Add counters and rates to solver statistics for greater insight (#312)
• Track what a request was 'requested by' and display it when relevant (#313)
• Add extra data and support additional output formats for spk bake (#331)
• Introduce two solver heuristic changes to improve solve performance (#334)
• Output the solution packages from solvers run by 'make-binary' (#365)
• Allow remote name to be specified directly in package requests (#240)
• Do not allow leading dash in pkg name (#332)
• Require non-numeric tag names (#370)
• Skip over missing builds in spk info (#403)
• Update ls behavior to be more like it was before removal of -l (#392)
• Handle publishing a build individually of a new version (#429)

Fixes and Updates

• Revamp documentation to build and deploy to getspk.io (#172)
• Add support for sentry via a compile-time feature flag (#293)
• Fix failing tests when the tmpdir is symlinked (#307)
• Add a formal type to describe package and option names (#316, #317)
• Convert spk codebase to async (#345)
• Introduce file I/O caching layer for storage::spfs (#350)
• Fix "=1.0" not being satisfied by pkg 1.0.0 (#357)
• Improve version range intersection logic to reduce solver search space (#337, #338, #343)
• Convert errors to use thiserror::Error (#361)
• Enable lto when building for release (#369)
• Add CI workflow to capture and report code coverage (#366)
• Variety of solver optimizations to improve performance (#377)
• Handle (ignore) invalid packages in solve (#380)
• Allow host options to be overridden (#388)
• Add sentry capture for errors, tracing messages, and long solves (#394)
• Bring back the ability for options to specify builds (#395)
• Allow Binary:1.2.3 and =1.2.3 requests to merge (#402)
• Fix spk ls failing if there is an invalid spec (#413)
• Add a name! macro to generate name types (#418)
• Fix codecov.io file viewer not working for spk project (#420)
• Stop creating a test runtime with the real "origin" repo (#419)
• Check that --variant arg is valid for better error messaging (#424)

Full Changelog: v0.31.0...v0.36.0

v0.31.0

Features and Changes

• package components: packages can now divide their files into different components, which helps to ensure that downstream consumers can get what they need in different contexts without also pulling in things that they don't. A great example of this is the default build and run components, which allow build-only files such as header files to be available at build time but not need to be pulled in at run-time.
• package metadata: you can now add key metadata fields and arbitrary labels to package specs
• allow requesting a specific compatibility level in package requests (eg python/API=2.7, or boost/Binary=1.70)
• support compatibility level only in fromBuildEnv for package install requirements (eg - {pkg: python, fromBuildEnv: API}) which generates a request for something API compatible with whatever was used at build time.
• support fromBuildEnv: true as simple alias for fromBuildEnv: Binary and the most reasonable canonical case
• update pip conversion process to prefix generated packages with python- so that they do not conflict with non-python libraries of the same name
• Formalize the package validation process. The new framework is extensible for the future and allows packages to opt-out of validators that are not relevant to them.
• Support for specifying environment variables directly in package specs, and auto-generating spfs startup scripts (#17)
• Convert the codebase to Rust milestone
• Support for a server-based repository using the latest spfs grpc backend (this implementation is solid, but does not handle error states very well, contributions to help build this out in spfs are very welcome)

Bug Fixes

• non-string scalars could not be provided on the command line (regression)
• solver was not processing some options when version conflicts arose during the resolve (regression)
• error messages sometimes ended up with repeated messages (from rust)
• spk test command was not actually running tests (regression)
• spk install with --save flag was not actually saving the change to the local spec file (regression)
• solver state hashing was updated to be incremental, reducing the need to rehash as often (#236)
• ensure that qualified var requests do not conflict with global ones (#284)
• fix bug where invalid package names were being allowed

** most of these regressions were caused by untested areas of code that no longer functioned after the conversion to rust

Package Fixes

• fix issue where perl could fail because sources come from a tarball
• fix issue where python2 package could fail to build on distros where the default python command is python 3
• update mpc version used for gcc
• fix the make packages.docker target to succeed in building gnu gcc and necessary dependencies to get to a python build under centos7

v0.30.0

Changes

• solver is now binary-only by default on the command line. The --binary-only flag has been replaced by --allow-builds
• convert api module to Rust, remaining entirely backwards compatible for now
• update to the latest spfs and ensure that spk can be built with rust 1.53

v0.29.2

Features

• Update source collection process to set build environment for scripts (#72)

Fixes

• Fix tests not resolving the right binary package when build vars are used (#73)
• Update to latest spfs patch release

Packages

Some of these may still be imperfect, but useful notheless

• cmake
• LLVM+Clang 12
• imath
• openexr
• opencolorio
• pybind11

v0.29.1

Fixes

• Allow local source rsync filter to be overridden/modified in spec file
• Update to latest spfs API

v0.29.0

Packages

• add an initial set of package recipes to get bootstrapped in a clean environment

Changes

• update export command to allow a filename to be specified (making it more useful for automation)
• add the ability to run spk info spec.yaml@source to get source package information (if any)

Fixes

• fix output of help messages, and make them more clear
• silence warnings logged by the sentry SDK that can no longer connect to the internal Imageworks instance

Infrastructure

• add GitHub ci code for running lint, test and generating rpm file