Packaging: Add venv and packs to deb/rpm packages #6328

cognifloyd · 2025-04-07T16:05:06Z

Reviewers: Please respond to the Open Questions section below.

This PR is working towards doing packaging via pantsbuild. Eventually, I hope to archive and stop using st2-packages.git.

Previous PRs in this series:

The pants nfpm backend was activated in Packaging: Inject st2 version into rpm/deb packages (pants-plugins/release) #6321.
Basic BUILD metadata for deb/rpm packages (nfpm_deb_package and nfpm_rpm_package) was added in Packaging: Add basic BUILD metadata for rpm/deb packages #6325 along with the deb maintainer scripts and rpm scriptlets.
Packaging: Add systemd files to rpm/deb packages #6326 added the systemd files (units and generators) to the deb/rpm packages
Packaging: Add directories and symlinks to deb/rpm packages #6327 added directories and symlinks to the deb/rpm packages

This PR adds the virtualenv (by way of st2.pex added in #6307) and packs (by way of pack archives added in #6311). Both st2.pex and the pack archives get extracted by the post install deb maintainer script or rpm scriptlet (added in #6320).

Virtualenv (`st2.pex`)

For the virtualenv, we add an nfpm_content_file target for st2.pex:

st2/packaging/BUILD.venv

Lines 84 to 90 in 850cc78

    
           nfpm_content_file( 
        
               name="st2_venv", 
        
               description="Pex file that system packages can use to generate /opt/stackstorm/st2", 
        
               file_owner="root", 
        
               file_group="root", 
        
               file_mode="rwxr-x---", 
        
               dst="/opt/stackstorm/install/st2.pex",  # or st2-py3*.pex

This target is parametrized to produce a pex for each python minor version we support. Because the parametrization was very repetitive, I added this helper to simplify defining each of them:

st2/packaging/BUILD.venv

Lines 76 to 81 in 850cc78

    
           def _venv_py3(minor: str): 
        
               return parametrize( 
        
                   f"py3{minor}", 
        
                   dependencies=[f":st2.pex@parametrize=py3{minor}"], 
        
                   src=f"st2-py3{minor}.pex",  # relative to this BUILD file 
        
               )

Then, this is how we use that helper to actually parametrize the st2.pex target for each python minor version:

st2/packaging/BUILD.venv

Lines 91 to 94 in 850cc78

    
           **_venv_py3("8"), 
        
           **_venv_py3("9"), 
        
           **_venv_py3("10"), 
        
           **_venv_py3("11"),

Then we add a dependency to the nfpm_content_file target on the nfpm_deb_package target and the nfpm_rpm_package target:

st2/packaging/BUILD

Lines 25 to 31 in 1ebd199

    
           def _st2_venv_deps() -> list[str]: 
        
               """Return a list of deps required to build the st2 venv. 
        
               The :st2_venv parametrizations depend on parametrizations of :st2.pex, 
        
               so, make sure to keep these 3 lists of parametrizations in sync. 
        
               """ 
        
               return [f":st2_venv@parametrize=py3{m}" for m in ("8", "9", "10", "11")]

st2/packaging/BUILD

Line 50 in 1ebd199

*_st2_venv_deps(),

st2/packaging/BUILD

Line 85 in 1ebd199

*_st2_venv_deps(),

I have an open question (below) about how tightly we should couple a python version to an OS. Based on feedback here and in the TSC meeting, we will not couple the OS with the python version. If someone wants to use pyenv or otherwise install a different (typically newer) version of python, they should be able to do that simply without fighting with the system package.

Packs (pack archives)

First, we add an nfpm_content_file target to the st2_pack_archive(...) macro:

st2/pants-plugins/macros.py

Lines 173 to 181 in 850cc78

    
           nfpm_content_file(  # noqa: F821 
        
               name="archive_for_nfpm", 
        
               dependencies=[":archive"], 
        
               src=f"packaging/packs/{pack_name}.tgz.run", 
        
               dst=f"/opt/stackstorm/install/packs/{pack_name}.tgz.run", 
        
               file_owner="root", 
        
               file_group=ST2_PACKS_GROUP, 
        
               file_mode="rwxr-x---", 
        
           )

Then we add some generic targets to make it easier for the deb/rpm packages to depend on all the pack archives.

st2/packaging/BUILD.packs

Lines 17 to 25 in 850cc78

    
           # :archive_for_npfm targets created by st2_pack_archive() macro (see pants-plugins/macros.py) 
        
           target( 
        
               name="default_packs", 
        
               dependencies=[f"//contrib/{pack}:archive_for_nfpm" for pack in _DEFAULT_PACKS], 
        
           ) 
        
           target( 
        
               name="extra_packs", 
        
               dependencies=[f"//contrib/{pack}:archive_for_nfpm" for pack in _EXTRA_PACKS], 
        
           )

The post-install scripts only install the default packs, which are defined in 3 places:

st2/packaging/BUILD.packs

Lines 1 to 8 in 850cc78

    
           # These are packs that should be installed with the default ST2 install. 
        
           _DEFAULT_PACKS = ( 
        
               "chatops", 
        
               "core", 
        
               "default", 
        
               "linux", 
        
               "packs", 
        
           )

st2/packaging/deb/scripts/post-install.sh

Lines 32 to 39 in 850cc78

    
           # The default set of packs installed with st2. 
        
           _ST2_PACKS=" 
        
           chatops 
        
           core 
        
           default 
        
           linux 
        
           packs 
        
           "

st2/packaging/rpm/scripts/post-install.sh

Lines 10 to 17 in 850cc78

    
           # The default set of packs installed with st2. 
        
           _ST2_PACKS=" 
        
           chatops 
        
           core 
        
           default 
        
           linux 
        
           packs 
        
           "

Plus the post-install scripts install the examples pack in /usr/share/doc/st2/examples:

st2/packaging/deb/scripts/post-install.sh

Line 113 in 850cc78

extract_st2_pack examples --target /usr/share/doc/st2/examples || :

st2/packaging/rpm/scripts/post-install.sh

Line 49 in 850cc78

extract_st2_pack examples --target /usr/share/doc/st2/examples || :

So, we need the packs that get installed by default, and the examples pack. The rest of the packs are optional, and have not been made available in older versions of st2. There is an open question (below) about these extra packs:

st2/packaging/BUILD.packs

Lines 10 to 15 in 850cc78

    
           # These are packs that should NOT be installed with the default ST2 install. 
        
           _EXTRA_PACKS = ( 
        
               "debug", 
        
               "examples", 
        
               "hello_st2", 
        
           )

Finally, we add a dependency on the nfpm_content_file targets to the nfpm_deb_package and nfpm_rpm_package targets:

st2/packaging/BUILD

Lines 42 to 43 in 850cc78

    
           ":default_packs", 
        
           ":extra_packs",

st2/packaging/BUILD

Lines 77 to 78 in 850cc78

    
           ":default_packs", 
        
           ":extra_packs",

Open Questions

How tightly should we couple a python version to an OS?
1. ~~We could include only one st2.pex on each OS based on which python version is packaged on that OS.~~
  - In this scenario, the pex gets installed as /opt/stackstorm/install/st2.pex.
  - This is what I've done by using the dependency: :st2_venv@parametrize=py38.
  - This is also the most efficient use of disk space, as only one copy of the pex gets installed, not 4.
2. ✔️ We could include an st2.pex file for each python version we support, on every OS, so that people can easily rebuild the venv when they update to a newer python version.
  - ✔️ In this scenario, we need to modify the _venv_py3(minor) helper function to define dst=f"/opt/stackstorm/install/st2-py3{minor}.pex"
  - In this scenario, we also need to either:
    1. ~~modify the post-install script to pick which pex to use, or~~
    2. ✔️ add a symlink for the default python version as /opt/stackstorm/install/st2.pex -> /opt/stackstorm/install/st2-py38.pex. Maybe we can record this symlink as a config|noreplace file so that people can modify their preferred python version.
  - This would allow people to update to newer python versions using something like pyenv or a ppa to install a newer version of python than the OS packages provide.
Should we install the pack archives for all of the optional extra packs?
- The examples pack is not optional, as we install it in /usr/share/doc/st2/examples.
- Making the debug pack easier to install would be great, so I'd like to include its pack archive.
- Should we include the hello_st2 pack archive? This is the "answer key" for the "Creating your First Pack" tutorial.

…ives

…/install

…son not to

guzzijones · 2025-04-07T23:24:40Z

The Enterprise Linux versions are lagging behind python. I vote:

We could include an st2.pex file for each python version we support, on every OS, so that people can easily rebuild the venv when they update to a newer python version.
In this scenario, we need to modify the _venv_py3(minor) helper function to define dst=f"/opt/stackstorm/install/st2-py3{minor}.pex" add a symlink for the default python version as /opt/stackstorm/install/st2.pex -> /opt/stackstorm/install/st2-py38.pex. Maybe we can record this symlink as a config|noreplace file so that people can modify their preferred python version.

There was a multi-minute difference between el8 and el9. The compression algorithm is the only significant difference, so I timed building the same el8 package with different algorithms. - xz and lzma were the slowest at over 3 minutes. - gzip and zstd were the fastest at under 1 minute (at any level). The size differences were marginal, but zstd:best was the closest match in size to xz, so I used that to keep the package size as small as possible while speeding up the package build process.

cognifloyd · 2025-04-08T19:46:47Z

I modified the package deps so that pexes for all supported python versions are included. I also modified the post-install.sh script to create a symlink at /opt/stackstorm/install/st2.pex if the symlink is missing or is broken.

I included a bugfix for pants-plugins/release (I forgot to set some dataclasses as frozen).
I also included a modification to the compression algorithm used for building rpms. xz might be the default for el8, but it's significantly slower at over 3 minutes to build the rpm vs <1 minute for zstd. So, now we just use zstd:best compression for rpms.

NOTE: when we start making releases with pants, we need to:

announce in the release notes that all deb/rpm package installs/upgrades will remove and rebuild the /opt/stackstorm/st2 virtualenv. So, any modifications to the venv will need to be re-applied after upgrades.
add instructions to the docs on rebuilding the st2 venv (update the st2.pex symlink and run it) to upgrade python.

cognifloyd added 6 commits April 7, 2025 09:55

Packaging: Add /opt/stackstorm/install to hold st2.pex and other arch…

7b161e4

…ives

Pants: Move venv packaging targets to separate BUILD file

43c1227

pants: Dry parametrization in packaging/BUILD.venv

6db2044

Packaging: Add self-install archives for each pack to /opt/stackstorm…

0ece94f

…/install

packaging: drop TODO - just include extra packs unless there is a rea…

6d3ef4c

…son not to

update changelog entry

850cc78

cognifloyd added feature st2-packages python3 packs pantsbuild labels Apr 7, 2025

cognifloyd added this to the pants milestone Apr 7, 2025

cognifloyd requested review from winem, nzlosh, guzzijones, amanda11 and a team April 7, 2025 16:05

cognifloyd self-assigned this Apr 7, 2025

pull-request-size bot added the size/M PR that changes 30-99 lines. Good size to review. label Apr 7, 2025

guzzijones approved these changes Apr 7, 2025

View reviewed changes

cognifloyd added 3 commits April 8, 2025 13:38

pants-plugins/release: rule input/output must be hashable/immutable

4446469

Packaging: include st2-py3*.pex in deb/rpm packages

2cf9ea1

pull-request-size bot added size/L PR that changes 100-499 lines. Requires some effort to review. and removed size/M PR that changes 30-99 lines. Good size to review. labels Apr 8, 2025

cognifloyd requested a review from a team April 8, 2025 19:51

cognifloyd enabled auto-merge April 8, 2025 19:51

cognifloyd requested a review from rush-skills April 8, 2025 20:14

nzlosh approved these changes Apr 8, 2025

View reviewed changes

cognifloyd merged commit 7c080a6 into master Apr 8, 2025
120 of 123 checks passed

cognifloyd deleted the packaging-nfpm_contents-archives branch April 8, 2025 20:49

cognifloyd mentioned this pull request Apr 11, 2025

Packaging: Add conf files to deb/rpm packages #6330

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Packaging: Add venv and packs to deb/rpm packages #6328

Packaging: Add venv and packs to deb/rpm packages #6328

Uh oh!

cognifloyd commented Apr 7, 2025 •

edited

Loading

Uh oh!

guzzijones commented Apr 7, 2025 •

edited

Loading

Uh oh!

cognifloyd commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

	nfpm_content_file(
	name="st2_venv",
	description="Pex file that system packages can use to generate /opt/stackstorm/st2",
	file_owner="root",
	file_group="root",
	file_mode="rwxr-x---",
	dst="/opt/stackstorm/install/st2.pex", # or st2-py3*.pex

	def _venv_py3(minor: str):
	return parametrize(
	f"py3{minor}",
	dependencies=[f":st2.pex@parametrize=py3{minor}"],
	src=f"st2-py3{minor}.pex", # relative to this BUILD file
	)

	**_venv_py3("8"),
	**_venv_py3("9"),
	**_venv_py3("10"),
	**_venv_py3("11"),

	def _st2_venv_deps() -> list[str]:
	"""Return a list of deps required to build the st2 venv.

	The :st2_venv parametrizations depend on parametrizations of :st2.pex,
	so, make sure to keep these 3 lists of parametrizations in sync.
	"""
	return [f":st2_venv@parametrize=py3{m}" for m in ("8", "9", "10", "11")]

	nfpm_content_file( # noqa: F821
	name="archive_for_nfpm",
	dependencies=[":archive"],
	src=f"packaging/packs/{pack_name}.tgz.run",
	dst=f"/opt/stackstorm/install/packs/{pack_name}.tgz.run",
	file_owner="root",
	file_group=ST2_PACKS_GROUP,
	file_mode="rwxr-x---",
	)

	# :archive_for_npfm targets created by st2_pack_archive() macro (see pants-plugins/macros.py)
	target(
	name="default_packs",
	dependencies=[f"//contrib/{pack}:archive_for_nfpm" for pack in _DEFAULT_PACKS],
	)
	target(
	name="extra_packs",
	dependencies=[f"//contrib/{pack}:archive_for_nfpm" for pack in _EXTRA_PACKS],
	)

	# These are packs that should be installed with the default ST2 install.
	_DEFAULT_PACKS = (
	"chatops",
	"core",
	"default",
	"linux",
	"packs",
	)

	# The default set of packs installed with st2.
	_ST2_PACKS="
	chatops
	core
	default
	linux
	packs
	"

	# These are packs that should NOT be installed with the default ST2 install.
	_EXTRA_PACKS = (
	"debug",
	"examples",
	"hello_st2",
	)

Uh oh!

Packaging: Add venv and packs to deb/rpm packages #6328

Packaging: Add venv and packs to deb/rpm packages #6328

Uh oh!

Conversation

cognifloyd commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Virtualenv (st2.pex)

Packs (pack archives)

Open Questions

Uh oh!

guzzijones commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cognifloyd commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

cognifloyd commented Apr 7, 2025 •

edited

Loading

Virtualenv (`st2.pex`)

guzzijones commented Apr 7, 2025 •

edited

Loading