Zenhub Enterprise 4.1.0

These release notes are for Zenhub Enterprise for Virtual Machine and Zenhub Enterprise for Kubernetes.

• For all users using Zenhub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the "Important Upgrade Instructions for Administrators" section.

GitHub Enterprise Server compatibility

Zenhub Enterprise 4.1.0 supports GitHub Enterprise versions: 3.9, 3.10, and 3.11

What's new in Zenhub Enterprise 4.1.0

We are thrilled to announce the release of Zenhub Enterprise 4.1.0, packed with a host of exciting new features designed to optimize your workflow, boost productivity, and heighten collaboration.

Features

Daily Feed

Daily Feed is here! The new tab on the left hand side provides an overview of what you previously worked on, currently are working on and potential blockers. You can also see your team's Daily Feed too!

Within the ‘currently working on’ section you will see these helpful tags we’ve added, where if you are needed for a code review it will appear as well as if you are working on an issue that is blocking another and if you are needed for your input for an estimate. For the blocked section, we are providing issues that you may be blocked by, your issues that need code revision etc.

Sprint insights

Now along with your Sprint burndown chart, at a glance, you can

• Predict the likelihood of your team meeting your Sprint goals
• Identify bottlenecks and areas for improvement that could potentially get in the way of your Sprint goals
• Make data-driven decisions to adjust your sprint goals.
• Provide transparency for your stakeholders to easily access your team's Sprint progress

Dynamic Productivity Recommendations

This feature empowers developers and teams with intelligent insights to enhance their codebase operations and improve productivity. Say goodbye to the guesswork on improving your Issues completion and PR practices and hello to automated, data-driven suggestions that elevate your development workflow.

Notion Integration

Users can now paste a share link for a Notion page and have a document preview displayed in Zenhub. We often see users sharing Notion links within Zenhub Issues, Epics or Comments and wanted to introduce a more informative experience. With the new Notion Previews, users will be able to see the document title, author, created and edited date all within Zenhub. Check the docs to learn how to configure this integration.

Open Pull Request links in GitHub

We've introduced a setting that allows a user to enable PRs to open directly in GitHub from Zenhub. We've had customer feedback from both internal and external teams around this feature and appreciated the flexibility it brings

Phone Home metrics for ZHE VM

We have automated the collection of the admin usage report as well as some other metrics on ZHE VM. These metrics are not sensitive, are stored securely and will be used to enhance our support capabilities. As this feature requires a connection to AWS, please reach out to enterprise@zenhub.com to enable this, or if you have any questions or concerns about this feature.

Security Fixes

• Package security updates fixing critical and high CVE

Bug Fixes

• Resolved a problem that caused the Chrome Web Store to reject publishing the browser extension due to additional requirements for Manifest V3.
• Fixed a bug causing existing workspaces to sometimes not be found when using the browser extension in GitHub
• Fixed a bug that causes the admin usage report not to return any results

Changes

• With this release, our old Board is no longer available. We've worked hard over the past few months to make Board 2.0 better in every way (see recent updates), and with ZHE 4.1 it is now the de facto standard.
• PostgreSQL as been upgraded from 11.16 to 15.4
• K3s has been upgraded from v1.24.13+k3s1 to v1.26.8+k3s1
• Kubernetes has been upgraded from v1.24.13 to v1.26.8
• The built-in upgrade bundle rollback feature has been deprecated. VM or disk snapshots should be used to revert to a previous version in case of an upgrade failure.
• Zenhub Enterprise for VM upgrade instructions. have been updated for clarity and to add the use of the tmux terminal multiplexer to protect the upgrade from possible SSH disconnects.

Known Issues

• Authentication fails when self-signed TLS certificates are used
• Logrotate image can be cleaned up by K8s garbage collector, causing failed log rotation jobs that consume system resources

VM Embedded Component Versions

Component	Version
Ubuntu	22.04.2
K3s	v1.26.8+k3s1
Kubernetes	v1.26.8
Kustomize	v4.5.3
Fluentd	v1.16.2-debian-1.0
Postgresql	15.4
MongoDB	4.4.13
RabbitMQ	3.8.31
Redis	7.0.5
PgBouncer	1.20.1

Important Upgrade Instructions for Administrators

Zenhub Enterprise for Virtual Machine

• You must be running 4.0.X to perform the upgrade to ZHE 4.1.0

• For users currently running Zenhub Enterprise, contact our team for a download link for the 4.1.0 upgrade package. The upgrade process is detailed in our documentation.

Zenhub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The minimum version of Kubernetes required to run ZHE 4.1 is now v1.26.

1. Prepare

• Get the kustomization.yaml you configured to setup Zenhub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• In ZHE 4.0 and onwards, secret_key_base is required to be a 42 byte string, any existing 38 byte string should be updated as described in kustomization.yaml
• If you are using your own registry and not Zenhub's registry, upload the new images tagged with this release to your registry.

3. Run configmap-generator.sh

Once you have setup all the configurations in your copy of kustomization.yaml and are ready to deploy to your cluster, you must run the configmap-generator.sh script that will fill placeholders in our Kubernetes manifests with your configuration values.

To run the script, navigate to the root of the directory containing your kustomization.yaml file, which also contains the configmap-generator.sh script. Then, run the script with these two commands:

chmod 700 configmap-generator.sh
./configmap-generator.sh

4. Application updates

In your new kustomization.yaml, confirm that the cluster image tags are set to the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of Zenhub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-4.1.0
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-4.1.0
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-4.1.0
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-4.1.0
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-4.1.0
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-4.1.0
  - name: devsite
    newName: us.gcr.io/zenhub-public/devsite
    newTag: zhe-4.1.0
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• First, delete any raptor-db-migrate and sanitycheck jobs so they may be recreated without errors:

Make sure the status of the jobs are Complete and not Running

kubectl -n <your_dedicated_namespace> delete job/raptor-db-migrate job/sanitycheck

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

5. Database Changes for Zenhub Enterprise for Kubernetes

⚠️ NOTE: For upgrading to Zenhub Enterprise version 4.1.0, Zenhub will require a manual change to the Postgresql schema.
We require a constraint called pih_no_overlap from the pipeline_issue_histories table to be dropped and replaced.

To do this, you may follow the following steps:

1. Ensure you have a backup of the existing PostgreSQL database. Implementation may vary depending on the PostgreSQL service provider
2. Perform the following while connected to the PostgreSQL database:

ALTER table pipeline_issue_histories DROP CONSTRAINT pih_no_overlap;
ALTER TABLE ONLY public.pipeline_issue_histories
    ADD CONSTRAINT pih_no_overlap EXCLUDE USING gist (workspace_id WITH =, issue_id WITH =, tsrange(started_at, ended_at, '[)'::text) WITH &&) DEFERRABLE;

4. Verify the completion of the above with the following PostgreSQL command which should return an empty result:

SELECT * FROM pg_stat_activity WHERE query LIKE '%public.pipeline_issue_histories%';

Next, proceed to run the regular data migration script.

⚠️ NOTE: Running this script is only required if you are upgrading from ZHE 4.0.X
⚠️ NOTE: If you are using stunnel, be sure to uncomment the relevant sections in update/batch_v1_job_data_migration.yaml

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

6. Re Deploy Application

⚠️ NOTE: Running these commands is only required if you are upgrading from ZHE 4.0.X

The data migration script run in the previous step will have scaled down the application in order to safely migrate data without active database writes occurring. Now that the migration is complete, the application needs to be deployed to the normal desired operational state:

• First perform a diff to check what will change (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

7. Finalize

• Securely store the updated kustomization.yaml