| Metric | How to Measure |
|---|---|
| Ratio of story-to-bug tickets | Total of feature tickets over total of bug tickets. Include current quality release and previous feature release. |
| Mean time from P1 bug report to delivery |
With a new or existing Jira filter, with:
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”. |
| Mean time from P2 bug report to delivery |
With a new or existing Jira filter, with:
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”. |
| Mean time from P3 bug report to delivery |
With a new or existing Jira filter, with:
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”. |
| Number of P1 bugs reported in production | Count of major issues reported in product, documentation or marketing by customers, CSMs, Support, or Product (e.g. dot releases from customer reports, undocumented breaking changes). |
| Number of P2 bugs reported in production | Count of medium level issues reported in product, documentation or marketing by customers, CSMs, Support, or Product (e.g. dot releases from customer reports, undocumented breaking changes). |
| Number of P3 bugs reported in production | Count of small issues reported in product, documentation or marketing by customers, CSMs, Support, or Product (e.g. dot releases from customer reports, undocumented breaking changes). |
| Metric | How to Measure |
|---|---|
| Number of misses in each release cycle | Count of issues encountered during the release cycle (e.g. no process for an early advance notification for upcoming deprecated ESR). |
| Number of setbacks to users/staff resulting in a negative reaction | Manual count of negative reactions reported by CSMs, Support, or Product (e.g. features that got pushed). |
| Number of measurable actions from each Release Retrospective driven from root causes and completed and reported to R&D | Count of completed and reported items from Release Retrospective document. |
| Number of deadlines missed in Release Checklist | Manual count of deadlines not met from items posted to Release Checklist. |
| Metric | How to Measure |
|---|---|
| Time updated to RC1 (PST) | Check Release Discussion channel history for date/time RC1 was cut. |
| How many RCs cut | Check Release Discussion channel history for how many RCs were cut for that release. |
| Number of days between when final RC is cut and the release date | Check Release Discussion channel for post with official release build. Oxygen = 16th - Day Final RC is cut |
| Community + customer bugs reported during release timeframe (17th to 16th) |
With a new or existing Jira filter, with:
|
| Number of customer bugs fixed during release |
With a new or existing Jira filter, with:
|
| Total valid bugs in fix version |
After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND fixVersion = latestReleasedVersion() |
| Total valid bugs in fix version found by test automation | Check "Se", "Selenium-found, "Rainforest-found" Jira labels. |
| Total valid bugs found after RC1 is cut |
After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't fix") AND created > "START" AND created < "END" |
| Valid bugs found after RC1 fixed in release |
After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = latestReleasedVersion() |
| Valid bugs found after RC1 pushed to next release |
After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND created > "START" AND created < "END" AND fixVersion = earliestUnreleasedVersion() |
| Valid bugs found after RC1 fix version = other (eg unscheduled, not set) |
After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND created > "START" AND created < "END" AND resolution not in (Duplicate, "Cannot Reproduce", "Won't Fix") AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY) |
| (Non-security) Bugs requiring patch release | After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases. |
| Total features/improvements in fix version |
With a new or existing Jira filter, with:
|
| Critical security issues found during release timeframe |
With a new or existing Jira filter, check for Security Vulnerability tickets:
|
| Moderate security issues found during release timeframe |
With a new or existing Jira filter, check for Security Vulnerability tickets:
|
| Minor security issues found during release timeframe |
With a new or existing Jira filter, check for Security Vulnerability tickets:
|