FAQs
Q: My project is using two different tools at the same time. How do you consolidate the execution data into one and compute the metrics?
A: To be answered soon.
Q: Why do you synchronize execution history to local? Will this become a security risk?
A: To be answered soon.
Q: Do you include failure deployments when calculating DF?
A: No. DF is supposed to reflect the frequency we ship value to the target environments. A failure deployment does not ship any change to the target environments hence should not be included.
Q: Do you include "empty builds" (i.e. the builds which include no new commits) when calculating DF?
A: Guess a better approach is including the empty builds when calculating DF, but ignore its Lead Time.
Q: Is there a better way to calculate MLT instead of calculate the mean value of multiple deployments? Shouldn't we consider the weight of different deployments due to the various batch size?
A: Correct. Batch size need to be considered. Refer to the latest algorithm to see how it works in our system.
Q: How should we compute "delayed deployments"? Does "delayed deployments" result in negative impact to the MLT value?
A: When compute MLT value, commits are selected base on the time range in which the deployment occurs instead of the build trigger.
i.e. When deployment occurs, compute its lead time. Otherwise no.
There's one special case though - imagine we have two deployments, where the deployment#1 happens before deployment #2. However if we trace back to their build time respective, It is possible though the time when build#1 occurs is actually after build#2.
Q: How do you classify or deem a failure from the CI? A build failure? A deployment failure? Or .... how do you know the actual deployed release failed to deliver value?
Q: Is "an aborted build" a failure build? Should it be included in CFR?
A: No. Since for each failure we would also want to calculate its TTR value. An aborted build doesn't necessarily result in a failure to which team wants to apply a fix.
Q: When calculating CFR, except for the failure execution to target environment, should we also calculate the ones failed at preceding stages? For example, failed at build stage.
A: No.
Because it doesn't sound reasonable for counting an execution as failure when it did not even attempt to deploy to the target environment. On the other hand, technically it is also cumbersome to identify the order of different stages.
Q: For consecutive failures, do we measure the TTR time for all of them, or the first failure only?
A: For consecutive failures, we measure the TTR value for first failure only.