As with all four key DORA metrics, measurement is the first step to improvement. Change fail rate is critical to track because it demonstrates your team’s success at delivering reliable, stable code. Having strong scores when it comes to deployment frequency and lead time for changes is only impressive if you have a comparatively low change failure rate. Here’s how you can assess how stable your application is.
What is change fail rate (CFR)?
Change fail rate (CFR) is one of the four DORA key metrics. It represents how often a change (or deployment) to production doesn’t perform as expected, or introduces a bug/regression.
DORA measures software throughput and stability, which are arguably the most critical dimensions when it comes to evaluating your team’s performance. Change fail rate (as of 2024) is considered the primary stability metric, among the four keys. This means that CFR is equally important to track as the velocity metrics, and will demonstrate if your fast deployment rates are at the expense of often-faulty code.
Any time you push a patch or hotfix, the code change you’re resolving is counted as a “failed deployment”. Sometimes these don’t surface until much later, when you find a bug that had been laying low for months or years.This is all important to keep track of, because it reveals faulty testing or QA practices, which may be either process or pipeline-based.
How to measure change fail rate
Change fail rate is represented as a percentage: it is the percentage of faulty deployments from your number of total deployments.
Change fail rate = ((number of failed deployments) / (number of total deployments)) * 100
It’s a simple formula, but the challenging part comes from collecting enough deployment data to measure trends. If you haven’t already, it’s helpful to find a way to automate this data collection, preferably into a spreadsheet. From there, you can track change fail rate over the time intervals of your choosing, and compare those rates to your other DORA metrics.
Measuring CFR through story status
One straightforward way to get change fail rate is by crunching story data. Keep count of every ticket that results in a deployment, as well as every ticket that goes from Completed back to In progress. With most ticket systems, this can be a good way to approximate how often your completed tickets need a rework. However, keep in mind that it’s only as accurate as your ticket tracking — if you aren’t documenting thoroughly, you won’t get a reliable estimate on your change fail rate.
Measuring CFR through GitHub
If you’re using GitHub’s Deployment feature for production releases, you can use GitHub’s REST API to collect deployment events. At minimum, you’ll want to record timestamps for when each deployment was created and/or modified, and grab deployment statuses (success, failure, error).
GET /repos/my-username/my-project/deployments
Measuring CFR through GitLab
GitLab’s designated Deployment API can give you the data you need to calculate change fail rate. A simple call returns all deployments for a given project, and you can also grab the timestamp and its status. From there, you can compare the number of deployments with failed status to the overall number.
GET /projects/:id/deployments
For Ultimate subscribers, GitLab offers a DORA metrics API, where you can directly request DORA change fail rate.
GET /projects/:id/dora/metrics
What is a good change fail rate?
As of the 2024 DORA report, the highest-performing teams (those who have attained the “elite” performance standard) have a change fail rate of 5%. About one-fifth of teams surveyed fall within this range.
Interestingly, teams within DORA’s “high” performance standard tend to have a worse change fail rate than those within the “medium” standard. Teams with a lead time of under a week tend to see a change fail rate of 20%, while those with a lead time between one week and one month see a change fail rate of 10%. Often, teams struggle to maintain a low change fail rate when they’re pushing code fast (can be from lack of QA automation, code review, test coverage, etc.).
There doesn’t have to be a tradeoff between throughput and stability. With thorough pipelines and processes in place, your team should be able to deliver features fast, and trust the features they’re delivering.
TLDR: change fail rate should be taken with the context of throughput. If you have a high throughput and low change fail rate, you’re doing well. Remember, DORA metrics are best used to measure your improvement. As long as your change fail rate is going down (and not at the expense of your deployment velocity or lead time), this is a good indicator of performance. You don’t need to aim for “elite” numbers, as your team’s ideal change fail rate might look different from that of another org.
Trying to reduce change fail rate?
The number one enemy of a good change fail rate is poor testing practices. When teams are trying to deploy fast, it can be tricky to ensure each feature gets thorough testing and review. But how do you remedy that, while still keeping lead times short?
Ephemeral environments are an effective way to make sure your features get enough eyes (and automation) prior to staging. Since infrastructure wait times become nonexistent, you can test your features as soon as you’re done building them, and share them with all stakeholders to get sufficient feedback. Instead of going back to the drawing board, you can make all changes on the spot, and then run tests until your pipeline is green. Having multiple gates before production (and even staging) can inspire trust in every deployment, resulting in a low change fail rate.
And Shipyard gives you the tools and automation you need to do exactly that! Kick off a 30 day free trial and see for yourself. Your DORA metrics will thank you.
