DORA metrics have taken software engineering by storm, being broadly adopted across the industry as the way to measure team performance. But they don’t tell the whole story of how your teams are operating. You need additional insights to add context to your engineering efforts.
Using a DORA-only approach, your “High” performing teams would be those that deploy frequently, quickly, responsively, and accurately — all good indicators of success. But what if they’re working on the wrong priorities or burning themselves out to get things into production, putting future work at risk? Would you still consider those high-performing teams?
The problem is, DORA treats burnout, capacity, and certain other key insights as outcomes of performance when they’re actually part of the same equation. This article will:
- Break down DORA metrics and what they actually measure.
- Propose a more comprehensive way to measure engineering performance.
DORA metrics and what they actually measure
Many software engineering leaders are looking for the best new way to measure performance, but it’s hard to keep up with the rapidly evolving equation. In fact, just a few years after publishing their book, Accelerate, the DORA team introduced a fifth metric to their list: reliability. That shows you how quickly even the DORA team’s findings are evolving.
The point is, this is all still new. We’ve only been trying to measure engineering performance for the last 10 years or so, and as an industry, we haven’t really figured it out yet. So while DORA metrics are a good start, we’re all still learning and iterating on how to measure performance.
As such, there’s been a recent explosion of new tools to measure DORA metrics, as well as competing methodologies for measuring engineering performance overall — most notably the SPACE framework introduced in 2021. And yet, the problem remains unsolved. But before we jump into what’s missing, it’s important to understand what’s already there.
Throughput + Stability + Reliability = Efficiency
As they stand now, DORA metrics are a measure of efficiency (throughput, stability, and reliability) — not overall performance. Efficiency often means maximizing speed and output. It answers the question: How quickly and often can your teams deploy new code that works and does what is promised?
✗ Throughput + Stability + Reliability = Performance
✓ Throughput + Stability + Reliability = Efficiency
As seen above, these efficiency metrics represent three categories outlined by the DORA team:
- Throughput is a combination of the first two DORA metrics: deployment frequency and lead time for changes. This is how often your teams successfully release to production and how long that process takes — what we at Uplevel call release frequency and cycle time. Calculated along with capacity (see below), throughput metrics can give you a better idea of delivery performance vs. arbitrary velocity measurements based on story points or “planning poker.”
- Stability covers the next two metrics: change failure rate and time to restore service, which measure deployments causing failures in production and how long it takes to recover. Along with the throughput metrics above, these stability metrics make up the delivery side of the performance equation.
- Reliability is the fifth DORA metric, helping measure your operational performance. This is your teams’ ability to keep promises about the product they’re building.
Together, these metrics give you a base-level view of your delivery and operational performance. But they’re still missing some key insights, which we’ll break down in the next section.
A more comprehensive measure of engineering performance
The categories in the above sections leave out three important variables that can have a major impact on performance now and as you scale: alignment, capacity, and burnout.
These metrics are concerned with team effectiveness rather than efficiency. Effectiveness is all about working on the right things in the right ways. That means aligning teams around the right priorities, giving them time to work on those priorities, and doing so in a way that maintains team health. The DORA team doesn’t ignore the importance of these variables in their research, but they don’t attempt to quantify them as part of their key metrics.
✓ Alignment + Capacity + Burnout = Effectiveness
Alignment
Alignment is making sure your teams are focused on the most high-impact projects. A team working on quick, low-priority deployments all day may appear more effective than a team working slowly on a complicated feature build. That’s why it’s important to look at how closely engineering efforts align with business priorities.
Prioritization insights help you measure performance against greater organizational goals. They also put some of the responsibility on leadership’s shoulders. With insights into how your teams are spending their time — separated into individual investment buckets — you can help allocate people, effort, and investments to activities that would make a greater impact on the organization.
Capacity
Capacity is how much time your team has to get work done. If you’re measuring deployment frequency in weeks or months, how much of that time did your teams spend on delivery? If their days were packed with meetings and other interruptions, they may not have had the capacity to deploy tons of new code. In that case, low deployment frequency may not accurately depict team performance. In fact, they may have performed at a relatively high level given their lower capacity.
With DORA metrics, you might see that your teams are deploying quickly and frequently and infer they have more capacity, but you would have no way to quantify it. Instead, we measure how much time your teams have for Deep Work, which we define as uninterrupted blocks of two or more hours. Deep Work insights are what make up the capacity part of our performance equation. They account for not only planned interruptions like meetings but also chat and other distractions that break your teams’ concentration throughout the day.
The more capacity you open up for consistent Deep Work, the more effective your teams can be. Just be careful not to confuse capacity with “availability,” which is measured as part of DORA’s reliability metrics.
Together, alignment and capacity insights are a measure of organizational focus: what your teams should be working on and how much time they have to work on it. Viewing your engineering efforts through this lens can give you a more accurate idea of effectiveness and overall performance, as well as the role leadership plays in it.
Burnout
Burnout isn’t an outcome of low-performing teams. But it could be the cause of that low performance. The longer burnout goes unidentified and unchecked, the greater the impact on your teams and their work. It doesn’t matter if you have the best month ever in terms of delivery performance. If your teams are struggling to keep up and burning out in the process, they aren’t really performing at a high level. It’s actually pretty unsafe and unsustainable.
Any true measure of engineering performance should account for the negative impacts of burnout. To quantify burnout, we measure “Always On” behaviors like Slack/Teams messaging, Jira, and calendar activity. This is perhaps the most important context you can give to your efficiency metrics (i.e., DORA), as it actually quantifies your teams’ burnout risk to show the human impact of their efforts.
Conclusion
Let’s be clear. DORA metrics are helpful for many engineering teams to measure efficiency. They’re the result of the largest ever study of its kind, and the research is sound. While these metrics are an important part of the performance equation, you need capacity, prioritization, and burnout (effectiveness) insights to provide context — which is needed to identify areas for improvement.
When put together, your efficiency and effectiveness metrics can give you a better overall way of measuring engineering performance.
✓ Efficiency + Effectiveness = Performance
See how Uplevel is rethinking how engineering teams use data to measure performance, helping turn DORA metrics into actionable insights.