How to measure developer productivity is something that I've thought a lot about for a majority of my career in software engineering and product management. I strongly believe that engineering leaders can build strong team cultures and ways of working that create high-performing and empowered teams — and that data can be a multiplier in how they do.
There’s never been a better time for it.
Software Engineering's Big Shift
Technology has exploded and evolved in a very short time, with most of us seeing major shifts just over the course of our careers. Even just twenty years ago, computing required expensive servers, and hardware was the limiting factor. With the emergence of cloud computing we have unbounded computing power for relatively cheap.
Today, we are in a time where every company has to be a software company, whether they want to be or not. That means that many large organizations are learning to operate in new ways far outside of their core competencies. They have to learn how to build software and shift to running a technology company. On top of that, massive market shifts in the last two years have changed a lot in the industry. In 2021, we were in a zero-interest rate policy (ZIRP) era, with a bull market where VC money was flush and tech companies couldn’t hire enough engineers. In the midst of tech’s “Great Resignation,” our biggest challenge was keeping our people and keeping them happy.
But ZIRP is gone, and for the past few years we’ve seen mass layoffs across our industry. Thousands of developers are losing their jobs as companies try to do more with less. More than ever, high performing teams are in demand. The limiting factor (at least for the foreseeable future until AI changes everything again) is people.
And in the end, the successful companies will be the ones that can figure out how to focus on the most important business priorities, retain their people, and make the most of the time and budgets that they have.
The problem? Everyone’s trying to figure out how to do it, but no one has agreed on how to measure if it works.
Developer Productivity Metrics: What Are We Measuring?
There are several frameworks being used to measure engineering productivity today, plus a lot of metrics that don’t fall into either framework.
DORA Metrics
The DORA metrics are seen as a fairly accepted standard for measuring software development performance. They consist of five metrics that provide insight into velocity, stability, and reliability of your systems:
- deployment frequency
- lead time for changes
- change failure rate
- mean time to restore
- the ability to meet or exceed your reliability targets
Ultimately, DORA metrics emerged from the rise of DevOps, and are a picture of your operational health within the organization. They’re incredibly helpful if you don't measure anything today, but they do have diminishing returns. Once you achieve these good levels of performance, you don't need to keep making them better — it won’t always help your engineering organization operate more effectively.
Anecdotally, I’ve spoken with developers who are at “high-performing” companies, according to their DORA metrics, and yet, the developers say they don’t feel like they’re working on an elite team. The reason is that while the four DORA metrics are helpful, they’re only one aspect of understanding your engineering organization. Even DORA itself recognizes this complexity — while the industry has only really adopted the five DORA metrics, there are dozens more core competencies included in the framework that are often forgotten. The real story is much more complex.
SPACE Framework
The SPACE framework, developed by some of the same researchers behind the DORA metrics, is meant to present a more encompassing view of your organization. It includes five different aspects of good engineering teams:
- Satisfaction and well-being
- Performance
- Activity
- Communication and collaboration
- Efficiency and flow
While SPACE offers some suggestions of the types of metrics that you may want to manage, it doesn’t specify what the right metrics are. Where DORA is very specific about what “good” looks like, SPACE leaves a lot open to interpretation. Engineering leaders wanting clear direction and criteria for measurement find SPACE a little “hand-wavy” for this reason.
... and More
There are many other metrics out there that aren’t included in either framework, often accompanied by a wide range of opinions on why they’re critical.
When you put all of these different metrics together, it ends up being a lot. Too many metrics can be almost as unhelpful as no metrics at all. You get what you measure, and there can be unintended consequences if you're not really thoughtful about what you need to know and what you don’t.
Part of the reason that productivity can be hard to quantify within engineering organizations is because you ultimately need to be able to measure value created for the business. Moving faster only matters when there is alignment that you are working on the right things in the right ways.
A Holistic View of Engineering Performance
Telemetry for systems, like cloud monitoring or event logging, isn’t the same as telemetry for engineering teams. Every team is unique — what “good” looks like will vary not only from team to team but from developer to developer. The metrics of a healthy team can look completely different within different organizations, with different missions, with different humans.
What is consistent are the categories of metrics that you need to consider and the value of creating transparency and understanding about what's happening across your organization. What's missing here from most productivity conversations (and most productivity data) is context of what's actually happening within your organization.
At Uplevel, we think of an engineering organization as a triangle composed of three elements:
1. Engineering Culture
If burnout is negatively impacting your team and individuals don't have sufficient time for their work, leaders need to identify the root cause. Is it due to a surge in planning activities or strategic initiatives? Or is it an ongoing issue of inadequate work hours?
We've observed a significant correlation between being constantly available or “Always On” and the ability to engage in deep work across various organizations. This suggests that when devs don’t have enough time for deep work during regular hours, they often compensate by working after hours, which is a recipe for burnout.
Similarly, examining context switching can provide insights into whether teams are being overloaded with tasks. Excessive switching between WIPs, combined with frequent interruptions, can also serve as indicators of burnout. By recognizing these signs, you can proactively address issues and ensure your team operates at its best, as people who are burned out are unlikely to deliver high-quality, efficient work.
2. Delivery and Quality
Many of the metrics that speak to delivery and quality come from DORA standards. Leaders have to know that systems are stable and reliable and your tools don't hinder velocity. Here’s an example where metrics might provide an opportunity to learn about the differences in cycle teams across various teams.
For leaders, seeing these metrics would then provoke some deeper questions around what’s actually happening. What stage of development is the work in? Is the long cycle time because of a process issue? Maybe this team is located in a different time zone, and delay in getting code reviewed due to the time lag is really just slowing down their ability to ship. Or is there something else going on here that they need help with?
Cycle time in itself won’t tell the story, but it can start to open up the right lines of inquiry to get to the root cause of the problem.
3. Commitments
Commitment metrics indicate how well the team is actually working together, and whether they’re effective and aligned as an organization. Is their time actually being allocated effectively, or are they not able to be productive and effective because they’re spread too thin?
Some of these metrics focus on meeting health. How much time is spent in sprint meetings and 1:1s? Are those meetings all in one block, or are they distributed throughout the day, which leaves less time for deep work? (We often find that companies aren’t spending too much time in meetings, but they do need to implement some calendar optimization to enable enough solid working blocks.)
Another metric here is the type of work that engineers are spending their time on. Are they the right things? Is the amount of KTLO work increasing? We find that engineering leaders often can’t tell if their teams are spending time on the right things, and it’s not uncommon for them to discover that way more time is being spent on maintenance and support than delivering key initiatives that the business thinks are the top priority. Having a clear picture of what people are working on across the organization is key to understanding that.
Actioning Change
Gathering these metrics is the first step, but it’s most important to understand the outcome that you’re ultimately trying to achieve. Optimizing for all these metrics at once is a recipe for disaster — you can’t have it all, or at least not all at once.
The next step, then, is to focus on your biggest area of opportunity. Fix it, measure the change, and then move on to the next biggest opportunity.
One way we advise engineering leaders to action the change needed is to take a supportive lens to developer productivity. Stack ranking engineers actually creates the opposite type of generative and productive culture that leaders are trying to effect. You’ll see more finger pointing, more burnout, and less deployments to production as people get nervous about shipping their code and start holding back. You might see more errors, and incidents might take longer to resolve, because people are holding back and protecting their knowledge silos. It’s not pretty.
Instead, the job of a great leader is to look for the ways that you can help your developers and managers. With more data comes great responsibility, and so it's important to use that data wisely to empower rather than micromanage. Present the data, make suggestions, and then give them cover so that they feel empowered to solve their own problems. Creating transparency and visibility feeds a shared understanding of the biggest challenges that your organization faces, which will inspire your teams to buy into the change needed for a productive and effective engineering culture.