In our previous posts, we’ve established three things.
- We need metrics and measurement to guide our activity
- If we have something we’d like to focus on, we can measure it.
- It’s very important to be careful with our metrics. Measuring the wrong stuff, or even less important stuff is a really bad idea.
As an organization producing software, what should we measure. What are our KPIs?
We’ve been in the software industry for several decades now, and we’ve heard a lot of proposed metrics. Some proposals:
Uptime is important: During required hours, how often is your system offline? Given 31 million seconds in a year, 99.99% uptime (Usually called “Four Nines”) means that your system averages being down for less than an hour a year. Five nines uptime is five minutes a year, and six nines is 30 seconds in a year downtime. Do you need to measure uptime?
Performance is important: How quick does your system respond? When a user makes a request, how much time passes between receiving the request and responding? Even more important is the question of how much time passes after the user pushing a button and the user getting a response. This is difficult to measure, the best approximation comes from using system response time as a proxy. That still leaves the question of response time. Should we measure response time?
Security is very important: Can we measure number of security incidents? Can we measure the severity of security incidents? Can we measure the number of users impacted, and get a measure of how secure our software system is? Should we do that?
Performance, uptime, and security are just the beginning. When you start researching KPIs and IT, you’re likely to stumble across links like this, which list a full hundred and fifty potential KPIs for IT systems.
One hundred and fifty potential metrics, and they are all worth measuring.
BUT…
We also know that measuring too many things is a bad idea. Vanity metrics are problematic. Remember, never measure something just because you can. Measure what’s most important.
And once you refactor the understanding from “What is important” to “What is most important”, the measurements become more valuable, and potentially, measurement is an easier problem.
There are three important metrics in the software development industry. Everything else falls in one of the following three circumstances. You are measuring something that is not worth measuring. You are looking at a vanity metric, or you are measuring something worth measuring for a while, but not for long.
The ultimate software metric is simple. Are you building software that is worth building?* Are you building the right product? We sometimes call this the webvan test, after the famous Dotcom bust, but it’s the primary problem with huge amounts of software. It is what happens when no one wants what you’re selling (at that price).
If you build software that isn’t what people want, then nothing else about the software matters. How good is the software? Don’t care. How secure? Don’t care. Either build software that people want to use, or you have wasted money on a worthless product.
Unfortunately, the ultimate metric is not about Software. The ultimate metric is about the business decisions regarding what software to build. It doesn’t help developers to figure out what to measure as you build your software. What makes for software built right, as opposed to just building the right software?
The Second metric is just one question. How quickly can you make a small change to your software, on average, without breaking other stuff? This is the single biggest issue in software development. We began our discussion with the stories of what happens when maintainability or ease of change drops, and it’s clear to us at this point that if you’re building the right software, nothing else matters even close to as much.
Do you have performance issues? Change your code** so you don’t. How easy is it to change your code?
Do you have response time problems? Change your code so you don’t. How easy is it to change your code?
Do you have bugs in your code? Change the code so you don’t. How easy is it to change?
User experience? Change your code. Reliability? Change your code.
If you can change your software quickly, you can solve the problems you encounter. If you can’t change your software quickly, you can’t.
The Second metric is software maintainability. How quickly can you make a change without other impacts? If you get this right, then everything else that you do in software works out well because you can control your software. If you get this wrong then there is little that you can do to keep up with the modern world and it’s incessant change.
The third metric is the happiness metric. Once you’re building a useful product, and you’re building the product with high maintainability, the other thing that matters is: Are your people happy? And by people we mean all the people involved. Are your employees happy? Are your customers happy? Are your other business contacts (contractors, suppliers, clients) happy? It’s an underappreciated metric…and a topic for another time. If you can’t wait, read Joy, Inc. by Rich Sheridan.
The Three Metrics:
- How useful is your software?
- How maintainable is your software?
- How happy are your people?
The remainder of this discussion will focus on the second metric: the part that software developers, managers, and executives have control over.
The Second Metric 