)
)
)
Imagine actually being excited about an upcoming review meeting. You have three key metrics you are responsible for, and your overall numbers were pretty decent this quarter. However, you are most excited to present some big ideas about ways to scale and innovate.
Your part of the review meeting starts and you toss up the first chart to show past performance. As you start to move to the second slide, someone speaks up, “Can you explain what happened between the second and third week of August? It looks like there was a big dip there?”
And there goes the meeting.
You spend the next 25 minutes coming up with 20 ideas about why there was a dip one week.
Bad metric habits
Metrics are powerful tools for every business leader but most of us never got training on how to present or analyze data. As such, we’ve picked up a lot of bad behaviors around metrics:
- Our charts tend to unnecessarily highlight natural fluctuations
- We spend way too much time and energy trying to explain these fluctuations
- People with no statistical training feel compelled to make a comment on data points that don’t need to be discussed
- We set arbitrary targets for metrics without changing how we will achieve them
As soon as I finished my conversation with Mark Graban about Work Minus Overreacting to Metrics, I knew that I needed to understand his area of expertise better. Mark’s book Measures of Success lays out some foundational principles that are essential to anyone who needs to report data.
It’s not about the numbers
The main point of Mark’s book is that metrics can only tell the story of your system. If you have a metric you are chasing, you have to set up a system to try to achieve it. There are natural upper and lower limits to the metrics any system can produce, and the system will almost always fluctuate around an average.
When you plot all of these data points on a Process Behavior Chart, it becomes clear that certain data points are only noise–random blips within the natural limits that aren’t worth talking about. However, other points are real signals–clues to tell you that something has either changed in the system or its environment.
When given a target metric, most of us are focused on that number. However, Mark shares this powerful thought:
We don’t manage the metric; we manage the system that leads to the results, and we lead the people who help us improve the system.
Numbers don’t happen on their own. When we miss or achieve a target, it is rarely the result of trying harder; mostly it is a reflection of the quality of the system we have in place. So, the metric shows us if the system is adequate or not. If it isn’t, then we need to work with the people who are a part of it to improve it.
Don’t chase numbers, chase improvements
Unfortunately, most leaders are taught to just look at the number. When the goal is the number and not the improvement of the system, then Goodhart’s law comes into effect and people tend to either distort the number or distort the system to artificially affect the number.
How many times have you walked out of a planning meeting where someone sets a ridiculously high target metric and then doesn’t plan to change the system at all to achieve it? Mark says this is due to the misconception that you can easily squeeze better results out of any system just by trying harder, or hiring a different manager. Instead, if you want a higher target than your system is naturally capable of, you need to make significant changes to the system.
Worry about the signal, not the noise
A good chunk of this book shows that any system will have natural fluctuations–two data points are not a trend. When leaders don’t acknowledge these fluctuations, they end up spending a lot of time and energy trying to explain missed targets. Instead, when you accept that the numbers will go up and down, you can spend time focused on the real signals–signs that there has been a change in the system or its environment. When these signals happen, you can test it by making a change and see how it affects the metric.
Mark Graban makes process behavior charts easily accessible and shows how to create and plot them without needing a statistics degree.
Taking it deeper
Here are a few more of my takeaways from the book:
- Weekly data points are much more helpful than monthly when it comes to analyzing signals and noise, and daily is better than weekly.
- Be wary of using linear trend lines on your charts, because they suggest that your metrics will forever only go up and you are setting yourself up to have to explain what happens when they reach their limit.
- It’s best to test out one idea at a time to improve a system so that you can know its effect, but you may not always have this luxury.
Lingering questions
Mark brings up the idea of success theater, which is basically how we can bend and distort numbers to say what we want. In fact, when reviewing data in a meeting, I find that most of what is happening is everyone is deciding what kind of narrative to tell about the data. If the data looks bad, can you spin a story that excuses it and assume it will get better on its own? Can you cherry pick a few data points to spin a positive story? It’s this interpretation and narrative casting that seems much more powerful than the data itself.
I find that I’m always tweaking my systems, so how can I trust my metrics? Mark says that you need 20 data points to really be able to plot out a trend. I’ve rarely gone 20 weeks without making some kind of change. Perhaps this is a case for either collecting more frequent data or allowing a system to exist on its own for a longer time to test changes.
As much time as Mark Graban has spent with charts and data, he acknowledges that it’s really hard to know the causes of changes; we can usually only guess. This is a hard pill to swallow sometimes. Even if we know our metrics are accurate, we often have no idea what they mean. That doesn’t mean we shouldn’t pay attention, but it needs to be acknowledged.
Measures of Success
If you are in any kind of leadership role, you’ve probably got metrics that you need to track. Mark Graban’s book, Measures of Success, is a great tool to help you focus not on those metrics but on the systems that produce the numbers, and to acknowledge that there are natural fluctuations. Check out the book, and Mark’s blog.
Imagine actually being excited about an upcoming review meeting. You have three key metrics you are responsible for, and your overall numbers were pretty decent this quarter. However, you are most excited to present some big ideas about ways to scale and innovate.
Your part of the review meeting starts and you toss up the first chart to show past performance. As you start to move to the second slide, someone speaks up, “Can you explain what happened between the second and third week of August? It looks like there was a big dip there?”
And there goes the meeting.
You spend the next 25 minutes coming up with 20 ideas about why there was a dip one week.
Bad metric habits
Metrics are powerful tools for every business leader but most of us never got training on how to present or analyze data. As such, we’ve picked up a lot of bad behaviors around metrics:
- Our charts tend to unnecessarily highlight natural fluctuations
- We spend way too much time and energy trying to explain these fluctuations
- People with no statistical training feel compelled to make a comment on data points that don’t need to be discussed
- We set arbitrary targets for metrics without changing how we will achieve them
As soon as I finished my conversation with Mark Graban about Work Minus Overreacting to Metrics, I knew that I needed to understand his area of expertise better. Mark’s book Measures of Success lays out some foundational principles that are essential to anyone who needs to report data.
It’s not about the numbers
The main point of Mark’s book is that metrics can only tell the story of your system. If you have a metric you are chasing, you have to set up a system to try to achieve it. There are natural upper and lower limits to the metrics any system can produce, and the system will almost always fluctuate around an average.
When you plot all of these data points on a Process Behavior Chart, it becomes clear that certain data points are only noise–random blips within the natural limits that aren’t worth talking about. However, other points are real signals–clues to tell you that something has either changed in the system or its environment.
When given a target metric, most of us are focused on that number. However, Mark shares this powerful thought:
We don’t manage the metric; we manage the system that leads to the results, and we lead the people who help us improve the system.
Numbers don’t happen on their own. When we miss or achieve a target, it is rarely the result of trying harder; mostly it is a reflection of the quality of the system we have in place. So, the metric shows us if the system is adequate or not. If it isn’t, then we need to work with the people who are a part of it to improve it.
Don’t chase numbers, chase improvements
Unfortunately, most leaders are taught to just look at the number. When the goal is the number and not the improvement of the system, then Goodhart’s law comes into effect and people tend to either distort the number or distort the system to artificially affect the number.
How many times have you walked out of a planning meeting where someone sets a ridiculously high target metric and then doesn’t plan to change the system at all to achieve it? Mark says this is due to the misconception that you can easily squeeze better results out of any system just by trying harder, or hiring a different manager. Instead, if you want a higher target than your system is naturally capable of, you need to make significant changes to the system.
Worry about the signal, not the noise
A good chunk of this book shows that any system will have natural fluctuations–two data points are not a trend. When leaders don’t acknowledge these fluctuations, they end up spending a lot of time and energy trying to explain missed targets. Instead, when you accept that the numbers will go up and down, you can spend time focused on the real signals–signs that there has been a change in the system or its environment. When these signals happen, you can test it by making a change and see how it affects the metric.
Mark Graban makes process behavior charts easily accessible and shows how to create and plot them without needing a statistics degree.
Taking it deeper
Here are a few more of my takeaways from the book:
- Weekly data points are much more helpful than monthly when it comes to analyzing signals and noise, and daily is better than weekly.
- Be wary of using linear trend lines on your charts, because they suggest that your metrics will forever only go up and you are setting yourself up to have to explain what happens when they reach their limit.
- It’s best to test out one idea at a time to improve a system so that you can know its effect, but you may not always have this luxury.
Lingering questions
Mark brings up the idea of success theater, which is basically how we can bend and distort numbers to say what we want. In fact, when reviewing data in a meeting, I find that most of what is happening is everyone is deciding what kind of narrative to tell about the data. If the data looks bad, can you spin a story that excuses it and assume it will get better on its own? Can you cherry pick a few data points to spin a positive story? It’s this interpretation and narrative casting that seems much more powerful than the data itself.
I find that I’m always tweaking my systems, so how can I trust my metrics? Mark says that you need 20 data points to really be able to plot out a trend. I’ve rarely gone 20 weeks without making some kind of change. Perhaps this is a case for either collecting more frequent data or allowing a system to exist on its own for a longer time to test changes.
As much time as Mark Graban has spent with charts and data, he acknowledges that it’s really hard to know the causes of changes; we can usually only guess. This is a hard pill to swallow sometimes. Even if we know our metrics are accurate, we often have no idea what they mean. That doesn’t mean we shouldn’t pay attention, but it needs to be acknowledged.
Measures of Success
If you are in any kind of leadership role, you’ve probably got metrics that you need to track. Mark Graban’s book, Measures of Success, is a great tool to help you focus not on those metrics but on the systems that produce the numbers, and to acknowledge that there are natural fluctuations. Check out the book, and Mark’s blog.
