How I Used Metrics to Change Behavior in Organization

Did you ever want to make a change in your organization but didn’t have authority to do it? Or didn’t want to use your power because you want people to make own decisions? Then this text is for you.

This is a story of how I discovered a way to improve things by using metrics right.

Usually, metrics are brought up as a way to control (you cannot control what you cannot measure) or to understand the situation.

I will tell you about a different quality of metrics that works not only for managers: communicating importance in a very clear way.

If people consider a metric important — they will figure out a way to improve it, given they care about their company.

And the recipe to do it is: measure right things and present them in a correct way.

Sounds simple, but I didn’t understand all the components of it from the start. Below is my story of learning how to make it work.

A Failure Story

For a long time, I was passionate about improving qualify and effectiveness of software engineering. I also like to measure stuff. So I had figured: I will start measuring how things are going in engineering at my company, people will look at the metrics, see what doesn’t work well and improve it.

The first part was a success: after few weeks we had a lovely dashboard with a couple of dozens of metrics displayed.

That’s when I discovered a problem with the second part of the plan: nobody cared.

I bit of explanation required: I work in a flat organization. And a relative large too: more than 100 people. Which means people have more information and requests coming at them from different directions than they could handle.

And metrics my team collected was just another piece of information. None of the things they showed were urgent. And I didn’t have authority to make them important enough to motivate an action.

The experiment ended up being a failure.

A Success Story

A couple of months later I accidentally discovered the missing piece: a presentation.

As a flat organization, we rely on a few tools to deliver relevant information to everybody. One of them is a weekly all-hands company meeting where people make short demos and presentations.

I had just developed a tool which integrated Github pull requests (for non-engineers: it’s how programmers review each other’s work before it’s pushed to production) with Slack (company-wide chat) and allowed developers to ask for code review and respond to other people’s review requests.

As a side-effect, this tool allowed measuring how fast people responded to a request.

I made a presentation to the team about the tool. It included a measurement of how fast people responded to code review requests. I also promised to present how this metric changes in the future.

Feedback on the presentation was surprising. A couple of people told me that they don’t like how this tool will affect developers’ behavior related to code reviews. They thought that since I’m showing this metric, developers are supposed to do code reviews using this tool and if they don’t — metric for them will look bad, and they will look bad.

This was an Aha! moment.

I was collecting more important metrics before and nobody cared. What made a difference is an idea that others will look at those numbers and judge you by them, even though numbers were not personalized. Just an idea that those numbers could potentially affect how your work is perceived was enough.

As long as numbers were on a dashboard in your personal browser — they were not important. When you are convinced that many people are looking at those numbers — they become important.

This is not a revolutionary idea, but it took me time to put all the pieces together: if you want to change a behavior, create a metric that reflects that behavior and convince people that metric is being regularly seen by others.

Results of this were huge. Median time to respond to a code review requests reduced from 17.5 to 0.8 hours in just one week and then to less than 10 minutes next week.

This is an example of quantity becoming a quality. Suddenly developers had a way to request a code review and get it within one hour and not in a couple of days. If you ever had a development process with code reviews, you know how much of a difference does it make.

As soon as I realized that it works, I started using this approach for more things: improving the quality of our code and infrastructure, feedback cycles, planning and project management.

Of course, you cannot just pick any number and convince people it’s important. It needs to be already meaningful to them, and it needs to have a clear link to the quality of their work. If nobody can judge your work using the metric, you won’t be improving it.

This became one of my favorite tools when I want to improve things.

Do it Yourself

Now, since you read this article, you probably want to change something in your organization. Here is how you do it.

1. Select an activity or behavior you care about.
2. Find a metric that reflects the quality of it.
   Choose meaningful metric. Ask people around if you are not sure. If you cannot find a meaningful metric to reflect the improvement, then you should re-think if it’s the right one.
3. Collect data.
   Sometimes you cannot find a hard data to measure. It’s fine. You can always send surveys to people and ask for their subjective opinion. Aggregated subjective opinion is almost as good as hard data.
4. Make a presentation. If you don’t have a venue to present live — use email. Or message board. Or print it on a piece of paper and pin it on a wall in your office.
5. Repeat. It’s important for people to know that changes in their behavior will be visible going forward. If it was a one-time thing, there is no motivation to change.
6. Keep it interesting. Find correlations and trends, drill down into data to show underlying causes, identify and explain outliers, etc.

Here is the toolset I used:

• Python for collecting data. Most applications have APIs, and it’s very fast to write scrapers in Python. Use a language you are the most familiar with (unless it’s Java).
• Data storage is CSV files. It’s straightforward, and many tools support import/export.
• Tableau to look at the data and find interesting metrics or patterns worth presenting. It’s a fantastic piece of software, and they have a free Tableau Public.
• Keynote for presentation. I’m not much of a designer, so I appreciated the fact that it’s hard to create an ugly presentation with this tool.

Some wisdom I’ve learned in the process you may want to use:

• Try to collect as much raw data as you could because you never know exactly what metric will be interesting/useful. Rarely I ended up using a metric I assumed before playing with data.
• Don’t get too personal with metrics. People may get offended. Don’t show any names in a context that may be considered negative. Avoid showing groups in a negative context, unless you are sure you want to put a spotlight on them.
• You can go only so far with keeping people’s interest on the same metric. Switch them or make sure that numbers change significantly and you can tell a story behind that.

If this article helped you, or if you did something like this before and succeeded (or failed) — please, leave a comment or shoot me a message to bratchenko@gmail.com.