The Problem With Measuring Things

Posted by

In sport and some (but not all) other human activities ‘performance’ is a requirement.  Simply put if you don’t ‘perform’ you’re out – of a job, of the team, of business.  But what is ‘performance’?  Sometimes it’s just about winning and losing.  In a single game, there is one winner and one loser.  Easy.  One coach is a hero, one coach is sacked and everyone goes home understanding how things worked..  Unfortunately (or perhaps fortunately) it is rarely as clear cut as that and so we must look in other areas in order to define ‘performance’.  If we want a definition we need concrete facts, indicators.  Nothing is more concrete than numbers and so we look for things to which we can attach a number, things we can measure.  Some of these numbers seem to very clearly imply ‘performance’, and so we take those ones we know as our starting point.  Now that we know things, it is easy.  If the numbers go up we are getting better/stronger/more popular.  If the numbers go down we are getting worse/weaker/less popular.  We set our goals (SMART goals), laminate them, hang them up, and go to work.  So far, so good.

But it is that moment when people make a fatal (not literally, for the most part) error.

The fatal error is the assumption that the link between ‘performance’ and those goals/indicators is a two way relationship.  That is, it is assumed that given improved ‘performance’ is reflected by improvement in specific measurable  goals, then it must also hold that improvements in those individual goal areas reflect an improvement in ‘performance’.  So in an effort to improve ‘performance’ and in accordance with the theory, people concentrate their efforts on improving in the goal areas.  In doing so, their attention and focus every so subtly change from ‘improving performance’ to ‘achieving goals’.  See footnote*.

Unfortunately, the relationship is not two way.

Examples abound….

If I were a professional sporting league (for example, the AFL), clear measurements of my strength and health would be things like attendance and television ratings.  Addressing those goals could result in the biggest clubs having the most advantages as those are the clubs that most directly (and immediately) impact attendance and ratings.  Let’s call that the ‘Rule of Ed’.  But the health of the league also has a long term component and while the league registers short term improvement in goal areas, individual members may suffer.  Let’s call that the ‘Rule of Wanna Be Ed’ or ‘Rule of James Brayshaw’.  For other examples, see the Spanish Football League, which has the two biggest and best clubs in the world, but has a player’s strike because other players in the league are not being paid.

If I were an NFL scout trying to find the best players for my team, I would have to choose between hundreds of players who are essentially the same.  I need some way to differentiate between them somehow, so I go the NFL Combine.  I know all the measurable characteristics of all the best players, so now I can compare, and with each measurement I would get further away from actual ‘performance’.**  Actually that’s not much different from putting together a professional volleyball team.  Except there is no combine.  Let’s move on.

If I were a volleyball program, I could see that the best volleyball teams in the world have certain physical characteristics.  By making my goal to replicate the physical characteristics of the best teams in the world, I would have an eminently SMART goal.  But concentrating my resources on the most easily measurable area I could under resource other equally important, but less measurable areas that are equally important to ‘performance’.

If I were a volleyball coach, I could see that victory was closely related to reaching certain performance goals, for example some attack percentage or error percentage.  I will never forget talking to a well known coach who was quite surprised to discover that having worked long and hard to reach specific performance goals, and succeeding, the results (ie ‘performance’) were no different than before.

If I were a company director… I shouldn’t go there as it’s not my area of expertise but I’ve read that those who created the financial crisis still got their bonuses.  I’ve seen that somewhere else too…

Measuring things leads to lazy and short sighted thinking.

Measuring things leads to devoting resources to ever smaller, more measurable, less important areas.

Measuring things leads to losing that essential overview.

Measuring things leads to forgetting what it was we were trying to do in the first place.

Measuring things leads to worse ‘performance’.

* Long after originally writing this post, I discovered that this process has been identified and named in the field of economics.  It is known as ‘Goodhart’s Law’ and states that “When a measure becomes a target, it ceases to be a good measure.”  That seems to fit pretty well here. It is also implicit in the economic theory of rational expectations, that states that those who are aware of a system of rewards and punishments will optimize their actions within that system to achieve their desired results. For example, if an employee is rewarded by the number of cars sold each month, they will try to sell more cars, even at a loss.

** This point was prompted by reading a passage in the Mike Leach book about scouting players.

Read about the great new Vyacheslav Platonov coaching book here.

Cover v2


    1. Absolutely not. You have to measure things. I’m a stats guy at heart. You just have to understand what it is you are measuring and what those things mean. For example, I use as a goal of an attacking percentage of 50%. If I just looked at those numbers I would say that 46% is better than 45% and therefore is progress. And I would have to say that 50% means that I played well. None of those things are true. The numbers give me a starting point for my actual analysis, they are not analysis in themselves.
      Measure, but understand what it really means.


    1. I think my point is different. My point is that taking easily measurable things and assuming they represent ‘performance’ or improvement or something is the wrong way. It becomes dangerous, and self defeating, when they are used as things like KPI’s that determine whether individuals keep their jobs or earn bonuses.
      In that case the analysis of one party (the coach or CEO in the examples above) is used the paralyse the logic of another (the federations or employers), for personal gain.


  1. Hello, my name is Danny O’Toole and I am a teacher at St Laurence’s College, an independent all boys school in Brisbane.

    I have contacted you hoping that you will be able to assist me.

    In September of this year, I am taking 2 volleyball teams on tour in Europe visiting Paris, Rome and Krakow. I am hoping to get in contact with schools or clubs in each of these cities in the hope of organising volleyball matches between our schools. Playing against foreign schools is a memorable experience for all those involved and I am sure, like our boys, the local students would relish this opportunity.

    We are hoping to play 2 games in each city, either international schools or local schools or clubs. I have found it difficult to access schools/clubs in each city. I was hoping you may have contacts that could help me.

    Before their summer school holidays I was able to arrange games with the American School of Paris, Marymount School in Rome and the American Overseas School of Rome. However, it would be great to try to organise 2 or 3 more. In particular Krakow has been difficult to make any contacts.

    As part of the tour it would be tremendous to make contact with either an elite professional or semi professional club, player or coach in these cities to maybe watch play or train.

    The purpose of our tour is to develop the boys’ interest in volleyball, prepare for the 2012 season in February/March, as well as appreciate a different culture to that which they are use to. Volleyball is vastly more popular in Europe than in Australia, so it is hoped that by visiting these cities and immersing the boys in volleyball they will appreciate just how big the game is in a world context.

    We are arriving in Paris on the 15th September, Krakow on the 20th September and departing for Rome on the 24th of September. We will have 2 teams, a junior team made up of mainly 14 and 15 year olds and a senior team made up of 16 and 17 year olds. Our teams will be competitive but not of too high a standard as we are out of season and this tour is primarily about developing the profile of the game in our school as well as identifying emerging players for next year’s team selection therefore the abilities of the boys involved ranges considerably.

    I appreciate that you are probably very busy at this point of time however, any information you can forward me will be greatly appreciated. Or if you could forward this email to any of your contacts that would help me out as well.

    Kind Regards
    Danny O’Toole
    First VI Volleyball Coach St Laurence’s College


  2. Hi Mark. Really enjoying the blog. I came across this line the other day that I thought fit the bill for this post:

    Not everything that counts can be counted and not everything that is counted, counts


    1. Not only does the line fit, it would have been a perfect last line for the post.
      Thanks for that, I’ll definitely be using that again.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s