The Problem With Measuring Things

In sport and some (but not all) other human activities ‘performance’ is a requirement.  Simply put if you don’t ‘perform’ you’re out – of a job, of the team, of business.  But what is ‘performance’?  Sometimes it’s just about winning and losing.  In a single game, there is one winner and one loser.  Easy.  One coach is a hero, one coach is sacked and everyone goes home understanding how things worked..  Unfortunately (or perhaps fortunately) it is rarely as clear cut as that and so we must look in other areas in order to define ‘performance’.  If we want a definition we need concrete facts, indicators.  Nothing is more concrete than numbers and so we look for things to which we can attach a number, things we can measure.  Some of these numbers seem to very clearly imply ‘performance’, and so we take those ones we know as our starting point.  Now that we know things, it is easy.  If the numbers go up we are getting better/stronger/more popular.  If the numbers go down we are getting worse/weaker/less popular.  We set our goals (SMART goals), laminate them, hang them up, and go to work.  So far, so good.

But it is that moment when people make a fatal (not literally, for the most part) error.

The fatal error is the assumption that the link between ‘performance’ and those goals/indicators is a two way relationship.  That is, it is assumed that given improved ‘performance’ is reflected by improvement in specific measurable  goals, then it must also hold that improvements in those individual goal areas reflect an improvement in ‘performance’.  So in an effort to improve ‘performance’ and in accordance with the theory, people concentrate their efforts on improving in the goal areas.  In doing so, their attention and focus every so subtly change from ‘improving performance’ to ‘achieving goals’.  See footnote*.

Unfortunately, the relationship is not two way.

Examples abound….

If I were a professional sporting league (for example, the AFL), clear measurements of my strength and health would be things like attendance and television ratings.  Addressing those goals could result in the biggest clubs having the most advantages as those are the clubs that most directly (and immediately) impact attendance and ratings.  Let’s call that the ‘Rule of Ed’.  But the health of the league also has a long term component and while the league registers short term improvement in goal areas, individual members may suffer.  Let’s call that the ‘Rule of Wanna Be Ed’ or ‘Rule of James Brayshaw’.  For other examples, see the Spanish Football League, which has the two biggest and best clubs in the world, but has a player’s strike because other players in the league are not being paid.

If I were an NFL scout trying to find the best players for my team, I would have to choose between hundreds of players who are essentially the same.  I need some way to differentiate between them somehow, so I go the NFL Combine.  I know all the measurable characteristics of all the best players, so now I can compare, and with each measurement I would get further away from actual ‘performance’.**  Actually that’s not much different from putting together a professional volleyball team.  Except there is no combine.  Let’s move on.

If I were a volleyball program, I could see that the best volleyball teams in the world have certain physical characteristics.  By making my goal to replicate the physical characteristics of the best teams in the world, I would have an eminently SMART goal.  But concentrating my resources on the most easily measurable area I could under resource other equally important, but less measurable areas that are equally important to ‘performance’.

If I were a volleyball coach, I could see that victory was closely related to reaching certain performance goals, for example some attack percentage or error percentage.  I will never forget talking to a well known coach who was quite surprised to discover that having worked long and hard to reach specific performance goals, and succeeding, the results (ie ‘performance’) were no different than before.

If I were a company director… I shouldn’t go there as it’s not my area of expertise but I’ve read that those who created the financial crisis still got their bonuses.  I’ve seen that somewhere else too…

Measuring things leads to lazy and short sighted thinking.

Measuring things leads to devoting resources to ever smaller, more measurable, less important areas.

Measuring things leads to losing that essential overview.

Measuring things leads to forgetting what it was we were trying to do in the first place.

Measuring things leads to worse ‘performance’.

---

* Long after originally writing this post, I discovered that this process has been identified and named in the field of economics.  It is known as ‘Goodhart’s Law’ and states that “When a measure becomes a target, it ceases to be a good measure.”  That seems to fit pretty well here. It is also implicit in the economic theory of rational expectations, that states that those who are aware of a system of rewards and punishments will optimize their actions within that system to achieve their desired results. For example, if an employee is rewarded by the number of cars sold each month, they will try to sell more cars, even at a loss.

** This point was prompted by reading a passage in the Mike Leach book about scouting players.

---

Read about the great new Vyacheslav Platonov coaching book here.