What you measure is what you get!

Communication occurs at many levels and the result of any one communication is based on what the receiver perceives as important[1]. Asking questions can be a very effective way of communicating and influencing stakeholders[2], but asking the wrong questions can easily be the cause of undesirable outcomes; care is needed!

One of the key communication channels in most projects is the request from PMOs and similar bodies to project management teams for information and the supply of the data requested. Certainly quality guru W. Edwards Deming said ‘In God we trust, all others bring data’, and a modern management mantra is ‘you cannot manage what you cannot measure’, so the collecting of data is important, you also have to make sure you’re measuring, and communicating, the right things.

Whilst exchanging data with the PMO may not look like a communication, and is very often structured within formal policies, the reality can be very different; if you ignore what’s actually being communicated, you can get very undesirable outcomes.

A policy introduced to measure the performance of our local hospitals a few years ago offers a salient lesson. Our State government decided to ‘incentivize’ its hospitals by rewarding good performance and fining them for poor performance, using a standard set of Key Performance Indicators (KPIs).

One key measure was the time patients wait in casualty/emergency area before being admitted to the hospital. To achieve a good KPI, some administrators were manipulating data to avoid fines and win bonuses – both adjusted against the hospital’s funding for the year (ie, no one personally benefitted from the manipulation). But the problems only surfaced after an audit report found KPI induced behaviors that in many cases were worse for the patients and worse for the hospital than if nothing had occurred.

One example was the practice of transferring patients from emergency care to the operating theatre waiting area to avoid a fine for failing to admit the patient within the prescribed maximum time. The consequences of this action included:

• The patient being removed from an area focused on emergency care to an area with little monitoring capability – a reduction in care to the patient.
• The reduction in throughput in the operating suites due to overcrowding and skilled theatre staff having to spend time on patient care rather than operations.
• A net reduction in the overall delivery of service by the hospital; but an improved statistical report to Government.

This structure of the KPI induced behavior was in the interests of everyone except the people the Government and hospital are supposed to serve, the public needing hospital care. Some of the vested interests included:

• The government’s desire to look good by ‘reducing’ patient waiting time.
• The hospitals desire to achieve the maximum budget income for the year.

• The administrators’ desire, both in the hospital and the government, to avoid ‘rocking the boat’.

The Government had its data and the system responded to the stimulus of the KPI, but everyone forgot the key objective – enhanced patient care. Management consultant John Blakemore tells a different story focused on the correctness of the KPI measurements. He was engaged by the management of a precision injection moulding facility to help overcome production problems and help the plant double its capacity.  The current plan was to invest $20 million to install 14 new machines and his contract was to help improve the production of the existing plant.  As a starting point, he trained the operators in basic data collection and as a consequence the reported defect rate jumped from 15% to its real rate of over 45%. This change was solely due to the new measurement system that accurately recorded the machine performance.  The old system measured the ‘defective parts’ in the output bins; the new system compared the ‘good parts’ with the machine capacity of 176 units per injection cycle. The difference was caused by a work practice of blocking off defective cavities in the mould (to avoid wasting materials).  The consequence was each production cycle was producing less than 100 pieces rather than the expected 176.  Management were completely unaware of this practice.

Over the next few weeks, the plant workers focused on rectifying the defects in the dies and improved the general maintenance of the machines. As a consequence, the defect rate dropped to almost zero, output almost doubled and the original need for the expansion was eliminated. There are a number of important lessons in these stories for all PMO managers, portfolio, program and project managers to consider when setting up ‘project dash boards’ and the like:

• The KPIs you choose are communicating information to stakeholders on what you think is ‘most important’ – what is easy to measure is not necessarily important.
• What you choose to measure will change behaviors. Focus on things that matter like value and benefits not easy to measure statistics like time and cost.
• Make sure the data you use is validated.
• When studying any system to set up KPIs go directly to the source; the Japanese call this ‘genchi genbutsu’ – go see for yourself!  This requires a trusting and open communication environment to be most effective.
• A KPI system can not solve ‘the problem’ but it can be a powerful facilitator of solutions if it is set to measure the right statistics and ask the right questions.

But remember, if you pick the wrong KPI you will get behavior changes that are worse than before the KPI was introduced.  This paradox operates like Gresham’s Law with a twist: the easy to measure drives out the harder to quantify, even when the latter is more important.

Gresham\’s law is an economic principle that states: ‘When a government compulsorily overvalues one type of money and undervalues another, the undervalued money will leave the country or disappear from circulation into hoards, while the overvalued money will flood into circulation’. It is commonly stated as: ‘Bad money drives out good’, or in our case simplistic information drives out useful information. Is it really worth destroying $thousands in value by de-scoping a project simply to avoid finishing two weeks later than planed?  Or is it better to have an informed conversation with the key stakeholders over value and what really matters?

Simply identifying a problem and creating a KPI is not enough! You should work with the project team and to make sure an effective solution is crafted, and then measure the effectiveness of the solution. This is far more challenging than simply processing monthly reports on easily accessed information such a schedule performance, but is also one that can really contribute to the overall performance of your organization.

Project reports are part of the project communication framework and the purpose of communication is to achieve an effect – you just need to make sure what you are communicating will actually achieve the effect you desire. What messages are the KPIs in your monthly reports sending?

[1] For more on the basic communication model see:  http://www.mosaicprojects.com.au/WhitePapers/WP1066_Communcation_Theory.pdf

[2] For more on effective questioning see: http://www.mosaicprojects.com.au/WhitePapers/WP1012_Active_Listening.pdf