I often say that organizations are cultures, and projects are their stories.
We often use numbers to narrate these stories, lending credibility because, as the saying goes, numbers don't lie.
Or do they?
While numbers themselves are neutral, the way we use them isn't. We select, analyze, summarize, and present data with KPIs and charts - which means our interpretation can introduce biases.
Andrew Lang said it perfectly, talking about a politician:
He uses statistics as a drunken man uses lamp posts—for support rather than illumination.
This metaphor highlights the misuse of data: using statistics more for support of preconceived notions than for shedding light on the truth.
Here are two common patterns showing how KPIs can give a false impression of objectivity:
Top-Down: summarize the data by choosing metrics and KPIs that support the narrative we wish to present, regardless of whether they fully represent the reality. I call it "top-down", because it is typically used at a hierarchical level that is higher than the observed initiative - like a project sponsor demonstrating the positive effects of the project he or she promoted.
Bottom-Up: "gaming the system" by concentrating activities on producing the right numbers to paint a favourable picture, and not the results that the metrics are meant to measure. "Bottom-up" because it can be used by someone who must report to a higher authority.
To see what I mean with "gaming the system", have a laugh with this video.
More seriously, it is interesting how University of Zurich recently opted out of a prominent ranking, illustrating a critical approach against such practices.
KPIs are powerful tools, but when misused, they can distort reality instead of clarifying it.
In the spirit of "defense against the dark arts", you have to know the bad tricks in order to avoid falling victim to them.
Notes
Andrew Lang FBA (31 March 1844 – 20 July 1912) was a Scottish poet, novelist, literary critic, and contributor to the field of anthropology.