The “observer effect”, a concept having its genesis in physics, notes that the measurements of certain systems cannot be made without affecting the system itself. It is interesting to note that this physical phenomenon could also be considered to be at play in the social sciences when economic actors interact.
And such interactions could lead to both good and bad results, due to the act of measuring and depending on what is being measured.
You would expect a business that scores poorly in customer satisfaction surveys to strive to improve its products and services in order to achieve greater customer satisfaction. The race to achieve industry leadership hinges on surpassing the market share metric. One of the markers of the arrival of a start-up is reflected in its status as a unicorn. Thus, the attractiveness of the measure and its apparent relevance to achieving the desired results is undeniable. After all, “what gets measured gets managed” as the saying goes.
On the other hand, if the choice of metrics is less rigorous, this would ensure that what is being measured is played out. The Ease of Doing Business rankings saga, which involved methodological manipulation to favor rankings for certain economies, presents the dark side of questionable metrics. Determining the value of businesses based on vanity metrics such as the total number of registered accounts (as opposed to active accounts) is another approach that arguably could be disparaged. The thickening of annual reports due to the push towards more disclosure – a lofty goal in itself – tends to allow companies to mix valuable information with less substantial information and, therefore, obscure more than reveal.
History has enough examples to suggest that when a metric is confused with the larger outcome it aims to measure, it leads to unintended or sub-optimal consequences. So optimizing efforts to get results around certain narrow metrics could violate the goal itself, as the famous Cobra effect demonstrates.
Therefore, the choice of the metric and what it seeks to measure should be an important design consideration. Overall, while simplicity of design is desirable, thoroughness should not be sacrificed at the altar of simplicity.
For example, choosing simple metrics such as “averages” or “medians” to track the performance of any target parameter might be appealing, but one might lose vital information if measuring “variability” around. the mean / median is ignored. This would be analogous to an image projection of a 3D object, say a cube onto a 2D surface, which would make us see a square, but would reflect a metaphorical loss of information.
Likewise, choosing metrics that do not fully embody the essence of what is meant to be measured could be counterproductive. This only underscores the fact that not everything that matters can be measured, and not everything that can be measured does not matter. Another relevant aspect is to be clear whether the metric aims to measure the elements of the process adopted to achieve the result or to measure the result itself. There is an argument to be made that goals only serve the limited purpose of guiding the direction in which progress needs to be galvanized.
It is the “process” that needs to do the heavy lifting and therefore progress on process dimensions needs to be measured, if the efficiency and effectiveness of the process are to be improved. Indeed, this approach works in most cases.
However, it must also be recognized that the outcome should not become subordinate to the process. While companies that are among the biggest contributors to greenhouse gas emissions or companies that manufacture addictive products are seen to achieve healthy environmental, social and governance (ESG) ratings – by doing enough to ensure that they perform well on the ESG rating criteria for companies – it reflects the triumph of the process over the bottom line.
Conversely, a measurement system focused primarily on results, giving less respect to the process / path, might be useful in some contexts, but would have its own follies. This would be a credo that is a supporter of Milton Friedman, with its focus squarely on the profitability of a business, not necessarily on social good. Indeed, knowing what is intended to be measured and knowing the limits of the metrics chosen to do it, could be considered as the cornerstones of performance monitoring.
For example, if one follows the state of the economy and uses indicators such as automobile sales, air passenger traffic, hotel occupancy, sales of quick-service restaurants, income of shopping centers and others, to judge the strength of the recovery, one should be aware that these measures would only transmit the consumption patterns of a small stratum of the economy. Even if the growth impulses for these parameters were to strengthen, they would not be indicative of a generalized economic recovery.
Finally, from a systemic perspective, there is also a need for standardized measurement metrics, which could be applied uniformly to the measurands to ensure consistency and comparability. Accounting standards are one example. An illustration of the manifestation of different accounting standards prevailing in different jurisdictions was when German automaker Daimler declared DM 615 million in net profits under German accounting rules, but a loss of DM 1.84 billion according to American rules. The implications of such variations in reporting for company management and investors could be substantial. Closer to home, a few years ago, the capital markets regulator SEBI introduced standardized benchmarks of probability of default in different rating categories as measures of the performance of credit rating agencies (CRAs).
As the standards are quite demanding for the highest rating categories (NIL default rates allowed in AAA and AA categories over certain time horizons), if credit rating agencies were to “target” performance benchmarks, there could be have unwarranted conservatism creeping in while making rating decisions. The nuance is that the financial system would be better served if rating agencies tighten and maintain rating standards and assign ratings in such a way as to ensure earned credit ranking, while performance benchmarks are only used ex-post. and not ex-ante.
Philosophically, this would be tantamount to stating that success does not need to be sought, but can follow by not caring. With this, the “observer effect” in the example above could be put to rest.
(The author is responsible for credit policy, ICRA)