Metric loss

Why time saved on intranets is always a bad metric

The element of time, which not so long ago drove the analytics industry, is now a given

PHOTO: Veri Ivanova

Fourteen years ago, I wrote an article titled “25 Reasons Why Saving Time on Your Intranet Is Bad Metrics.” Over the years I’ve written thousands of articles (and several books), but this blog post remains a favorite for more than a few practitioners.

The argument goes: we’ve done usability testing and it takes an average of 2.5 minutes for a user to find the leave policy. We improved navigation and structure of the intranet, and re-tested, it only takes 30 seconds. That’s a savings of 2 minutes, multiplied by the number of users and the number of times they are expected to do so in a year, and the result is 150,000 minutes saved!

At the time, a leading usability expert went further, claiming not only that companies with average intranet usability could save $2.4 million per year by improving user experience, but also that “the global economy would save $311,294,070,513 a year”. There are still many organizations that focus primarily – or exclusively – on using intranet time savings to build business cases or demonstrate benefits.

I had a big problem with these approaches and laid out 25 reasons why they were fatally flawed. More than a decade later, I return to the fray.

Real vs. “false” time savings

Let me start by acknowledging that in the era of sudden labor shortages, productivity gains are extremely important for organizations. If it is possible to save time during a key stage in the manufacture of a car, more cars will be able to roll off the production line. These kinds of simplifications and automations can deliver real, and often substantial, benefits.

Similar benefits can be provided by digital means. In 2020, Step Two’s Intranet & Digital Workplace Awards awarded a gold trophy to SP Group (Singapore’s power utility) for automate frontline processes using Power Platform in Microsoft 365.

SP Group conducted a “value stream map” to identify tangible time savings that would allow the field workforce to be more productive:

value stream mapping

Compared to the real benefits identified by SP Group, the “false” benefits focus only on the time saved by searching for items on the intranet, ie usability in the narrow sense.

It’s these less tangible benefits that can get us all in trouble, and that’s what I want to denounce. Let’s revisit the issues I raised 14 years ago, to see if they still apply.

Related article: Intranet Platforms 2022: The State of the Industry

The big problems with “time-saving” intranet metrics

Here are my main concerns regarding the measurement of time saved:

1. We do not measure end-to-end task completion. The “time saving” metric typically focuses only on the time it takes to find information on the intranet, not on the task or activity itself.

2. Can we realize the value? In financial jargon, financial benefits must be “realized” before they count. In other words, if we “save $1.7 million” when the CEO shows up at our office, can we provide it in cash or the equivalent?

3. Time saving and productivity are complex. We’ve saved each staff member two minutes a day, so are they now spending all that time productively? Or do they spend it reading the newspaper or chatting on Facebook?

4. We multiply a lot of hypotheses. How valid is a number when we have to make a dozen different assumptions and then multiply them all together?

5. Many financiers will not accept it. In many organizations, CFOs and their staff simply don’t accept these types of metrics, for the intranet or any other system. Better check first!

To this I added a long list of other problems, including:

6. It’s just a matter of usability. Measuring the time spent on the intranet is simple, but not very impactful.

7. How many employees use the intranet? We comfortably multiply the time saved by the number of employees in the organization, but that seems somewhat optimistic. How many employees actually use the intranet?

8. Increased usage is now costing us money. We argued that reducing the time spent on the intranet would save the organization money, but at the same time, we aim to increase intranet adoption. According to this argument, the more users there are on the intranet, the more money we lose. We cannot have it both ways.

9. The intranet is considered in isolation. The intranet sits alongside – and competes with – all the other means of accomplishing a task or obtaining an answer. To validate our calculations, we must assume that the intranet is the only source of information.

10. We don’t consider experience. Experienced staff just know how things work, and the answers to common questions and issues, so they don’t have to search the intranet for information.

11. What about learning? No matter how bad an intranet is, given enough time, staff will often learn to use what they need (or give up altogether).

12. How often are these tasks performed? Savings are usually calculated based on the before-and-after testing of a number of “common” intranet tasks, such as finding a phone number or requesting time off. How often does a staff member actually perform these tasks? In the case of a request for leave, it can only be once or twice a year, certainly not daily.

13. Are we testing the right tasks? In order to be relevant to a wide range of employees, testing tends to focus on general intranet activities. These generic tasks often relate to the staff directory, HR, finance, news, etc. that are not directly related to the day-to-day activities of staff.

14. Can we generalize to the entire intranet? We tested a few “representative” tasks and saw time savings before and after. But this is only for 6-12 tasks. Can this be generalized to time savings across the entire intranet?

15. What are we actually testing? A typical task might be how to request time off. Should it be a fast or slow task? Do we want people to read the policy carefully (maybe) or are we just measuring how long it takes to find it?

16. We overestimate the value of these measures. As indicative results, they are of great value. But many organizations go way beyond that, claiming millions of dollars in savings, which is dangerous territory.

17. We treat our managers like fools. The many numbers involved in justifying time savings sound like “blinding by science”. By taking this approach, we risk treating our managers and stakeholders like fools, selling them vague numbers as concrete business cases.

18. Negative business cases are dangerous. Once we get started down the path of negative business cases based on the money saved, we can be in a lot of trouble. Wouldn’t the quickest and easiest way to save money be to fire the intranet team and delegate the management of the intranet entirely?

Related article: Why Collaboration Analytics is Critical to Employee Experience

Measure real benefits

Now is the time for me to admit the use of these time-saving measures in our Stage 2 consulting work… Used appropriately, they significantly demonstrate that we have improved the architecture of information from the intranet, but that’s where we leave off.

For intranet teams, the real opportunity is to make the intranet a ‘place to do stuff’, not just a ‘place to read stuff’. It then becomes much easier to demonstrate real benefits, including quantifiable dollar savings.

Teams should focus on demonstrating improvements to end-to-end tasks and providing direct solutions to business needs. The closer the intranet gets to supporting a company’s day-to-day operational needs (often related to frontline workers), the easier it becomes to demonstrate tangible results.

Where do you stand on this argument? Share your thoughts, let’s continue the discussion.