Because it’s such a comprehensive metric, a high MTTR metric can indicate warning issues or that your engineers are spending a lot of time on repairs. Therefore, it is essential to examine the MTTR over time and analyze each component of your incident management workflow: it’s time to alert engineers, diagnose the problem, test fixes, to ship to production, make revisions and learn from the incident.
It can also be useful to look at mean time to resolution in conjunction with other metrics. To determine if your DevOps team is facing production challenges, assess your change failure rate (CFR) to see how many versions result in downgraded service. Other DORA Metricslike deployment frequency and change time, are perfect companions to mean time to resolution.
To establish the reliability of your software, you can look at the mean time to resolution next to the mean time between failures (MTBF), which calculates the mean time between incidents. If you update your software often, compare the mean time to resolution with the mean time to failure (MTTF), which measures how long it takes before a program needs to be redesigned for functionality. To better understand your alerting processes, look at Mean Time to Detection (MTTD), which measures the time it takes for your team to recognize that a problem exists.
How to improve mean time to resolution
Alerting is the first step in incident response and should be one of the first areas to focus on when working to reduce mean time to resolution. Ensure alerts are actionable and DevOps team members have the tools they need to take immediate action. A simple escalation process is essential: define each member’s responsibilities and train the team on each person’s role so that the process never stops if someone is unavailable.
Preventative monitoring can help you anticipate problems before they occur. By proactively checking for potential incidents, you can avoid unplanned downtime.
The best way to improve MTTR is to standardize your operating procedures with runbooks. Without runbooks, DevOps teams have to react without clear direction and spend time messaging each other for information. They cannot act immediately. With runbooks, however, your organization’s knowledge base is centralized and accessible to all team members, allowing them to respond as soon as an issue arises.
If you already use runbooks, consider automating responses. Automation not only improves your mean time to resolution, but gives your DevOps team more time to spend on long-term implementation changes that improve the stability of your service.