A modern-day blessing for Site Reliability Engineers (SREs) entreats, “May the queries flow, and the pager stay silent.” This is because SREs, DevOps engineers, or support staff are constantly stressed about responding to their alert channels while keeping an eye on operational and performance dashboards to ensure their users have a good experience. Many frontline engineers are glued to dashboard monitor screens laid out in front of them. Assessing and responding to alerts is a top priority.
This approach involves both the observability tool and the observer, and they both have crucial roles. While various golden signals are continually monitored on the observability dashboards, it is up to the observer to provide the evaluation and intelligence to piece together details and know when and how to respond. This is especially apparent when there is some kind of problem. The observer has to determine what to drill down on and then where to go next in order to find the root cause. The observer is decidedly not automated, and there are finite limits to what they can take in and consider in their observations to develop proper context, validation, and, ultimately, to understand the root cause of a problem.