In the summer of 2023, my team entered into a code yellow to stabilize the health of the service we own. This service powers the visualization on the dashboard product. The decision was made following high-severity incidents impacting the availability of the service.
For context, the service provides aggregation data to dashboard visualizations by gathering aggregations through the data pipeline. This is a critical path service for dashboard rendering. Any impact on the availability of this service manifests itself as dashboard viewers experiencing delays in rendering visualizations and rendering failures in some cases.