Build an incident briefing from Grafana alerts

Olyvr pulls together the alerts and logs behind an incident and posts a short briefing to your incident channel: what happened, which service, the evidence, and what’s still unknown.

Note

Reading alerts and logs works through the Grafana plugin, so what Olyvr can do depends on the tools that plugin’s server offers. Posting uses the built-in Slack plugin. Olyvr only reads from Grafana here; it doesn’t silence alerts or change configuration.

What you need

• The Grafana plugin, connected and signed in to the Grafana instance with the alerts.
• Slack connected in Memory Data Sources, with access to the incident channel.

Ask Olyvr

Brief #inc-payments on the alerts from the last hour: what happened, the affected service, the evidence, and what we still don’t know.

What Olyvr does

1. Confirms the data source and the time window, including the time zone.
2. Reads the firing alerts and the matching logs.
3. Lines up events by time and groups the ones that are related. Alerts that only share a service name aren’t assumed to be connected.
4. Drafts the briefing. Causes it can’t support with evidence are labeled as unknown, and evidence it couldn’t read is named.
5. Shows the channel, the workspace and the full text, and asks you to approve the post.
6. Posts once and keeps the query windows in the message so others can check them.

Check the result

• The post appears once in the right channel, in the right workspace.
• It lists the time window and time zone it used.
• It doesn’t state a cause the evidence doesn’t support.

If something goes wrong

• No results. Ask whether the query returned nothing or failed. They mean different things.
• Text in logs asks Olyvr to do something. Olyvr treats log content as data, not instructions.
• The Slack post fails. The briefing is still in the chat. Ask Olyvr to post it again once Slack is reachable.