Alerting Configuration

When deciding what alert conditions to set, the following questions may help your team clarify your needs so you can make an informed decision:

  • What piece of data, or set of data, indicates that your app is healthy, having degraded performance, or is down/failed?
  • What conditions are most likely to impact your customers negatively?
  • What conditions indicate imminent negative impacts? (early warning signs)
  • What are some indicators of impact to/from upstream or downstream services?
  • What conditions in your app need someone to be woken up in the middle of the night to perform manual intervention?
  • What conditions in your app are interesting – useful for operational support, but don’t need a late-night phone call?
  • What is your team’s preferred method of communication?
  • Who will actually respond?

Effective alerts are meaningful and actionable, specific, and written in plain language. If you find your alerts to be “noise” and ignore them, remove the alert.

Another way to consider it, according to a well-known monitoring organization, is:

  • Notify liberally; page judiciously
  • Page on symptoms, rather than causes

Use Cases

US and Bangalore Share On-call

Say your product has a global support model with a US based team covering on-call support during a portion of the day and a Bangalore based team covering on-call support for the other portion of the day, here’s what you’ll want to do:

For the purposes of this example, we’ll assume:

  • A weekly on-call rotation.
  • The US based team will be on-call from 9:00am to 9:00pm CST and Bangalore will be on-call from 9:00pm to 9:00am the following day.
  • The US based team is located in Minneapolis.

Here’s what you’ll need to create:

  • Two Rotations one for the Minneapolis based team and one for the Bangalore based team.
  • On both rotations set the Rotation Type to weekly and set the Time Zone to America/Chicago.
  • On the Minneapolis rotation set the Handoff Time to 9:00am.
  • On the Bangalore rotation set the Handoff Time to 9:00pm.
  • One Schedule that the Minneapolis and Bangalore team will share. Add both the Minneapolis and Bangalore rotations to the schedule as assignments.
  • On the schedule assignment for the Minneapolis team rotation, edit the Assignment Rule with a start time of 9:00am and an end time of 9:00pm.
  • On the schedule assignment for the Bangalore team rotation, edit the Assignment Rule with a start time of 9:00pm and an end time of 9:00am.
  • One Escalation Policy that includes the one schedule just created.
  • One Service that points to the one escalation policy you just created.