Nagios

On-Call Rotations


Up ToUp To: Contents
See Also See Also: Timeperiods, Notifications

Introduction

Contacts Timeperiods

Admins often have to shoulder the burden of answering pagers, cell phone calls, etc. when they least desire them. No one likes to be woken up at 4 am to fix a problem. But its often better to fix the problem in the middle of the night, rather than face the wrath of an unhappy boss when you stroll in at 9 am the next morning.

For those lucky admins who have a team of gurus who can help share the responsibility of answering alerts, on-call rotations are often setup. Multiple admins will often alternate taking notifications on weekends, weeknights, holidays, etc.

I'll show you how you can create timeperiod definitions in a way that can facilitate most on-call notification rotations. These definitions won't handle human issues that will inevitably crop up (admins calling in sick, swapping shifts, or throwing their pagers into the river), but they will allow you to setup a basic structure that should work the majority of the time.

Scenario 1: Holidays and Weekends

Two admins - John and Bob - are responsible for responding to Nagios alerts. John receives all notifications for weekdays (with 24 hour days), excluding holidays; Bob handles notifications during the weekends and holidays. Lucky Bob. Here's how you can define this type of rotation using timeperiods...

First, define 3 timeperiods that contains time ranges for holidays, weekdays, and weekends:

define timeperiod{
	name			weekdays
	timeperiod_name		weekdays
	monday			00:00-24:00
	tuesday			00:00-24:00
	wednesday		00:00-24:00
	thursday		00:00-24:00
	friday			00:00-24:00
	}

define timeperiod{
	name			weekends
	timeperiod_name		weekends
	saturday		00:00-24:00
	sunday			00:00-24:00
	}
define timeperiod{
	name			holidays
	timeperiod_name		holidays
	january 1		00:00-24:00	; New Year's Day
	2008-03-23		00:00-24:00	; Easter (2008)
	2009-04-12		00:00-24:00	; Easter (2009)
	monday -1 may		00:00-24:00	; Memorial Day (Last Monday in May)
	july 4			00:00-24:00	; Independence Day
	monday 1 september	00:00-24:00	; Labor Day (1st Monday in September)
	thursday 4 november	00:00-24:00	; Thanksgiving (4th Thursday in November)
	december 25		00:00-24:00	; Christmas
	december 31		17:00-24:00	; New Year's Eve (5pm onwards)
	}

Next, define a timeperiod for John's on-call times that include weekdays, but excludes the dates/times defined in the holidays timeperiod above:

define timeperiod{
	timeperiod_name		john-oncall
	use			weekdays	; Include weekdays
	exclude			holidays	; Exclude holiday dates/times defined elsewhere
	}

You can now reference this timeperiod in John's contact definition:

define contact{
	contact_name			john
	...
	host_notification_period	john-oncall
	service_notification_period	john-oncall
	}

Define a new timeperiod for Bob's on-call times that include weekends and the dates/times defined in the holidays timeperiod above:

define timeperiod{
	timeperiod_name		bob-oncall
	use			weekends,holidays	; Include weekend and holiday date/times defined elsewhere
	}

You can now reference this timeperiod in Bob's contact definition:

define contact{
	contact_name			bob
	...
	host_notification_period	bob-oncall
	service_notification_period	bob-oncall
	}

Scenario 2: Alternating Days

In this scenario John and Bob alternate handling alerts every other day - regardless of whether its a weekend, weekday, or holiday.

Define a timeperiod for when John should receive notifications. Assuming today's date is August 1st, 2007 and John is handling notifications starting today, the definition would look like this:

define timeperiod{
	timeperiod_name		john-oncall
	2007-08-01 / 2		00:00-24:00	; Every two days, starting August 1st, 2007
	}

Now define a timeperiod for when Bob should receive notifications. Bob gets notifications on the days that John doesn't, so his first on-call day starts tomorrow (August 2nd, 2007).

define timeperiod{
	timeperiod_name		bob-oncall
	2007-08-02 / 2		00:00-24:00	; Every two days, starting August 2nd, 2007
	}

Now you need to reference these timeperiod definitions in the contact definitions for John and Bob:

define contact{
	contact_name			john
	...
	host_notification_period	john-oncall
	service_notification_period	john-oncall
	}
define contact{
	contact_name			bob
	...
	host_notification_period	bob-oncall
	service_notification_period	bob-oncall
	}

Scenario 3: Alternating Weeks

In this scenario John and Bob alternate handling alerts every other week. John handles alerts Sunday through Saturday one week, and Bob handles alerts for the following seven days. This continues in perpetuity.

Define a timeperiod for when John should receive notifications. Assuming today's date is Sunday, July 29th, 2007 and John is handling notifications this week (starting today), the definition would look like this:

define timeperiod{
	timeperiod_name		john-oncall
	2007-07-29 / 14	00:00-24:00	; Every 14 days (two weeks), starting Sunday, July 29th, 2007
	2007-07-30 / 14	00:00-24:00	; Every other Monday starting July 30th, 2007
	2007-07-31 / 14	00:00-24:00	; Every other Tuesday starting July 31st, 2007
	2007-08-01 / 14	00:00-24:00	; Every other Wednesday starting August 1st, 2007
	2007-08-02 / 14	00:00-24:00	; Every other Thursday starting August 2nd, 2007
	2007-08-03 / 14	00:00-24:00	; Every other Friday starting August 3rd, 2007
	2007-08-04 / 14	00:00-24:00	; Every other Saturday starting August 4th, 2007
	}

Now define a timeperiod for when Bob should receive notifications. Bob gets notifications on the weeks that John doesn't, so his first on-call day starts next Sunday (August 5th, 2007).

define timeperiod{
	timeperiod_name		bob-oncall
	2007-08-05 / 14	00:00-24:00	; Every 14 days (two weeks), starting Sunday, August 5th, 2007
	2007-08-06 / 14	00:00-24:00	; Every other Monday starting August 6th, 2007
	2007-08-07 / 14	00:00-24:00	; Every other Tuesday starting August 7th, 2007
	2007-08-08 / 14	00:00-24:00	; Every other Wednesday starting August 8th, 2007
	2007-08-09 / 14	00:00-24:00	; Every other Thursday starting August 9th, 2007
	2007-08-10 / 14	00:00-24:00	; Every other Friday starting August 10th, 2007
	2007-08-11 / 14	00:00-24:00	; Every other Saturday starting August 11th, 2007
	}

Now you need to reference these timeperiod definitions in the contact definitions for John and Bob:

define contact{
	contact_name			john
	...
	host_notification_period	john-oncall
	service_notification_period	john-oncall
	}
define contact{
	contact_name			bob
	...
	host_notification_period	bob-oncall
	service_notification_period	bob-oncall
	}

Scenario 4: Vacation Days

In this scenarios, John handles notifications for all days except those he has off. He has several standing days off each month, as well as some planned vacations. Bob handles notifications when John is on vacation or out of the office.

First, define a timeperiod that contains time ranges for John's vacation days and days off:

define timeperiod{
	name			john-out-of-office
	timeperiod_name		john-out-of-office
	day 15			00:00-24:00	; 15th day of each month
	day -1			00:00-24:00	; Last day of each month (28th, 29th, 30th, or 31st)
	day -2			00:00-24:00	; 2nd to last day of each month (27th, 28th, 29th, or 30th)
	january 2		00:00-24:00	; January 2nd each year
	june 1 - july 5		00:00-24:00	; Yearly camping trip (June 1st - July 5th)
	2007-11-01 - 2007-11-10	00:00-24:00	; Vacation to the US Virgin Islands (November 1st-10th, 2007)
	}

Next, define a timeperiod for John's on-call times that excludes the dates/times defined in the timeperiod above:

define timeperiod{
	timeperiod_name		john-oncall
	monday			00:00-24:00
	tuesday			00:00-24:00
	wednesday		00:00-24:00
	thursday		00:00-24:00
	friday			00:00-24:00
	exclude			john-out-of-office	; Exclude dates/times John is out
	}

You can now reference this timeperiod in John's contact definition:

define contact{
	contact_name			john
	...
	host_notification_period	john-oncall
	service_notification_period	john-oncall
	}

Define a new timeperiod for Bob's on-call times that include the dates/times that John is out of the office:

define timeperiod{
	timeperod_name		bob-oncall
	use			john-out-of-office	; Include holiday date/times that John is out
	}

You can now reference this timeperiod in Bob's contact definition:

define contact{
	contact_name			bob
	...
	host_notification_period	bob-oncall
	service_notification_period	bob-oncall
	}

Other Scenarios

There are a lot of other on-call notification rotation scenarios that you might have. The date exception directive in timeperiod definitions is capable of handling most dates and date ranges that you might need to use, so check out the different formats that you can use. If you make a mistake when creating timeperiod definitions, always err on the side of giving someone else more on-call duty time. :-)