skip to main content
Notifiactions
  
Notifications
Introduction
In this chapter we will take a deeper look at the notification function in op5 Monitor. We will look at how the
notification works
notification skins works (mail/sms/htmlpost)
dial up notification works
snmp trap notification works.
How does notifications work?
In the op5 Monitor user manual we describe some of the basics with notifications. Let us take a closer look at how it really works.
Notification filters
When a notification is about to be sent it has to go through a number of filters before op5 Monitor can determine whether a notification really is suppose to be sent or not.
Table 1 Notification filters
Filter
Description
Program-wide
This tells op5 Monitor if notifications are turned on or not in a program-wide basis.
Service and host filters
Is the host or service in scheduled downtime or not?
Is the host or service in a flapping state?
Does the host or service notification options says that this type of notification is supposed to be sent?
Are we in the right time period for notifications at the moment?
Have we already sent a notification about this alert?
Has the host or service remained in the same non-OK state that it was when the last notification went out?
Contact filters
Does the contacts notifications options says that this type of notification is supposed to be sent?
Are we in the right time period for notifications at the moment, according to the notification time period set on the contact?
 
Notification commands
How the notifications are sent is defined in either one of the two files below:
checkcomands.cfg
misccommands.cfg
The commands are divided into
host notification commands
service notification commands
The notification commands are then using scripts in the same way as the normal check commands does.
All default scripts shipped with op5 Monitor is located in:
/opt/monitor/op5/notify
Notification macros
Many of the arguments sent to the notification commands are macros. The macros are a sort of variables containing a, in most cases, program-wide value. You can read more about macros in the Nagios manual:
http://nagios.sourceforge.net/docs/3_0/macros.html
One of the most important macro used with notifications is:
$NOTIFICATIONTYPE$
This macro tells you what type of notification that is supposed to be sent. The $NOTIFICATIONTYPE$ macro can have one of the following values.
Table 2 Notification types
Notification type
Description
PROBLEM
A service or host has just entered (or is still in) a problem state.
RECOVERY
A service or host has recovered from a problem state.
ACKNOWLEDGEMENT
A service or host in a problem state has been acknowledged by a user.
FLAPPINGSTART
The host or service has entered a flapping state.
FLAPPINGSTOP
The host or service has left a flapping state.
FLAPPINGDISABLED
The host or service flapping detection has stopped and has there fore left the flapping state.
DOWNTIMESTART
The host or service has entered a scheduled downtime.
DOWNTIMESTOP
The host or service has left a scheduled downtime.
DOWNTIMECANCELLED
The scheduled downtime for a host or service has been cancelled.
The list of macros described in the Nagios manual is very useful when you are working with new notification commands and scripts. That list can be found here:
http://nagios.sourceforge.net/docs/3_0/macrolist.html
Notification e-mail sender
Notifications are by default sent from the e-mail address ”op5monitor” without any domain. The MTA adds the local domain name, witch by default is “@localhost.localdomain”.
To change the e-mail address that notification are sent from use the --from-mail argument for the notification command.
To change the sender e-mail address from op5monitor@localhost.localdomain to op5notification@mycompany.com simply go to the check command for the host-notify and add “--from-email op5notification@mycompany.com” without the “-signs.
command_name=host-notify
command_line=$USER3$/notify/poller_notify_send.pl --from-email op5notification@mycompany.com -c "$CONTACTNAME$" -h "$HOSTNAME$" -f "$NOTIFICATIONTYPE$" -m "$CONTACTEMAIL$" -p "$CONTACTPAGER$" "HOSTALIAS=$HOSTALIAS$" "HOSTADDRESS=$HOSTADDRESS$" "HOSTSTATE=$HOSTSTATE$" "HOSTSTATEID=$HOSTSTATEID$" "HOSTSTATETYPE=$HOSTSTATETYPE$" "HOSTATTEMPT=$HOSTATTEMPT$" "HOSTLATENCY=$HOSTLATENCY$" "HOSTEXECUTIONTIME=$HOSTEXECUTIONTIME$" "HOSTDURATION=$HOSTDURATION$" "HOSTDURATIONSEC=$HOSTDURATIONSEC$" "HOSTDOWNTIME=$HOSTDOWNTIME$" "HOSTPERCENTCHANGE=$HOSTPERCENTCHANGE$" "HOSTGROUPNAME=$HOSTGROUPNAME$" "HOSTGROUPALIAS=$HOSTGROUPALIAS$" "LASTHOSTCHECK=$LASTHOSTCHECK$" "LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$" "LASTHOSTUP=$LASTHOSTUP$" "LASTHOSTDOWN=$LASTHOSTDOWN$" "LASTHOSTUNREACHABLE=$LASTHOSTUNREACHABLE$" "HOSTOUTPUT=$HOSTOUTPUT$" "HOSTPERFDATA=$HOSTPERFDATA$" "HOSTACKAUTHOR=$HOSTACKAUTHOR$" "HOSTACKCOMMENT=$HOSTACKCOMMENT$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "CONTACTALIAS=$CONTACTALIAS$" "DATETIME=$DATETIME$" "SHORTDATETIME=$SHORTDATETIME$" "DATE=$DATE$" "TIME=$TIME$" "TIMET=$TIMET$" "HOSTACTIONURL=$HOSTACTIONURL$" "HOSTNOTESURL=$HOSTNOTESURL$" "ADMINPAGER=$ADMINPAGER$" "ADMINEMAIL=$ADMINEMAIL$" "NOTIFICATIONCOMMENT=$NOTIFICATIONCOMMENT$"
This has to be done for the command “service-notify” as well.
Notification skins
The three basic notifications (email, sms and htmlpost notifications) are all using something called notification skins. The notification skins are templates describing how the notification is supposed to look like when it is sent to its receiver.
If we will take a look at the notify folder we will find the following skins folders:
skins.htmlpost/
skins.mail/
skins.sms/
Each folder contains a number of notification skins divided into host and service notification filters.
host.ACKNOWLEDGEMENT
host.FLAPPINGSTART
host.FLAPPINGSTOP
host.PROBLEM
host.RECOVERY
 
service.ACKNOWLEDGEMENT
service.FLAPPINGSTART
service.FLAPPINGSTOP
service.PROBLEM
service.RECOVERY
As you can see there is one skin for the most common notification types.
The content of a notification skin
Let us take a look at what a skin looks like.
Example 1 The sms service.PROBLEM skin
#SERVICEDESC# on #HOSTNAME# is #SERVICESTATE#. #SERVICEOUTPUT#
 
This is a very simple skin. The reason for that is that you can not send too much data with a normal sms.
Example 2 The mail service.PROBLEM skin
From: op5Monitor
To: #CONTACTEMAIL#
Subject: [op5] #NOTIFICATIONTYPE#: '#SERVICEDESC#' on '#HOSTNAME#' is #SERVICESTATE#
#extra_host_vars#
op5 Monitor
Service #NOTIFICATIONTYPE# detected #LASTSERVICESTATECHANGE#.
'#SERVICEDESC#' on host '#HOSTNAME#' has passed the #SERVICESTATE# threshold.
#STATUS_URL#
Additional info;
#SERVICEOUTPUT#
Host: #HOSTNAME#
Address: #HOSTADDRESS#
Alias: #HOSTALIAS#
Status: #HOSTSTATE#
Comment: #NOTIFICATIONCOMMENT#
Service: #SERVICEDESC#
Status : #SERVICESTATE#
Latency: Check was #SERVICELATENCY# seconds behind schedule
Misc : Check took #SERVICEEXECUTIONTIME# seconds to complete
Additional links (requires configuration);
Host actions: #HOSTACTIONURL#
Host notes: #HOSTNOTESURL#Service actions: #SERVICEACTIONURL#
Service notes: #SERVICENOTESURL#
 
The mail notifications can contain a lot more data and there we add a lot more to the mail skin file.
In both Example 1 on page 152 and Example 2 on page 152 you find text like:
#SERVICEDESC#
#HOSTNAME#
That text is called keywords.
The keywords will be replaced with the value of a command line argument looking like this:
FOO=BAR
So a command line argument like the one above will generate a keyword with the name FOO having the value BAR.
Note: If a notification macro, or other value sent to a corresponding keyword, is missing in the notification command it will not stop the notification from being sent. It is only the replacement that will be missing.
Creating custom notification skins
Sometimes the default notification skins needs to be changed. This shall not be done in the default folders.
To create custom notification skins
1 Go to the notify folder:
cd /opt/monitor/op5/notify
2 Create the custom-skins folder:
mkdir custom-skins
3 Copy the skins.* folders to the custom-skins folder:
cp skins.* custom-skins/
4 Make the changes you like to do and the new skins will be used at directly after you have saved the changes.
Dial-up notification
Many of the modern mobile phones are only giving you one tiny signal when a sms arrives. If you are on duty during the night you might not wake up or if you are in a very noisy environment it might take some time for you to notice the arrived sms. There for we have included a dial up notification in op5 Monitor.
This is a very simple, but effective, notification that works like this:
Table 3 Dial up notification workflow
Step
Action
1
op5 Monitor is scheduling a notification.
2
The notification goes through all the filters.
3
The notify_dial.pl script is called with the following command line:
/opt/monitor/op5/notify/notif_dial.pl <mobilephonenumber>
4
notify_dial.pl is shutting down smsd
5
notify_dial.pl tries to call the <mobilephonenumber>
If the line is busy or no one answer the call in 45 seconds notify_dial.pl will hang up and try again two more times before it quits.
6
The user answer the call and notify_dial.pl hangs up.
6
notify_dial.pl is starting up smsd again and the execution is over.
Adding a dial up notification command
This is done in two steps:
add the command
configure the contacts
To add a dial up notification command
1 Login to the op5 Monitor user interface and go to Configure.
2 Click Commands.
3 Add a new command with the following settings:
command_name notify_by_dial
command_line $USER3$/notify/notify_dial.pl "$CONTACTPAGER$"
4 Click Apply.
5 Click Save.
Configuring the contacts
To configure the contacts
1 Login to the op5 Monitor user interface and go to Configure.
2 Either open up an existing contact and create a new one.
3 On the contact set Pager to a phone number on the form like this (without the leading '+'-sign ):
46705123123
4 Set host_notification_commands and service_notification_commands to:
notify_by_dial
5 Click Apply.
6 Click Save.
Note: Make sure the contact is a member of the contact_group is associated with the correct objects.
SNMP trap notifications
op5 Monitor is shipped with the possibility to send notifications as SNMP traps. To start use the SNMP notifications you need to
add a few new commands
configure the contacts
Adding SNMP notification commands
Here we need to add two commands one for host notifications and one for service notifications.
To add a SNMP notification command
1 Login to the op5 Monitor user interface and go to Configure.
2 Click Commands.
3 Add the following new commands with the following settings:
command_name host_notify_by_snmp
command_line $USER3$/notify/notify_by_snmp.pl -H snmp.trap.host -C SNMPCOMMUNITY -t nHostNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "HOSTACKAUTHOR=$HOSTACKAUTHOR$" "HOSTACKCOMMENT=$HOSTACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "HOSTSTATETYPE=$HOSTSTATETYPE$" "HOSTATTEMPT=$HOSTATTEMPT$" "HOSTDURATIONSEC=$HOSTDURATIONSEC$" "HOSTGROUPNAME=$HOSTGROUPNAME$" "LASTHOSTCHECK=$LASTHOSTCHECK$" "LASTHOSTSTATECHANGE=$LASTHOSTSTATECHANGE$" "HOSTOUTPUT=$HOSTOUTPUT$"

command_name service_notify_by_snmp
command_line $USER3$/notify/notify_by_snmp.pl -H snmp.trap.host -C SNMPCOMMUNITY -t nSvcNotify "NOTIFICATIONTYPE=$NOTIFICATIONTYPE$" "NOTIFICATIONNUMBER=$NOTIFICATIONNUMBER$" "SERVICEACKAUTHOR=$SERVICEACKAUTHOR$" "SERVICEACKCOMMENT=$SERVICEACKCOMMENT$" "HOSTNAME=$HOSTNAME$" "HOSTSTATEID=$HOSTSTATEID$" "SERVICEDESCRIPTION=$SERVICEDESCRIPTION$" "SERVICESTATEID=$SERVICESTATEID$" "SERVICEATTEMPT=$SERVICEATTEMPT$" "SERVICEDURATIONSEC=$SERVICEDURATIONSEC$" "SERVICEGROUPNAME=$SERVICEGROUPNAME$" "LASTSERVICECHECK=$LASTSERVICECHECK$" "LASTSERVICESTATECHANGE=$LASTSERVICESTATECHANGE$" "SERVICEOUTPUT=$SERVICEOUTPUT$"

Change the following to their correct value, in both commands:
snmp.trap.host
SNMPCOMMUNITY
4 Click Apply.
5 Click Save.
Configuring the contacts
To configure the contacts
1 Login to the op5 Monitor user interface and go to Configure.
2 Either open up an existing contact or create a new one.
3 Set host_notification_commands to:
host_notify_by_snmp
4 Set service_notification_commands to:
service_notify_by_snmp
5 Click Apply.
6 Click Save.
Note: Make sure the contact is a member of the contact_group is associated with the correct objects.
 
to