skip to main content

Distributed monitoring

Distributed monitoring
About the distributed monitoring
This chapter covers the following topics:
Introduction
The op5 Monitor back end can easily be configured to be used as a distributed monitoring solution. The distributed model looks like this.
In the distributed monitoring solution
*
*
*
*
Before we start
There are a few things you need to take care of before you can start setting up a distributed monitoring solution. You need to make sure
*
you have at least two op5 Monitor servers of the same architecture up and running.
*
*
*-
15551, op5 Monitor back end communication port
*-
22, ssh (for the configuration sync).
*-
*
Make sure the host group, the one the poller will be responsible for, is added to the master configuration and that at least one host is added to that host group.
The configuration
Setting up the new distributed monitoring solution
This distributed configuration will have one master and one poller:
*
*
The poller will be monitoring the host group gbg.
During the setup we will use the command:
mon
The mon command is used to make life a bit easier when it comes to setting up a load balanced solution. To get more detailed information about the command mon just execute like this:
mon --help
To setup a distributed monitoring solution with one poller
1
2
Add the new poller to the configuration with the following command:
mon node add poller01 type=poller hostgroup=gbg
3
Create and add ssh keys to and from the second peer by
as root user:
mon sshkey push --all
mon sshkey fetch --all
4
Add master01 as master at poller01:
mon node ctrl --type=poller -- mon node add master01 type=master
5
Set up the configuration sync:
dir=/opt/monitor/etc/oconf
conf=/opt/monitor/etc/nagios.cfg
mon node ctrl -- sed -i /^cfg_file=/d $conf
mon node ctrl -- sed -i /^log_file=/acfg_dir=$dir $conf
mon node ctrl -- mkdir -m 775 $dir
mon node ctrl -- chown monitor:apache $dir
6
To make sure you have an empty configuration on poller01:
mon node ctrl -- mon oconf hash

This will give you an hash looking like this (“da 39” -hash):
da39a3ee5e6b4b0d3255bfef95601890afd80709
7
8
Adding a new poller
In this instruction we will add a new poller to our distributed solution. Here we have the following hosts:
*
*
To add a new poller
1
2
Add the new poller to the configuration with the following command:
mon node add poller02 type=poller hostgroup=gbg
3
Create and add ssh keys for the root user:
mon sshkey push poller02
mon sshkey fetch poller02
4
Add master01 as master at poller02:
mon node ctrl poller02 -- mon node add master01 type=master
5
Set up the configuration sync:
dir=/opt/monitor/etc/oconf
conf=/opt/monitor/etc/nagios.cfg
mon node ctrl poller02 -- sed -i /^cfg_file=/d $conf
mon node ctrl poller02 -- sed -i /^log_file=/acfg_dir=$dir $conf
mon node ctrl poller02 -- mkdir -m 775 $dir
mon node ctrl poller02 -- chown monitor:apache $dir
6
To make sure you have an empty configuration on poller01:
mon node ctrl poller02 -- mon oconf hash

This will give you an hash looking like this (“da 39” -hash):
da39a3ee5e6b4b0d3255bfef95601890afd80709
7
8
Restart and push the logs from master01 from poller01:
mon restart; sleep 3; mon oconf push
Adding a new host group to a poller
You might want to add an other host group for to a poller. You need to edit the merlin.conf file to do that. This is not doable with any comand as it is today.
To add new host group to a poller
1
Open up and edit /opt/monitor/op5/merlin/merlin.conf.
2
Add a new host group in the hostgroup line like this:
hostgroup = gbg,sth,citrix_servers

Remember to not put any space between the hostgroup name and comma.
3
4
Removing a poller
In this instruction we will remove a poller called:
poller01
The poller will be removed from the master configuration and all distributed configuration on the poller will also be removed.
To remove a poller
1
2
Deactivate and remove all distributed setup on the poller host.
mon node ctrl poller01 -- mon node remove master01
3
Restart monitor on the poller.
mon node ctrl poller02 -- mon restart
4
5
 
Master takeover
If a poller goes down the default configuration is for the master to take over all the checks from the poller. For this to work all hosts monitored from the poller most also be monitorable from the master.
If the master server not should take over the checks from the poller this can be set in the merlin configuration file.
To stop the master from taking over, edit the file /opt/monitor/op5/merlin/merlin.conf
Add the following to the poller that you want the master not to take over.
takeover = no
Note that this is done per poller.
File synchronization
To synchronize files from the master server to the poller add a sync paragraph in the file /opt/monitor/op5/merlin/merlin.conf
In the example below we will synchronize the htpasswd.users file to the poller “poller01”
poller poller01 {
address = <ip>
port = <port>
contact_group = <contactgroup>
 
sync {
/opt/monitor/etc/htpasswd.users /opt/monitor/etc/htpasswd.users
}
}
Note that this is done per poller
One way connections
If one peer is behind some kind of firewall or is on a NAT adress it might not be possilbe for the master server to connect to the peer.
To tell the master not to connect to the poller and let the poller open the session we need to add a option to the file /opt/monitor/op5/merlin/merlin.conf.
Under the section for the poller that the master should not try to connect to add the following:
connect = no
Example
In the example below we have a master “master01” that can not connect to “poller01” but “poller01” is allowed to connect to “master01”.
poller poller01 {
address = <ip>
port = <port>
contact_group = <contactgroup>
connect = no
}
Is is also possible to set this option on the peer instead then the master will always initiate the session.
Recovery
After a poller as been unavailable for a master (i.e of network outage) the report data will be synced from the poller to the master.
The report data on the poller will overwrite the data on the master system
More information
For more information and a more complex example please take a look at the howto in the git repository of the opensource project of Merlin:
http://git.op5.org/git/?p=nagios/merlin.git;a=blob;f=HOWTO;hb=master#l171

Feedback on documentation Feedback on documentation