Multiple Nagios Instances on a Single Server
In this post I will cover how to run Multiple Nagios Instances on a Single Server. This will take a single installation of Nagios and allow you to run separate instances as you would to separate out different environments, such as production, development, certification, etc….
It allows for flexibility of configurations and allows Nagios to run faster for checks and execution. This installation scenario is based on RedHat 5.x installation with Nagios 3.2.0, Nagios Plugins 1.4.13, Nagiosgraph, rrdtool and LDAP integration.
Installation
Download and Install Boutell’s GD library
wget http://www.libgd.org/releases/gd-2.0.35.tar.gz ./configure make install
Compile and install the Nagios and Plugins
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.0.tar.gz wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.13.tar.gz tar -xzf nagios-3.2.0.tar.gz tar -xzf nagios-plugins-1.4.13.tar.gz
export NAGIOSHOME=/usr/local/nagios cd nagios-3.2.0 ./configure --enable-embedded-perl --with-command-group=nagios --with-gd-lib=/usr/local/lib --with-gd-inc=/usr/local/include; make all; make install install-init;make-install-config;make install-commandmode
Compile and install the Nagios plugins
cd nagios-plugins-1.4.13 ./configure --prefix=$NAGIOSHOME/core/nagios-plugins --with-nagios-user=nagios --with-nagios-group=nagios; make ; make install
File Copy and Directory Creation
Create directories which hold your instances and their subdirectories
NAGIOSHOME=/usr/local/nagios mkdir $NAGIOSHOME/instances mkdir $NAGIOSHOME/instances/nagios_prod mkdir $NAGIOSHOME/instances/nagios_prod/etc mkdir $NAGIOSHOME/instances/nagios_prod/etc/monitors mkdir $NAGIOSHOME/instances/nagios_prod/var mkdir $NAGIOSHOME/instances/nagios_prod/var/rw mkdir $NAGIOSHOME/instances/nagios_prod/var/spool mkdir $NAGIOSHOME/instances/nagios_prod/var/spool/checkresults mkdir $NAGIOSHOME/instances/nagios_prod/share mkdir $NAGIOSHOME/instances/nagios_prod/sbin
Create Symbolic Links
ln -s $NAGIOSHOME/nagios/bin $NAGIOSHOME/instances/nagios_prod/bin ln -s $NAGIOSHOME/nagios-plugins/libexec $NAGIOSHOME/instances/nagios_prod/libexec
Copy files
cp -r $NAGIOSHOME/sbin/* $NAGIOSHOME/instances/nagios_prod/sbin/ cp –r $NAGIOSHOME/nagios/share/* $NAGIOSHOME/instances/nagios_prod/share/ cp $NAGIOSHOME/etc/cgi.cfg $NAGIOSHOME/instances/nagios_prod/etc/ cp $NAGIOSHOME/etc/resource.cfg $NAGIOSHOME/instances/nagios_prod/etc/ cp $NAGIOSHOME/etc/nagios.cfg $NAGIOSHOME/instances/nagios_prod/etc/
Do the same with all the other instances you are going to deploy.
Configuration
In the editing of the Nagios config files there will be instances that point to a central shared location and to individual settings. The configurations that are shared are checkcommands.cfg, misccommands.cfg, contacts.cfg, contactgroups.cfg, timeperiods.cfg, templates.cfg, dependencies.cfg & escalations.cfg.
The individual ones are the specific monitors and hosts, which in the nagios.cfg we will state a directory instead of the individual configuration files.
nagios.cfg modifications
Example:
log_file=/usr/local/nagios/instances/nagios_prod/var/nagios.log # OBJECT CONFIGURATION FILE(S) cfg_file=/usr/local/nagios/etc/objects/checkcommands.cfg cfg_file=/usr/local/nagios/etc/objects/misccommands.cfg cfg_file=/usr/local/nagios/etc/objects/contacts.cfg cfg_file=/usr/local/nagios/etc/objects/contactgroups.cfg cfg_file=/usr/local/nagios/etc/objects/timeperiods.cfg cfg_file=/usr/local/nagios/etc/objects/templates.cfg cfg_file=/usr/local/nagios/etc/objects/dependencies.cfg cfg_file=/usr/local/nagios/etc/objects/escalations.cfg # Definitions for monitoring cfg_dir=/usr/local/nagios/instances/nagios_prod/etc/monitors object_cache_file=/usr/local/nagios/instances/nagios_prod/var/objects.cache precached_object_file=/usr/local/nagios/instances/nagios_prod/var/objects.precache resource_file=/usr/local/nagios/instances/nagios_prod/etc/resource.cfg status_file=/usr/local/nagios/instances/nagios_prod/var/status.dat command_file=/usr/local/nagios/instances/nagios_prod/var/rw/nagios.cmd lock_file=/usr/local/nagios/instances/nagios_prod/var/nagios.lock temp_file=/usr/local/nagios/instances/nagios_prod/var/nagios.tmp log_archive_path=/usr/local/nagios/instances/nagios_prod/var/archives state_retention_file=/usr/local/nagios/instances/nagios_prod/var/retention.dat debug_file=/usr/local/nagios/instances/nagios_prod/var/nagios.debug
CGI.CFG File Edits
The cgi.cfg file must be edit for Apache to be able to run the correct CGI scripts for each instance you plan to run. To follow the examples in previous sections we use nagios_prod as our instance.
main_config_file=/usr/local/nagios/instances/nagios_prod/etc/nagios.cfg physical_html_path=/usr/local/nagios/instances/nagios_prod/share url_html_path=/nagios_prod/
Resource.cfg file edits
$USER1$=/etc/nagios/instances/nagios_prod/libexec
Create nagios configuration for Apache
Create a nagios_prod.conf file in the /etc/httpd/conf.d directory with the following contents.
Note: this example uses ldap and the string will vary for your instance.
ScriptAlias /nagios_prod/cgi-bin/ "/usr/local/nagios/instances/nagios_prod/sbin/" SetEnv NAGIOS_CGI_CONFIG /usr/local/nagios/instances/nagios_prod/etc/cgi.cfg Options ExecCGI Options FollowSymLinks AllowOverride None Order allow,deny Allow from all AuthType Basic AuthName "Nagios Area51" AuthBasicProvider ldap AuthzLDAPAuthoritative off AuthLDAPURL ldaps://myldaprod:636/ou=IMEROOT,o=corp?uid Alias /nagios_prod/ "/usr/local/nagios/instances/nagios_prod/share/" Options None AllowOverride None Order allow,deny Allow from all AuthType Basic AuthName "Nagios Area51" AuthBasicProvider ldap AuthzLDAPAuthoritative off AuthLDAPURL ldaps://myldaprod:636/ou=IMEROOT,o=corp?uid
Edit config.inc.php
The config.inc.php file is located in the /usr/local/nagios/instances/nagios_prod/share/ directory. Make the path changes for the instance.
Example:
Correct Directory and File PermissionsAt this point it is a good time to correct the file and directory permissions to the correct owner. Nagios is very particular on the correct settings. This is a painful process and is a good candidate to dump in a script and ran.
# chown –R nagios.nagios /usr/local/nagios/ # chown –R root.root /usr/local/nagios/instances/nagios_prod/share/locale # chmod 775 /usr/local/nagios/instances # chmod 775 /usr/local/nagios/instances/nagios_prod # chmod 775 /usr/local/nagios/instances/ nagios_prod /etc # chmod 775 /usr/local/nagios/instances/ nagios_prod /etc/monitors # chmod 775 /usr/local/nagios/instances/ nagios_prod/sbin # chmod 775 /usr/local/nagios/instances/ nagios_prod/share # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/docs # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/contexthelp # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/images # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/images/logos # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/includes # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/media # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/ssi # chmod 775 /usr/local/nagios/instances/ nagios_prod/share/stylesheets # chmod 775 /usr/local/nagios/instances/ nagios_prod/var # chmod 775 /usr/local/nagios/instances/ nagios_prod/var/rw # chmod 775 /usr/local/nagios/instances/ nagios_prod/var/spool # chmod 775 /usr/local/nagios/instances/ nagios_prod/var/spool/checkresults
Now the files
# chmod 664 /usr/local/nagios/instances/nagios_prod/* # chmod 664 /usr/local/nagios/instances/ nagios_prod /etc/* # chmod 664 /usr/local/nagios/instances/ nagios_prod /etc/monitors/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/sbin/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/docs/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/contexthelp/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/images/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/images/logos/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/includes/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/media/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/ssi/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/share/stylesheets/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/var/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/var/rw/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/var/spool/* # chmod 664 /usr/local/nagios/instances/ nagios_prod/var/spool/checkresults/*
Create Monitors
At this point you can start the creation of the monitors, hosts, hostgroups, contacts. Contactgroups and all the other dependent settings. Refer to Nagios site for syntax on each of the different objects. Foe the monitors you can use the default monitors created at install in /usr/local/nagios/etc/objects and copy the configs for localhost.cfg and windows.cfg as a starting basis.
Once you have your settings completed you need to run a check on the configs running the command
# /usr/local/nagios/instances/nagios_prod/bin/nagios -v /usr/local/nagios/instances/nagios_prod/etc/nagios.cfg
Fix any errors at this point and recheck.
Startup Scripts (init.d)
Add Nagios to the list of system services and have it automatically start when the system boots.
Create init.d scripts for each instance you want to create. Copy the Nagios installed script, and name it nagios_prod. Open in your favorite editor and find the line prefix= and change the path.
Example:
prefix=/usr/local/nagios/instances/nagios_prod
Save the changes and change the permissions to 755 on the script.
Use chkconfig to add it to the startup and enable it.
# chkconfig --add nagios_prod # chkconfig nagios_prod on
At this point you can already start your instances:
# service nagios_prod start
Now restart your Apache and open your browser
# service httpd restart
NagiosGraph (Performance Monitoring)
This section explains how to configure Nagiosgraph for multiple instances to gathering and graphing performance data created by Nagios. Each of the steps will have to be done for each individual instance you want to collect and graph performance data with the exception of rrdtool and intltool installs, those are only required installed once.
Requirements
intltool.x86_64 to 0.35 or higher
rrdtool
nagiosgraph
Update intltool.x86_64 to 0.35 or higher
# yum -y install intltool.x86_64
Install rrdtool
# wget 'http://dag.wieers.com/rpm/packages/rrdtool/rrdtool-1.2.23-1.el5.rf.x86_64.rpm' # wget 'http://dag.wieers.com/rpm/packages/rrdtool/perl-rrdtool-1.2.23-1.el5.rf.x86_64.rpm' # wget 'http://dag.wieers.com/rpm/packages/rrdtool/rrdtool-devel-1.2.23-1.el5.rf.x86_64.rpm' # wget 'http://dag.wieers.com/rpm/packages/rrdtool/ruby-rrdtool-1.2.23-1.el5.rf.x86_64.rpm' # wget 'http://dag.wieers.com/rpm/packages/rrdtool/python-rrdtool-1.2.23-1.el5.rf.x86_64.rpm' # wget 'http://dag.wieers.com/rpm/packages/rrdtool/php-rrdtool-1.0.50-3.el5.rf.x86_64.rpm' # wget 'http://dag.wieers.com/rpm/packages/rrdtool/rrdtool-1.2.23-1.el5.rf.x86_64.rpm' # wget 'http://mesh.dl.sourceforge.net/sourceforge/nagiosgraph/nagiosgraph-1.3.1.tar.gz' # yum install -y libart_lgpl # rpm -hiv *rrdtool*.rpm
Install NagiosGraph
Download NagiosGraph 1.3.1 from http://nagiosgraph.sourceforge.net/
# cd /usr/local/src # tar xvf nagiosgraph-1.3.1.tar.gz # cd nagiosgraph-1.3.1
Create the directory structure and copy files
# mkdir /usr/local/nagios/instances/nagios_prod/nagiosgraph # mkdir /usr/local/nagios/instances/nagios_prod/nagiosgraph/rrd # cp -r . /usr/local/nagios/instances/nagios_prod/nagiosgraph/ # chmod go+rX /usr/local/nagios/instances/nagios_prod/nagiosgraph # chown nagios.apache /usr/local/nagios/instances/nagios_prod/nagiosgraph/rrd # cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/lib # cp insert.* /usr/local/nagios/instances/nagios_prod/sbin/ # touch /usr/local/nagios/instances/nagios_prod/var/nagiosgraph.log /usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log # chown nagios.apache /usr/local/nagios/instances/nagios_prod/var/nagiosgraph.log /usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log # chmod 664 /usr/local/nagios/nagios_prod/var/nagiosgraph.log # chmod 644 /usr/local/nagios/nagios_prod/var/spool/perfdata.log # cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/share # cp nagiosgraph.js /usr/local/nagios/instances/nagios_prod/share/ # cp nagiosgraph.css /usr/local/nagios/instances/nagios_prod/share/stylesheets/ # chown nagios.nagios /usr/local/nagios/instances/nagios_prod/share/nagiosgraph.js # chown nagios.nagios /usr/local/nagios/instances/nagios_prod/share/stylesheets/nagiosgraph.css # cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/cgi # cp * /usr/local/nagios/instances/nagios_prod/sbin/ # cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/etc/ # mv ngshared.pm /usr/local/nagios/instances/nagios_prod/sbin/ # mv nagiosgraph.conf /usr/local/nagios/instances/nagios_prod/sbin/ # mv servdb.conf /usr/local/nagios/instances/nagios_prod/etc/ # mv hostdb.conf /usr/local/nagios/instances/nagios_prod/etc/ # mv map /usr/local/nagios/instances/nagios_prod/etc/ # chown nagios.nagios /usr/local/nagios/instances/nagios_prod/sbin/* # cp /usr/local/nagios/instances/nagios_prod/nagiosgraph/etc/* /usr/local/nagios/instances/nagios_prod/etc/ # mv /usr/local/nagios/instances/nagios_prod/etc/nagiosgraph.conf /usr/local/nagios/instances/nagios_prod/sbin/ # mv /usr/local/nagios/instances/nagios_prod/etc/ngshared.pm /usr/local/nagios/instances/nagios_prod/sbin/
Configuration file edits
Insert.pl & insert.sh file
Edit insert.pl & insert.sh and modify the lib path.
# The configuration file and ngshared.pm must be in this directory.
use lib '/usr/local/nagios/instances/nagios_prod/sbin';
Various CGI files
Edit the perl scripts in the cgi and lib directories, modifying the “use lib” for the following files in the /usr/local/nagios/instances/nagios_prod/sbin directory.
showhost.cgi
show.cgi
showhost.cgi
showservice.cgi
testcolor.cgi
Change the following line.
use lib '/usr/local/nagios/instances/nagios_prod/sbin';
Nagiosgraph.conf file
Edit nagiosgraph.conf change the following lines.
log_file=/usr/local/nagios/instances/nagios_prod/var/nagiosgraph.log rrddir = /usr/local/nagios/instances/nagios_prod/nagiosgraph/rrd mapfile = /usr/local/nagios/instances/nagios_prod/etc/map # Location of performance data log file. Comment out if not used. perflog = /usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log # Location of hosts/database control file hostdb = /usr/local/nagios/instances/nagios_prod/etc/hostdb.conf # Location of services/database control file servdb = /usr/local/nagios/instances/nagios_prod/etc/servdb.conf # Nagios CGI URL nagioscgiurl = https://cvglp9999/nagios_prod/cgi-bin # JavaScript configuration # The URL for the nagiosgraph.js file javascript = /nagios_prod/nagiosgraph.js # Stylesheet - added to head of show.cgi, etc. Comment out if not used stylesheet = /nagios_prod/stylesheets/nagiosgraph.css
nagios.cfg file
Edit nagios.cfg and change /add.
Add the line
# NagiosGraph cfg_file=/usr/local/nagios/instances/nagios_prod/etc/serviceextinfo.cfg
Modify the following lines.
process_performance_data=1 service_perfdata_file_processing_command=process-service-perfdata-prod service_perfdata_file=/usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$ service_perfdata_file_mode=a service_perfdata_file_processing_interval=30
checkcommands.cfg file
Add to your checkcommands.cfg file in the shared Nagios libexec directory
define command {
command_name process-service-perfdata-prod
command_line /usr/bin/perl /usr/local/nagios/instances/nagios_prod/sbin/insert.pl
}
serviceextinfo.cfg file
Create serviceextinfo.cfg in /usr/local/nagios/instances/nagios_prod/etc/ and add the entries in for the performance data to graph. Refer to Nagios documentation on the syntax for the serviceextinfo.cfg file. This file will tell which service monitors you would like to collect performance data on.
Example entry:
define serviceextinfo {
service_description Memory Usage
hostgroup_name windows-prod-servers
notes_url /nagios_prod/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&db=ntmem,memused,memwarn,memcrit,memmmax
icon_image extinfo.gif
}
define serviceextinfo {
service_description CPU_Load
hostgroup_name windows-prod-servers
notes_url /nagios_prod/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&db=ntload,avg05min,avg30min
icon_image extinfo.gif
}
CGI.CFG File
Edit cgi.cfg and add line under main_config_file entry.
xedtemplate_config_file=/usr/local/nagios/instances/nagios_prod/etc/serviceextinfo.cfg
Map file
Edit the map file in usr/bin/perl /usr/local/nagios/instances/nagios_prod/etc directory. The default Nagiosgraph supplied one doesn’t have any Windows service map files. A good source for examples can be found at http://nerhood.wordpress.com/2004/09/22/nagiosgraph-with-windows-support/ and has the basic map files needed for a good start.
Example:
# Service type: ntload
# check command: check_nt -H Address -v CPULOAD -l5,80,90,30,70,80
# output: CPU Load 9% (5 min average) 11% (10 min average) 3% (30 min average)
# perfdata: .5 min avg Load.=9%;80;90;0;100 .15 min avg Load.=11%;70;80;0;100 .30 min average Load.=3%;80;90;0;100
/output:.*?Load (\d+)% .*?(\d+)% /
and push @s, [ ntload,
[ avg05min, GAUGE, $1 ],
[ avg30min, GAUGE, $2 ] ];
# Service type: ntmem
# check command: check_nt -H Address -v MEMUSE -w 50 -c 90
# output: Memory usage: total:2467.75 Mb - used: 510.38 Mb (21%) - free: 1957.37 Mb (79%)
# perfdata: Memory usage=510.38Mb;1233.88;2220.98;0.00;2467.75
/perfdata:.*usage=([.0-9]+)Mb;([.0-9]+);([.0-9]+);([.0-9]+);([.0-9]+)/
and push @s, [ ntmem,
[ memused, GAUGE, $1*1024**2 ],
[ memwarn, GAUGE, $2*1024**2 ],
[ memcrit, GAUGE, $3*1024**2 ],
[ memmmax, GAUGE, $5*1024**2 ] ];
Complete Install
Now that NagiosGraph is installed all that remains is to restart your Nagios instance and Apache and check the results.
# service nagios_prod restart # service httpd restart
I know this is a long and detailed article, but this can be done and has great benefit. If you would like a downloadable pdf of this post it is available Multiple_Nagios_instances_single_server.
Following this to the T gets a lot of errors.
Had to make my own mods and still cant get it to work.
Any updates on this site?