search
top

Multiple Nagios Instances on a Single Server

In this post I will cover how to run Multiple Nagios Instances on a Single Server. This will take a single installation of Nagios and allow you to run separate instances as you would to separate out different environments, such as production, development, certification, etc….

It allows for flexibility of configurations and allows Nagios to run faster for checks and execution. This installation scenario is based on RedHat 5.x installation with Nagios 3.2.0, Nagios Plugins 1.4.13, Nagiosgraph, rrdtool and LDAP integration.

Installation

Download and Install Boutell’s GD library

wget http://www.libgd.org/releases/gd-2.0.35.tar.gz
./configure
make install

Compile and install the Nagios and Plugins

wget http://prdownloads.sourceforge.net/sourceforge/nagios/nagios-3.2.0.tar.gz
wget http://prdownloads.sourceforge.net/sourceforge/nagiosplug/nagios-plugins-1.4.13.tar.gz
tar -xzf nagios-3.2.0.tar.gz
tar -xzf nagios-plugins-1.4.13.tar.gz
export NAGIOSHOME=/usr/local/nagios
cd nagios-3.2.0
./configure --enable-embedded-perl --with-command-group=nagios --with-gd-lib=/usr/local/lib --with-gd-inc=/usr/local/include; make  all; make install install-init;make-install-config;make install-commandmode

Compile and install the Nagios plugins

cd nagios-plugins-1.4.13
./configure --prefix=$NAGIOSHOME/core/nagios-plugins --with-nagios-user=nagios --with-nagios-group=nagios; make ; make install

File Copy and Directory Creation
Create directories which hold your instances and their subdirectories

NAGIOSHOME=/usr/local/nagios

mkdir $NAGIOSHOME/instances
mkdir $NAGIOSHOME/instances/nagios_prod
mkdir $NAGIOSHOME/instances/nagios_prod/etc
mkdir $NAGIOSHOME/instances/nagios_prod/etc/monitors
mkdir $NAGIOSHOME/instances/nagios_prod/var
mkdir $NAGIOSHOME/instances/nagios_prod/var/rw
mkdir $NAGIOSHOME/instances/nagios_prod/var/spool
mkdir $NAGIOSHOME/instances/nagios_prod/var/spool/checkresults
mkdir $NAGIOSHOME/instances/nagios_prod/share
mkdir $NAGIOSHOME/instances/nagios_prod/sbin

Create Symbolic Links

ln -s $NAGIOSHOME/nagios/bin $NAGIOSHOME/instances/nagios_prod/bin
ln -s $NAGIOSHOME/nagios-plugins/libexec $NAGIOSHOME/instances/nagios_prod/libexec

Copy files

cp -r $NAGIOSHOME/sbin/*  $NAGIOSHOME/instances/nagios_prod/sbin/
cp –r $NAGIOSHOME/nagios/share/* $NAGIOSHOME/instances/nagios_prod/share/
cp $NAGIOSHOME/etc/cgi.cfg  $NAGIOSHOME/instances/nagios_prod/etc/
cp $NAGIOSHOME/etc/resource.cfg  $NAGIOSHOME/instances/nagios_prod/etc/
cp $NAGIOSHOME/etc/nagios.cfg  $NAGIOSHOME/instances/nagios_prod/etc/

Do the same with all the other instances you are going to deploy.

Configuration
In the editing of the Nagios config files there will be instances that point to a central shared location and to individual settings. The configurations that are shared are checkcommands.cfg, misccommands.cfg, contacts.cfg, contactgroups.cfg, timeperiods.cfg, templates.cfg, dependencies.cfg & escalations.cfg.
The individual ones are the specific monitors and hosts, which in the nagios.cfg we will state a directory instead of the individual configuration files.

nagios.cfg modifications

Example:

log_file=/usr/local/nagios/instances/nagios_prod/var/nagios.log

# OBJECT CONFIGURATION FILE(S)
cfg_file=/usr/local/nagios/etc/objects/checkcommands.cfg
cfg_file=/usr/local/nagios/etc/objects/misccommands.cfg
cfg_file=/usr/local/nagios/etc/objects/contacts.cfg
cfg_file=/usr/local/nagios/etc/objects/contactgroups.cfg
cfg_file=/usr/local/nagios/etc/objects/timeperiods.cfg
cfg_file=/usr/local/nagios/etc/objects/templates.cfg
cfg_file=/usr/local/nagios/etc/objects/dependencies.cfg
cfg_file=/usr/local/nagios/etc/objects/escalations.cfg

# Definitions for monitoring
cfg_dir=/usr/local/nagios/instances/nagios_prod/etc/monitors

object_cache_file=/usr/local/nagios/instances/nagios_prod/var/objects.cache
precached_object_file=/usr/local/nagios/instances/nagios_prod/var/objects.precache
resource_file=/usr/local/nagios/instances/nagios_prod/etc/resource.cfg
status_file=/usr/local/nagios/instances/nagios_prod/var/status.dat
command_file=/usr/local/nagios/instances/nagios_prod/var/rw/nagios.cmd
lock_file=/usr/local/nagios/instances/nagios_prod/var/nagios.lock
temp_file=/usr/local/nagios/instances/nagios_prod/var/nagios.tmp
log_archive_path=/usr/local/nagios/instances/nagios_prod/var/archives
state_retention_file=/usr/local/nagios/instances/nagios_prod/var/retention.dat
debug_file=/usr/local/nagios/instances/nagios_prod/var/nagios.debug

CGI.CFG File Edits

The cgi.cfg file must be edit for Apache to be able to run the correct CGI scripts for each instance you plan to run. To follow the examples in previous sections we use nagios_prod as our instance.

main_config_file=/usr/local/nagios/instances/nagios_prod/etc/nagios.cfg
physical_html_path=/usr/local/nagios/instances/nagios_prod/share
url_html_path=/nagios_prod/

Resource.cfg file edits

$USER1$=/etc/nagios/instances/nagios_prod/libexec

Create nagios configuration for Apache
Create a nagios_prod.conf file in the /etc/httpd/conf.d directory with the following contents.
Note: this example uses ldap and the string will vary for your instance.

ScriptAlias /nagios_prod/cgi-bin/ "/usr/local/nagios/instances/nagios_prod/sbin/"

   SetEnv NAGIOS_CGI_CONFIG /usr/local/nagios/instances/nagios_prod/etc/cgi.cfg
   Options ExecCGI
   Options FollowSymLinks
   AllowOverride None
   Order allow,deny
   Allow from all
   AuthType Basic
   AuthName "Nagios Area51"
   AuthBasicProvider ldap
   AuthzLDAPAuthoritative off
   AuthLDAPURL ldaps://myldaprod:636/ou=IMEROOT,o=corp?uid

Alias /nagios_prod/ "/usr/local/nagios/instances/nagios_prod/share/"

   Options None
   AllowOverride None
   Order allow,deny
   Allow from all
   AuthType Basic
   AuthName "Nagios Area51"
   AuthBasicProvider ldap
   AuthzLDAPAuthoritative off
   AuthLDAPURL ldaps://myldaprod:636/ou=IMEROOT,o=corp?uid

Edit config.inc.php

The config.inc.php file is located in the /usr/local/nagios/instances/nagios_prod/share/ directory. Make the path changes for the instance.

Example:


Correct Directory and File PermissionsAt this point it is a good time to correct the file and directory permissions to the correct owner. Nagios is very particular on the correct settings. This is a painful process and is a good candidate to dump in a script and ran.

# chown –R nagios.nagios /usr/local/nagios/
# chown –R root.root /usr/local/nagios/instances/nagios_prod/share/locale
# chmod 775 /usr/local/nagios/instances
# chmod 775 /usr/local/nagios/instances/nagios_prod
# chmod 775 /usr/local/nagios/instances/ nagios_prod /etc
# chmod 775 /usr/local/nagios/instances/ nagios_prod /etc/monitors
# chmod 775 /usr/local/nagios/instances/ nagios_prod/sbin
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/docs
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/contexthelp
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/images
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/images/logos
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/includes
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/media
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/ssi
# chmod 775 /usr/local/nagios/instances/ nagios_prod/share/stylesheets
# chmod 775 /usr/local/nagios/instances/ nagios_prod/var
# chmod 775 /usr/local/nagios/instances/ nagios_prod/var/rw
# chmod 775 /usr/local/nagios/instances/ nagios_prod/var/spool
# chmod 775 /usr/local/nagios/instances/ nagios_prod/var/spool/checkresults

Now the files

# chmod 664 /usr/local/nagios/instances/nagios_prod/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod /etc/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod /etc/monitors/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/sbin/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/docs/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/contexthelp/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/images/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/images/logos/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/includes/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/media/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/ssi/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/share/stylesheets/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/var/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/var/rw/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/var/spool/*
# chmod 664 /usr/local/nagios/instances/ nagios_prod/var/spool/checkresults/*

Create Monitors
At this point you can start the creation of the monitors, hosts, hostgroups, contacts. Contactgroups and all the other dependent settings. Refer to Nagios site for syntax on each of the different objects. Foe the monitors you can use the default monitors created at install in /usr/local/nagios/etc/objects and copy the configs for localhost.cfg and windows.cfg as a starting basis.
Once you have your settings completed you need to run a check on the configs running the command

# /usr/local/nagios/instances/nagios_prod/bin/nagios -v /usr/local/nagios/instances/nagios_prod/etc/nagios.cfg

Fix any errors at this point and recheck.

Startup Scripts (init.d)
Add Nagios to the list of system services and have it automatically start when the system boots.
Create init.d scripts for each instance you want to create. Copy the Nagios installed script, and name it nagios_prod. Open in your favorite editor and find the line prefix= and change the path.

Example:

prefix=/usr/local/nagios/instances/nagios_prod

Save the changes and change the permissions to 755 on the script.

Use chkconfig to add it to the startup and enable it.

# chkconfig --add nagios_prod
# chkconfig nagios_prod on

At this point you can already start your instances:

# service nagios_prod start

Now restart your Apache and open your browser

# service httpd restart

NagiosGraph (Performance Monitoring)
This section explains how to configure Nagiosgraph for multiple instances to gathering and graphing performance data created by Nagios. Each of the steps will have to be done for each individual instance you want to collect and graph performance data with the exception of rrdtool and intltool installs, those are only required installed once.

Requirements
intltool.x86_64 to 0.35 or higher
rrdtool
nagiosgraph

Update intltool.x86_64 to 0.35 or higher

# yum -y install  intltool.x86_64

Install rrdtool

# wget 'http://dag.wieers.com/rpm/packages/rrdtool/rrdtool-1.2.23-1.el5.rf.x86_64.rpm'
# wget 'http://dag.wieers.com/rpm/packages/rrdtool/perl-rrdtool-1.2.23-1.el5.rf.x86_64.rpm'
# wget 'http://dag.wieers.com/rpm/packages/rrdtool/rrdtool-devel-1.2.23-1.el5.rf.x86_64.rpm'
# wget 'http://dag.wieers.com/rpm/packages/rrdtool/ruby-rrdtool-1.2.23-1.el5.rf.x86_64.rpm'
# wget 'http://dag.wieers.com/rpm/packages/rrdtool/python-rrdtool-1.2.23-1.el5.rf.x86_64.rpm'
# wget 'http://dag.wieers.com/rpm/packages/rrdtool/php-rrdtool-1.0.50-3.el5.rf.x86_64.rpm'
# wget 'http://dag.wieers.com/rpm/packages/rrdtool/rrdtool-1.2.23-1.el5.rf.x86_64.rpm'
# wget 'http://mesh.dl.sourceforge.net/sourceforge/nagiosgraph/nagiosgraph-1.3.1.tar.gz'
# yum install -y libart_lgpl
# rpm -hiv *rrdtool*.rpm

Install NagiosGraph
Download NagiosGraph 1.3.1 from http://nagiosgraph.sourceforge.net/

# cd /usr/local/src
# tar xvf nagiosgraph-1.3.1.tar.gz
# cd nagiosgraph-1.3.1

Create the directory structure and copy files

# mkdir /usr/local/nagios/instances/nagios_prod/nagiosgraph
# mkdir /usr/local/nagios/instances/nagios_prod/nagiosgraph/rrd
# cp -r . /usr/local/nagios/instances/nagios_prod/nagiosgraph/
# chmod go+rX /usr/local/nagios/instances/nagios_prod/nagiosgraph
# chown nagios.apache /usr/local/nagios/instances/nagios_prod/nagiosgraph/rrd

# cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/lib
# cp insert.* /usr/local/nagios/instances/nagios_prod/sbin/
# touch /usr/local/nagios/instances/nagios_prod/var/nagiosgraph.log /usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log
# chown nagios.apache /usr/local/nagios/instances/nagios_prod/var/nagiosgraph.log /usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log
# chmod 664 /usr/local/nagios/nagios_prod/var/nagiosgraph.log
# chmod 644 /usr/local/nagios/nagios_prod/var/spool/perfdata.log

# cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/share
# cp nagiosgraph.js /usr/local/nagios/instances/nagios_prod/share/
# cp nagiosgraph.css /usr/local/nagios/instances/nagios_prod/share/stylesheets/
# chown nagios.nagios /usr/local/nagios/instances/nagios_prod/share/nagiosgraph.js
# chown nagios.nagios /usr/local/nagios/instances/nagios_prod/share/stylesheets/nagiosgraph.css

# cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/cgi
# cp * /usr/local/nagios/instances/nagios_prod/sbin/
# cd /usr/local/nagios/instances/nagios_prod/nagiosgraph/etc/
# mv ngshared.pm  /usr/local/nagios/instances/nagios_prod/sbin/
# mv nagiosgraph.conf  /usr/local/nagios/instances/nagios_prod/sbin/
# mv servdb.conf /usr/local/nagios/instances/nagios_prod/etc/
# mv hostdb.conf /usr/local/nagios/instances/nagios_prod/etc/
# mv map  /usr/local/nagios/instances/nagios_prod/etc/
# chown nagios.nagios  /usr/local/nagios/instances/nagios_prod/sbin/*

# cp /usr/local/nagios/instances/nagios_prod/nagiosgraph/etc/*  /usr/local/nagios/instances/nagios_prod/etc/
# mv /usr/local/nagios/instances/nagios_prod/etc/nagiosgraph.conf  /usr/local/nagios/instances/nagios_prod/sbin/
# mv /usr/local/nagios/instances/nagios_prod/etc/ngshared.pm  /usr/local/nagios/instances/nagios_prod/sbin/

Configuration file edits
Insert.pl & insert.sh file
Edit insert.pl & insert.sh and modify the lib path.

# The configuration file and ngshared.pm must be in this directory.

use lib '/usr/local/nagios/instances/nagios_prod/sbin';

Various CGI files
Edit the perl scripts in the cgi and lib directories, modifying the “use lib” for the following files in the /usr/local/nagios/instances/nagios_prod/sbin directory.

showhost.cgi
show.cgi
showhost.cgi
showservice.cgi
testcolor.cgi

Change the following line.

use lib '/usr/local/nagios/instances/nagios_prod/sbin';

Nagiosgraph.conf file
Edit nagiosgraph.conf change the following lines.

log_file=/usr/local/nagios/instances/nagios_prod/var/nagiosgraph.log
rrddir = /usr/local/nagios/instances/nagios_prod/nagiosgraph/rrd
mapfile = /usr/local/nagios/instances/nagios_prod/etc/map

# Location of performance data log file. Comment out if not used.
perflog = /usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log

# Location of hosts/database control file
hostdb = /usr/local/nagios/instances/nagios_prod/etc/hostdb.conf

# Location of services/database control file
servdb = /usr/local/nagios/instances/nagios_prod/etc/servdb.conf

# Nagios CGI URL
nagioscgiurl = https://cvglp9999/nagios_prod/cgi-bin

# JavaScript configuration
# The URL for the nagiosgraph.js file
javascript = /nagios_prod/nagiosgraph.js

# Stylesheet - added to head of show.cgi, etc. Comment out if not used
stylesheet = /nagios_prod/stylesheets/nagiosgraph.css

nagios.cfg file
Edit nagios.cfg and change /add.
Add the line

# NagiosGraph
cfg_file=/usr/local/nagios/instances/nagios_prod/etc/serviceextinfo.cfg

Modify the following lines.

process_performance_data=1
service_perfdata_file_processing_command=process-service-perfdata-prod
service_perfdata_file=/usr/local/nagios/instances/nagios_prod/var/spool/perfdata.log
service_perfdata_file_template=$LASTSERVICECHECK$||$HOSTNAME$||$SERVICEDESC$||$SERVICEOUTPUT$||$SERVICEPERFDATA$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=30

checkcommands.cfg file
Add to your checkcommands.cfg file in the shared Nagios libexec directory

define command {
       command_name  process-service-perfdata-prod
       command_line  /usr/bin/perl /usr/local/nagios/instances/nagios_prod/sbin/insert.pl
     }

serviceextinfo.cfg file
Create serviceextinfo.cfg in /usr/local/nagios/instances/nagios_prod/etc/ and add the entries in for the performance data to graph. Refer to Nagios documentation on the syntax for the serviceextinfo.cfg file. This file will tell which service monitors you would like to collect performance data on.
Example entry:

define serviceextinfo {
  service_description  Memory Usage
  hostgroup_name       windows-prod-servers
  notes_url       /nagios_prod/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&db=ntmem,memused,memwarn,memcrit,memmmax
  icon_image     extinfo.gif
}

define serviceextinfo {
  service_description  CPU_Load
  hostgroup_name    windows-prod-servers
  notes_url       /nagios_prod/cgi-bin/show.cgi?host=$HOSTNAME$&service=$SERVICEDESC$&db=ntload,avg05min,avg30min
  icon_image     extinfo.gif
}

CGI.CFG File
Edit cgi.cfg and add line under main_config_file entry.

xedtemplate_config_file=/usr/local/nagios/instances/nagios_prod/etc/serviceextinfo.cfg

Map file
Edit the map file in usr/bin/perl /usr/local/nagios/instances/nagios_prod/etc directory. The default Nagiosgraph supplied one doesn’t have any Windows service map files. A good source for examples can be found at http://nerhood.wordpress.com/2004/09/22/nagiosgraph-with-windows-support/ and has the basic map files needed for a good start.

Example:

# Service type: ntload
# check command: check_nt -H Address -v CPULOAD -l5,80,90,30,70,80
# output: CPU Load 9% (5 min average) 11% (10 min average) 3% (30 min average)
# perfdata: .5 min avg Load.=9%;80;90;0;100 .15 min avg Load.=11%;70;80;0;100 .30 min average Load.=3%;80;90;0;100
/output:.*?Load (\d+)% .*?(\d+)% /
and push @s, [ ntload,
[ avg05min, GAUGE, $1 ],
[ avg30min, GAUGE, $2 ] ];

# Service type: ntmem
#   check command: check_nt -H Address -v MEMUSE -w 50 -c 90
#   output: Memory usage: total:2467.75 Mb - used: 510.38 Mb (21%) - free: 1957.37 Mb (79%)
#   perfdata: Memory usage=510.38Mb;1233.88;2220.98;0.00;2467.75
/perfdata:.*usage=([.0-9]+)Mb;([.0-9]+);([.0-9]+);([.0-9]+);([.0-9]+)/
and push @s, [ ntmem,
       [ memused, GAUGE, $1*1024**2 ],
       [ memwarn, GAUGE, $2*1024**2 ],
       [ memcrit, GAUGE, $3*1024**2 ],
       [ memmmax, GAUGE, $5*1024**2 ] ];

Complete Install
Now that NagiosGraph is installed all that remains is to restart your Nagios instance and Apache and check the results.

# service nagios_prod restart
# service httpd restart

I know this is a long and detailed article, but this can be done and has great benefit. If you would like a downloadable pdf of this post it is available Multiple_Nagios_instances_single_server.

One Response to “Multiple Nagios Instances on a Single Server”

  1. Eherr says:

    Following this to the T gets a lot of errors.

    Had to make my own mods and still cant get it to work.

    Any updates on this site?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

top