check_by_ssh_wrapper

Landry MINOZA — Sun, 27 Oct 2013 17:00:00 GMT

Trying to mutualise Nagios services (or Shinken in my case) and NRPE configuration, sometimes a check returns to many data for NRPE or a host in DMZ don't have NRPE port accessible.

On one infrastructure I monitor, while my NRPE configuration can check free disk space on local file systems (among other), I have a NAS in one DMZ with a lot of file systems (one LVM volume by share). The size of returned data, with Nagios perf data was above NRPE limit (1 kB). Rather than moving NRPE command line in Shinken services configuration and having to handle character escaping, rights and complexify Shinken configuration, I wrote this little script which use via check_by_ssh the same parameters than check_nrpe and read NRPE configuration one the client host to validate the command and execute it.

The code

It is a simple bash script, hosted on GitHub. At time of writing this post, the code is:

#!/bin/bash
#
# check_by_ssh_wrapper
#
# This script provides access to commands declared in nrpe to check_by_ssh
# Please set the nrpeconf variable to the right value
# You must generate a ssh key on the nagios server, copy it in .ssh/authorized_keys
# in nagios (or nrpe) user home directory on distant server, then add this at
# the begining of the line juste before the key:
# no-pty,no-port-forwarding,command="/usr/local/bin/check_by_ssh_wrapper $SSH_ORIGINAL_COMMAND"
#
# You can the use $USER1$/check_by_ssh -H <distant host> -l nagios -C 'check_cmd [args]'
#
# Freely inspired by nagios_ssh_framework. Can be used when nrpe port can't be accessed
# or check_command returns to many text for nrpe and you want to keep facilities of nrpe
# config files.
#
# Changes an Modifications:
# 2011-10-12 - Landry MINOZA - V 1.0
# Created because check_disks returns to many lines for nrpe with one of my servers
#
# Copyright: Telbase Formation Conseil, MUSEE D'ORSAY 2011
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

# Set main nrpe.cfg file
nrpeconf=/etc/nagios/nrpe.cfg

# Get cmdname
cmdname=$1
if [ -z "$cmdname" ] ; then
        echo "WARNING, no command name provided"
        exit 1
fi
shift
# Get optionnal args
cmdargs=$@

# Declare an array of commands
declare -A command

# find all nrpe conf files
include=$(sed -ne '/^include=/ s/^.*=// p' $nrpeconf)
include_dir=$(sed -ne '/^include_dir=/ s/^.*=// p' $nrpeconf)

# Parse all nrpe conf files for command[*]= arguments
for cfgfile in $(echo -e "$nrpeconf\n${include}\n${include_dir}*.cfg") ;do
        eval $(sed -ne '/^command\[/ s/=\(.*\)$/="\1"/ p' $cfgfile)
done

# Check if command[$cmdname] exists
if [ -z "${command[$cmdname]}" ] ; then
        echo "WARNING, command $cmdname does not exist"
        exit 1
fi

# Launch check
exec ${command[$cmdname]} $cmdargs
exit

The code can be divided in 8 parts:

Line 38, to customize, this is the path of NRPE configuration file (for Debian stable here).
Line 40 to 48, command line parameters validation.
Line 51, explicit (and mandatory) declaration of an associative array of commands, it is used to store for each command named in NRPE configuration the related command line.
Line 54 and 55, retrieval of include and include_dir statement in nrpe.conf file to take into account the referenced files. Take care, this include files research is not yet recusive, the statements are only analyzed in the main configuration file.
Line 58 and 60, loop on list on found files.
Line 59, this line is the core of the script, this is a trick base on NRPE configuration syntax, indeed, to declare a command in NRPE, one writes: command[command_name]=command_line options. Lines simply have to be extracted and the command line part (after “=”) have to be quoted, then this have to be executed via eval to fill the array command declared before.
Line 63, testing if a command whith the name given as parameter to the script exists.
At last, line 69, execution of the specified command with his parameters. The explicit exit at the end, ensure that the exit state of the script is the same that the one of the executed command, and so transmited to Nagios.

Installation

This script have to be installed on the monitored host, not on the monitoring server, for instance in /usr/local/bin. The user executing NRPe daemon (on Debian, it is named nagios) must have execution rights on the script. By default, this user have no password set, and we do not want to set one. Anyway, this is generaly a bad idea to have passwords in Nagios configuration files. First, generate a SSH key pair on the monitoring server using the command ssh_keygen as the monitoring daemon user. By default, this commande generates a RSA key, stores the private key in $HOME/.ssh/id_rsa and the public one in $HOME/.ssh/id_rsa.pub. This last is the one we want.

On the monitored host, create in the $HOME of the NRPE daemon user (/var/lib/nagios on Debian) a directory named .ssh whith unix rights rwx------

mkdir /var/lib/nagios/.ssh
chown nagios:nagios /var/lib/nagios/.ssh
chmod u=rwx,go= /var/lib/nagios/.ssh

In the folder, create a file named authorized_keys containing the following line:

no-pty,no-port-forwarding,command="/usr/local/bin/check_by_ssh_wrapper $SSH_ORIGINAL_COMMAND" ssh-rsa AAAA… user@server

Replace AAAA… user@server with the content of the file id_rsa.pub generated on the server. This line [1] permits to allow only the command chaeck_by_ssh_wrapper (only place here an absolute path) to be executed when connectiong with this SSH key and forbid ports forwarding and pseudo-tty allocation. The original command passed to ssh will be provided as parameter to our script via the variable $SSH_ORIGINAL_COMMAND.

Using

In Nagios configuration, just create a new command object:

define command{
        command_name check_by_ssh
        command_line $USER1$/check_by_ssh -H $HOSTADDRESS$ -t 60 -l nagios -C '$ARG1$ $ARG2$'
}

This can be used the same way than check_nrpe in services definitions:

define service{
        use                     base-service
        service_description     check_disks
        display_name            Check Disks
        check_command           check_by_ssh!check_disks
}

Conclusion

This is not perfect, it does not handle recusrive inclusions, but it work as I want since 2 years and permited me to simplify a lot the use of check_by_ssh plugin, securing it as only commands declared in NRPE configuration are usable.

[1]	I'm preparing a more complete post on ssh (and PuTTY).

Hobgoblin's Master (monitoring)

check_by_ssh_wrapper

The code

Installation

Using

Conclusion