An It-slave in the digital saltmine

This guide describe howto install and do a basic configure of SNMP on a RedHat Enterprise Linux or CentOS. Probably it will work on many other *nix systems.

1. Installation

Run command yum install net-snmp-utils

[root@dull etc]# yum install net-snmp-utils
Loading "fastestmirror" plugin
Loading "dellsysidplugin" plugin
...
...
Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 net-snmp-utils          i386       1:5.3.1-24.el5_2.2  updates           182 k
Installing for dependencies:
 net-snmp                i386       1:5.3.1-24.el5_2.2  updates           698 k

Transaction Summary
=============================================================================
Install      2 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 879 k
Is this ok [y/N]: y

Answer y

Downloading Packages:

(1/2): net-snmp-utils-5.3 100% |=========================| 182 kB    00:02
(2/2): net-snmp-5.3.1-24. 100% |=========================| 698 kB    00:06
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
 Installing: net-snmp                     ######################### [1/2]
  Installing: net-snmp-utils               ######################### [2/2]
Installed: net-snmp-utils.i386 1:5.3.1-24.el5_2.2
Dependency Installed: net-snmp.i386 1:5.3.1-24.el5_2.2
Complete!

Now it is installed
 
 
2. Configure
I’m careful so I do a backup of the snmpd config file.
[root@dull ~]# mv /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.org 
Create a new config file.
[root@dull ~]# vi /etc/snmp/snmpd.conf
rocommunity  public
syslocation  "PDC, Peters DataCenter"
syscontact  peter@it-slav.net
Start the snmpd service
[root@dull ~]# /etc/init.d/snmpd start
Do a snmpwalk to make sure it is working
[root@dull ~]# snmpwalk -v 1 -c public -O e 127.0.0.1
SNMPv2-MIB::sysDescr.0 = STRING: Linux dull 2.6.18-92.1.17.el5 #1 SMP Tue Nov 4 13:45:01 EST 2008 i686
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (16748) 0:02:47.48
SNMPv2-MIB::sysContact.0 = STRING: peter@it-slav.net
SNMPv2-MIB::sysName.0 = STRING: dull
SNMPv2-MIB::sysLocation.0 = STRING: "PDC, Peters DataCentral"
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (1) 0:00:00.01
...
...
Yes, it is working
And finally, make sure snmpd starts next time you restart your machine.
[root@dull ~]# chkconfig snmpd on

 
3. Monitor example
Below is an example of how it looks using op5 Monitor a Nagios based Enterprise Monitor solution.

 
 
4. Useful links

op5 Statistics, a cacti based graph tool
op5 Monitor, an Enterprise Class Monitoring system based on Nagios
Net-SNMP, an open source implementation of SNMP
Cacti, an open source graph tool
Nagios, the number 1 monitor tool

 
			
          

If you think Nagios got everything you need except nice reports. op5 Monitor has the solution to this issue. op5 have customers that are using op5 Monitor just for the reporting part and using other tools for monitoring.

op5 has put alot of resources into reporting because that is the feedback we got from our customers. Here is two examples of reports:

• Availability report
• SLA report

The reports is based on monitoring a friends home file server in Gothenburg, thats why it is not five nines 🙂

If you have problem of keeping track of your video collection. I can recommend VideoDB. It is web based and you enter the name of the movie you want add and VideoDB connects to IMDB and suggest which movie you meant.

When you have picked the correct movie it automatically adds thumbnail, plot and the main actors in the movie downloaded from IMDB.

Screen shot:

To be able to install VideoDB you need:

• Webserver i.e. apache
• MySQL Database backend
• Php

Read more at http://www.videodb.net/

Maybe it is a unix/linux sysadmins basic knowledge how to create ssh keypairs and exchange them. Well I do not do it often enough to remember it but often enough to be irritated that I do not remember how to do it. So here is an excellent description, I will study it and cross my fingers that I will remember the next time how to do it.

When I updated my CentOS system and could not mount XFS file system and my op5 Monitor did not react. I even run check_disk on these mount points but op5 Monitor did not tell me that anything was wrong. So when I managed to get the system to boot and my monitoring system tell me everything was alright I went to bed.
When I checked my email this morning I noticed that my backup script had failed and complained “file does not exist” and so on. I investigated this and found that my XFS file system was not mounted because of the missing XFS kernel module. I also looked in to why my monitor system has not warned me and figured out that check_disk check the file system you point it to and if that is not mounted it will take the parent and the thresholds for that was not reached.
After taken a look at http://www.nagiosexchange.org and found a ruby script that probably would do the job. But I do not want to install ruby on all my monitored systems, so I decided to write my own. It is written in bash and works on CentOS, and probably on most *nix systems.

The script looks in /etc/fstab and compares it with the file systems mounted. If anything is missing, return CRITICAL and the name of the missed mountpoint. The script does not check that some system file systems are mounted i.e. /proc

I have uploaded this script to nagios exchange.

#!/bin/sh
#By peter@it-slav.net
#GPLv2

RESULT=0
TMPFILE=`mktemp /tmp/mount.XXXXXXXXXX`
FSTABMOUNTS=`grep -e '^#' -v /etc/fstab|grep -v  tmpfs |grep -v devpts|grep -v sysfs|grep -v proc|grep -v swap| awk '{print $2}'`
for i in $FSTABMOUNTS
do
        mountpoint $i > /dev/null
        if [ $? != "0" ]
        then
                echo -n "$i " >>$TMPFILE
                RESULT=2
        fi

done
#echo $RESULT
if [ $RESULT != "0" ]
then
        echo "is not mounted" >> $TMPFILE
        echo -n "CRITICAL: "
        cat $TMPFILE
else
        echo "OK: All disks mounted"
fi
rm $TMPFILE
exit $RESULT

Links:

• check_mounts.sh at Nagiosexchange
• op5 Monitor
• My other blog about the problems when I did a kernel upgrade

Yesterday new CentOS kernels was released and could be updated with yum update. I did that on 3 systems and none of them worked afterwards.

The first system is the machine that runs this blog. It runs Xen kernel with two virtual system.Luckily the webserver and database runs in Dom0. After update the system did not start, it seems like the Xen kernel is broken, so now I run non-Xen kernel.

The second system is my op5 Monitor system that monitor my environment. That changed the default kernel to Xen and that was broken, after chenged boot kernel to the non-Xen it works again.

The third machine is my fileserver, with a couple of disks. Two of the filesystems are XFS, and the new kernel did not have support for XFS so my large file systems could not be mounted. So I started the old kernel with XFS support.

This is not an acceptable behavior of an enteprise operating system.

This is real fun. I found another It-Slav with a blog “En it-slav i samhällets tjänst“. He seems to been blogging since 2006 and likes food.

Thanks once again for copying a winner.

For reference see my other blog about Copy Cat.

If you want to test that your Email system is working as supposed with op5 Monitor or Nagios, i.e. send and recieve emails. I recommend Email Delivery.

Features:

• SMTP
• TLS SMTP
• IMAP
• IMAPS
• and much more…

It simply creates an email and send it through an email system and check that the message is delivered. The time is mesured and reported.

If you want to test it through an external emailsystem it is possible to use Gmail. E.g. set <mytestuser>@gmail.com as the receiver of the mail, create <mytestuser> at Gmail and configure forward to <myuser>@my.domain, and let email delivery pick up the email.

I have created the following map to nagios graph to graph the output data:

#Gjort av peter@it-slav.net
#/opt/monitor/op5/nagiosgraph/map_custom/email_delivery.map

# Service type: email delivery

#   output: EMAIL DELIVERY OK - 8 seconds, 8 delay

#   perfdata:

/output:.+?(\d+) seconds, (\d+) delay/

and push @s, [ Email_delivery,

               [ "delivery", GAUGE, $1 ],

               [ "delay", GAUGE, $2 ] ];

After succefully get your 1-wire bus working with OWFS and been able to read the temperatures, it is time to create some nice graphs. The red line is the temperature and the blue is the sliding avarage temperature the last 24 hours. I’m using RRDTools to store the data and create the graphs. If you want to use RRDTools and RRDGraph for your own purposes I recommend you to read the manual pages several times and start easy and build more and more complex solutions.

Here is the script that I’m using to read the temperatures, store the data in a round robin database and finally create the graphs.

#!/usr/bin/perl

# By Peter Andersson peter@it-slav.net

# rrd_tempsensor.pl

use RRDs;

use OW;

# define location of rrdtool databases

my $rrd = '/root/owfs/script/rrd_db';

# define location of images

my $img = '/home/peter/public_html/temps';

# process data for each interface (add/delete as required)

&ProcessSensor(0, "Temperatur i garaget", "127.0.0.1" , "3001" , "10.87507C010800");

&ProcessSensor(1, "Temperatur i PDCn", "op5.mynet",3001, "10.04E060010800");

&ProcessSensor(2, "Temperatur ute", "op5.mynet",3001,"10.DEF05F010800");

sub ProcessSensor

{

# process sensor

# inputs: $_[0]: sensor number (ie, 0/1/2/etc)

#         $_[1]: sensor description

        print "number:$_[0] desc:$_[1], server:$_[2] port:$_[3], id: $_[4]\n";

        my $owserver = "$_[2]:$_[3]";

        unless(OW::init($owserver)) {

            $status = $ERRORS{CRIT};

            $message = "OWServer not running at $owserver\n";

            exit $status;

        }

        # get temperature from sensor

        my $handle = OW::get("$_[4]/temperature");

#       print "handle=$handle\n";

        $handle =~ s/^\s*(.*?)\s*$/$1/;

        ## Check if input is an integer or decimal

        unless (($handle =~ /^-?(?:\d+(?:\.\d*)?|\.\d+)$/) || ($handle =~ /^[+-]?\d+$/))

        {

                print "Not an integer or a decimal\n";

                return($ERRORS{CRITICAL});

        }

        $temp=$handle;

        # remove eol chars

        chomp($temp);

        print "sensor $_[0]: $temp degrees C\n";

        # if rrdtool database doesn't exist, create it

        if (! -e "$rrd/temp$_[0].rrd")

        {

                print "creating rrd database for temp sensor $_[0]...\n";

                RRDs::create "$rrd/temp$_[0].rrd",

                        "-s 300",

                        "DS:temp:GAUGE:600:U:U",

                        "RRA:AVERAGE:0.5:1:2016",

                        "RRA:AVERAGE:0.5:6:1344",

                        "RRA:AVERAGE:0.5:24:2190",

                        "RRA:AVERAGE:0.5:144:3650";

        }

        if ($ERROR = RRDs::error) { print "$0: failed to create $_[0] database file: $ERROR\n"; }

        # check for error code from temp sensor

        if (int $temp eq 85)

        {

                print "failed to read value from sensor $_[0]\n";

        }

        else

        {

                # insert values into rrd

                RRDs::update "$rrd/temp$_[0].rrd",

                        "-t", "temp",

                        "N:$temp";

                if ($ERROR = RRDs::error) { print "$0: failed to insert $_[0] data into rrd: $ERROR\n"; }

        }

        # create graphs for current sensor

        &CreateGraph($_[0], "day", $_[1]);

        &CreateGraph($_[0], "week", $_[1]);

        &CreateGraph($_[0], "month", $_[1]);

        &CreateGraph($_[0], "year", $_[1]);

}

sub CreateGraph

{

# creates graph

# inputs: $_[0]: sensor number (ie, 0/1/2/etc)

#         $_[1]: interval (ie, day, week, month, year)

#         $_[2]: sensor description

        RRDs::graph "$img/temp$_[0]-$_[1].png",

                "-s -1$_[1]",

                "-t $_[2]",

                "-h", "150", "-w", "475",

                "-a", "PNG",

                "-v WWW.It-Slav.Net",

                "--slope-mode",

                "DEF:temp=$rrd/temp$_[0].rrd:temp:AVERAGE",

                "LINE2:temp#FF0000::",

                "CDEF:tempavarage=temp,86400,TREND",

                "LINE1:tempavarage#000088::",

                "GPRINT:temp:MIN:  Min\\: %6.1lf",

                "GPRINT:temp:MAX:  Max\\: %6.1lf",

                "GPRINT:temp:AVERAGE: Snitt\\: %6.1lf",

                "GPRINT:temp:LAST: Nuvarande\\: %6.1lf grader C\\n";

        if ($ERROR = RRDs::error) { print "$0: unable to generate sensor $_[0] $_[1] graph: $ERROR\n"; }

}

Use crontab to run it every 5 minutes:

*/5     *       *       *       *       /root/owfs/script/tempgraph.pl > /dev/null

Are you jealous at the cool scrolling signs that stockbrokers have at their offices. Of course, stock rates are boring, it is much more fun/cooler/geekier to have op5 Monitor or Nagios messages at your scrolling LED.

I find one at Clas Ohlson for 37SEK approx 4 EUR, and after a late night with some perl hacking I got it working. The price at the website is incorrect, when you buy it it costs 37SEK, I bought seven so I know:-)

To be able to program it from a Linux machine you need to download e-badge.

My serial port on my op5 Monitor system was occupied by a GSM modem so I used another machine and did a ssh key exchange so I could run remote commands.

Pre. req. to get my script working is the perl module Nagios::StatusLog. If you use op5 Monitor 4.x or Nagios 3.x you have to diff. patch the /usr/lib/perl5/site_perl/5.8.8/Nagios/StatusLog.pm or where your Statuslog.pm are located.

This is the result:

My perl script that collects the data and sends it to the LED display:

#!/usr/bin/perl
#By peter@it-slav.net
use Nagios::StatusLog;
my $log = Nagios::StatusLog->new(
                Filename => "/opt/monitor/var/status.log",
                Version  => 3.0
           );

my @hosts = $log->list_hosts;
foreach my $host (@hosts) {
  my $hcs = $log->host( "$host" );
  if ( $hcs->status eq 'OK' ) {
    my @host_services = $log->list_services_on_host($host);
    foreach my $service (@host_services) {
      my $svcs = $log->service( $host, $service );
      if ( $svcs->status ne 'OK' ) { # only print for service not in OK
                                     # comment out the if () { } to print everything
        my $output = "\'Service $service on $host is ".$svcs->status."\'";
        &sendtoled ($output);
        }

    }
  } else {
    my $output = "\'Host $host is ".$hcs->status."\'";
    &sendtoled ($output);
  }
}

sub sendtoled {
  print length($_[0])." ".$_[0]."\n";
  system "/usr/bin/ssh", "-l", "peter", "192.168.0.153", "/home/peter/dl/e-badge-1.1/e-badge", "-s", "4", "-r", "1", "$_[0]";
  sleep (int(length($_[0])/3+3));
}

The output from the script is the number of characters and the text sent to the LED.

If you want it to run over and over again you can start an infinite while loop:

 [root@op5 nagiostestscript]# while true; do ./monitor_status_to_led.pl ; done
38 'Service OpenVPN Op5 on gw is WARNING'
38 'Service dc0 Errors on gw is CRITICAL'
42 'Service OpenVPN Sjoberg on gw is WARNING'

Read the rest of this entry »