Nagios, Finished daemonizing ....

We came across a weird problem in Nagios, after it was working for almost an year. Searched the web, but didn't got any head way for the problem. Every body were complaining about the issue but the resolution were not clear.

I started looking the source code, and found the function in util.c, there were different error conditions but got a hint what the problem could be.

So, all set i started the nagios service and ran strace against the pid. Soon after about 40 seconds, it vanished.

Forgot to mention we were running nagios on RHEL 5.5 32 bit linux. I checked the various options for the nagios process. There is one option for running it in the foreground. The syntax check was fine.

As soon, it was started on the foreground, it dumped with the error message "File limit ....". Couldn't remember the exact output.

There are 2 files which could be the problem, one is nagios.log or nagios.debug. Both were OK. Their is another file perfcheck.pipe or perfmon.pipe (something of this type) in /var/log/nagios.

This was the problem, as it has reached 2Gb (32 bit file system on RHEL) limit. I just rotated the file, and it was working again.

Sometimes, basics work better than a deep dive.

Comments

  1. AnonymousMarch 27, 2012 at 12:53 PM

    Thank you!
    You are right. In my case I received the same log messages in nagios.log. The problem was in perfdata.log. The file reached 2GB.

    I configured the nagios.cfg with log_rotation_method=w (weekly) therefore troubles were happened.
    Now I reconfigured my nagios.cfg to log_rotation_method=d and everything is OK.

    ReplyDelete

Post a Comment

Popular posts from this blog

zpool and power path partial compatibility

Is power path fully supported with zpools, no. It's partial. You can have the pseudo devices assigned to a pool but it uses the native ctd disk only. Recently I lost 2 paths from the vnx array to m9000 server running Solaris 11.1. When I ran the zpool status -x command some of the pools were online while some of them were un available. Power path was reporting 2 paths only but it was active. So ideally the zpools should be online. But it's not. I enabled the remaining 2 paths and all became online. Internally the pools were pointing to one of the ctd devices that belonged to dead path and it was not coming online. Also few of the emcpower devices were renamed, so I had to do zpool import -d with the alternate disk. It seems power path is not completely supported now but may become in future.
Read more

Recoverpoint replication reporting with vmax splitter

Guys, Have you ever thought of automating the RP replication status and send status via email or post it to any reporting platform. Since the splitter impacts the VMAX FA's and there is an overhead of 13% on the utilization its good idea to see if the replication is driving the utilization of vmax FA. There are ways of pulling iops/read/write using symstat but its utilization (Director CPU utilization etc) which plays a significant role. As it crosses 80% we start seeing the increase in device response time. However, to measure FA utilization we will have to go through SMAS UI then pull out the reports. Its manual effort. But there is a way to automate it through REST API. You can get the details what type of objects are supported by pointing to : https://smc server :8443/univmax/restapi Coming to the point, getting recoverpoint consistency group status fully automated doesn't seems to be tough. Lets get through the steps. 1. Hash your SSH keys of the unix host to ...
Read more

Weird problem post power maintenance, no Powerpath pseudo name for Luns

We had the scheduled power maintenance activity in our data center. Every team was notified and we powered doen the servers through cron job. The servers were powered down first then storage (CX-380). And the reverse was followed during startup. After the power shutdown, the SAN was booted up. we forgot to disable the the snapshots. They were still active. Once it came up, the sessions were not there, but snapshot was active. Tried to deactivate the snapshot, failed with the error the "Failing Command: K10SnapCopyAdmin DBid 0 Op 1051. 71 00 80 43" Tried to create a session and attach the snapshot to the session, still unable to deactivate it, neither was able to attach to any other session. Looked at powerlink: It says the command issued to the snapshot is not owned by the SP. I trespassed all the snapshot source lun to SPA, and then tried to deactivate, nothing happened. Rebooted the SP's one by one through the management console. (http:/SP-IP/setup) Then I was able to...
Read more