Filesystem free space monitoring

Filesystem free space monitoring is probably one of the most widely used types of monitors in existence. It is deployed in some form in most organizations that use monitoring. We will discuss this topic using Perl as an implementation language.

First of all despite looking pretty trivial this is actually a pretty complex task. This is because there are many factors that you need to take into account -- thresholds for each filesystem can be different, Action required if "fatal" threshold is breached are individual for each server (or application).

There are several important issues in writing such a script:

• How to check for disappearing mount points.
• How to accommodate HP-UX and AIX, two the most "incompatible" Unix flavors.
• How to exclude CDROMs (false positives) and possibly some other filesystems (For example NSF filesystems might be monitored by their own systems).  Linux and Solaris has option -l in df command, but AIX does not and HP-UX df is completely idiosyncratic (you can use bdf, but in any case this is the worst df implementation on the planet).

  You can also get some relevant information by analyzing output of mount command (In case of SAS harddrives, local filesystems generally start with /dev/sda

  root@unlab12:/home/bezroun # mount
  /dev/sda2 on / type ext3 (rw)
  proc on /proc type proc (rw)
  sysfs on /sys type sysfs (rw)
  devpts on /dev/pts type devpts (rw,gid=5,mode=620)
  /dev/sda5 on /tmp type ext3 (rw)
  /dev/sda7 on /var type ext3 (rw)
  /dev/sda6 on /home type ext3 (rw)
  /dev/sda1 on /boot type ext3 (rw)
  /dev/sdb1 on /u01 type ext3 (rw)
  /dev/sdb2 on /backup type ext3 (rw)
  tmpfs on /dev/shm type tmpfs (rw)
  none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
  sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
  /backup/ISO/TB248-15000.iso on /media type iso9660 (rw,loop=/dev/loop0)

• How to provide individual per filesystem thresholds. There are filesystems where data is constantly written such as /tmp and /var and filesystems that are mostly static such as /usr and /opt. It does not make much sense to have identical thresholds for both.
• How to accommodate both absolute threshold and percentage-based threshold. If filesystem size vary greatly, comparison based on  percentage of space used is a mixed blessing: for example on 100G filesystem 80% used means that 20G is still free (and that's a lot of space) but for 1G filesystem 80% used means that only 200M is available and might well be a critical condition. 
  • Typically you need to provide three thresholds:
    • Warning (percentage). For example 75%
    • Critical (percentage). For example 90%
    • Fatal (in absolute units, for example megabytes). For example, 95%, or -300M(let's agree than negative numbers represent free space and free space percentages, while positive -- used space and used space percentages). This category is better treated as a subset of Serious, then as an independent option.
• How to suppress repeating messages. This is important in case the message triggers some action via enterprise management system; of course many ESMs have their own correlation engine that can do the job, but it is easier and simpler to do it within the script itself.
• How to provide more meaningful 'reactions'.  When filesystem is filling up it is often because there are some log file that was not rotated or older files that are not deleted periodically. Sometimes filesystem is "attacked" and file written are junk or worse. For example a useful report would be the list of all files modified in the last 24 hours sorted by size in reverse order plus list of largest file.
• It would be nice to predict time when the filesystem will be full based on the rate estimated from the last two or more measurements.
•  How best structure configuration files so that definitions form a hierarchy such as: 
  • Generic,
    • Linux
      • hostname
• How automatically detect filesystems that are used by Oracle.

Tips:

• It does not make much sense to use Perl modules like Filesys::DiskSpace. They are not maintained. Actually Filesys::DiskSpace  is a flashback from 1999 and should probably be deleted from SPAN.  I think df -kP is a safer bet. 
• Option -P in df prevent lines splitting and makes processing of output of df simpler. Please note that HP-UX and Solaris versions of df do not support  this option, unless GNU version of df is installed. Notes:

  • You can install GNU coreutils-8.20 on HP-UX boxes. It has df that supports option -P (do not split lines) and also several other utilities fully compatible with Linux.

  • Depot file for each of common versions of HP is available from http://hpux.connect.org.uk/hppd/hpux/Gnu/coreutils-8.20/

NEWS CONTENTS

• 20101114 : Perl script to monitor disk space and send an email by Vivek Gite ( Perl script to monitor disk space and send an email, Nov 14, 2010 )
• 20101114 : scripthat.com • View topic - Perl script to monitor free disk space and send email alert ( scripthat.com • View topic - Perl script to monitor free disk space and send email alert, )
• 20101114 : use FilesysDiskspace - Perl answers ( use FilesysDiskspace - Perl answers, )
• 20101114 : System monitoring tools ( )

Old News

[Nov 14, 2010] Perl script to monitor disk space and send an email by Vivek Gite

This approach is wrong. Filesys::DiskSpace is not maintained, primitive and better should not be used

How to write a perl script that can monitor my disk space under UNIX or Linux and send me an email alert?

There is a nice perl system routine called Perl df or Filesys::DiskSpace. This routine displays information on a file system such as its type, the amount of disk space occupied, the total disk space and the number of inodes etc. This is not a standard module and you need to install it Filesys::DiskSpace

First you need to install this perl module using apt-get or from cpan (Comprehensive Perl Archive Network).

$ sudo apt-get install libfilesys-diskspace-perl

Perl script code to monitor disk space

Now write a perl script called df.pl:
$ vi df.pl
Append following code:

#!/usr/bin/perl
use strict;
use warnings;
use Filesys::DiskSpace;

# file system /home or /dev/sda5
my $dir = "/home";

# get data for /home fs
my ($fs_type, $fs_desc, $used, $avail, $fused, $favail) = df $dir;

# calculate free space in %
my $df_free = (($avail) / ($avail+$used)) * 100.0;

# display message
my $out = sprintf("Disk space on $dir == %0.2f\n",$df_free);
print $out;

Save and close the file. Run this script as follows:
$ chmod +x df.pl
$ ./df.pl
Output:

Disk space on /home == 75.35

So /home has 75.35% free disk space. Next logical step is to compare this number to limit so that you can send an email if only 10% free disk space is left on /home file system. Here is the code with

#!/usr/bin/perl
use strict;
use warnings;
use Filesys::DiskSpace;

my $dir = "/home";

# warning level 10%
my $warning_level=10;

my ($fs_type, $fs_desc, $used, $avail, $fused, $favail) = df $dir;
my $df_free = (($avail) / ($avail+$used)) * 100.0;

# compare free disk space with warning level
if ($df_free %0.2f%% (WARNING Low Disk Space)\n",$df_free);
print $out;
}
else
{
my $out = sprintf("Disk space on $dir => %0.2f%% (OK)\n",$df_free);
print $out;
}

Run script as follows:
$ ./df.pl
Output:

Send an Email – Disk space on /home => 3.99% (WARNING Low Disk Space)

Here is final code that send an email alert ( download):

#!/usr/bin/perl
# Available under BSD License. See url for more info:
# http://www.cyberciti.biz/tips/howto-write-perl-script-to-monitor-disk-space.html
use strict;
use warnings;
use Filesys::DiskSpace;

# file system to monitor
my $dir = "/home";

# warning level
my $warning_level=10;

# email setup
my $to='[email protected]';
my $from='[email protected]';
my $subject='Low Disk Space';

# get df
my ($fs_type, $fs_desc, $used, $avail, $fused, $favail) = df $dir;

# calculate
my $df_free = (($avail) / ($avail+$used)) * 100.0;

# compare
if ($df_free Read man page of this module by typing following command:
$ man filesys::diskspace

scripthat.com • View topic - Perl script to monitor free disk space and send email alert

1. #!/usr/bin/perl -w
2. # Script to check free diskspace and email notifications. Change the email and alert levels and you should be good to go.
3. # created by lb
4. use strict;
6. # Alert levels Warning and Critical - Below what percent level of free disk space do you want an alert?
7. my $alert1 = 30; #Warning level free space below 30%
8. my $alert2 = 10; #Critical level free space below 10%
10. # Put the email address to notify here
11. my $email = '[email protected]';
13. my ($size,$used,$avail,$use,$mounted);
14. my $message;
15. my @list;
16. my $sysname = `/bin/uname -n`;
17. chomp $sysname;
18. my @df = `/bin/df`;
19. my $df;
20. foreach $df (@df) {
21. if ($df =~ /\/\n/) {
22. @list = split(/\s+/, $df);
23. }
24. else {next;}
25. }
27. # Check the usage
28. my $diskfree = (($list[3]) / ($list[2]+$list[3])) * 100.00;
30. # Round the number off to 2 decimals
31. $diskfree = sprintf("%.2f", $diskfree);
33. # See if free disk space is below any of our levels
34. if ( ($diskfree < $alert1) && ($diskfree > $alert2) ) {
35. $message = "Warning Diskspace threshold reached...free space below $alert1% at $diskfree%\n";
36. &mailer;
37. }
39. elsif ( ($diskfree < $alert1) && ($diskfree < $alert2) ) {
40. $message = "Critical Diskspace threshold reached...free space below $alert2% at $diskfree%\n";
41. &mailer;
42. }
44. else {
45. $message = "Free diskspace is good at $diskfree%\n";
46. }
48. #Output to terminal (comment out if you wish)
49. print $message;
50. print "~" x 75, "\n@df","~" x 75,"\n","From system: $sysname\n";
52. #Subroutine for Mail, notifies on warning and critical levels.
53. sub mailer {
54. open(MAIL, "|/usr/sbin/sendmail -t") or die "Cannot open sendmail!: $!";
55. print MAIL "To: $email\n";
56. print MAIL "From: $sysname\n";
57. print MAIL "Subject: $message\n\n";
58. print MAIL "$message";
59. print MAIL "~" x 75, "\n@df","~" x 75,"\n","From system: $sysname";
60. close(MAIL);
61. }

use FilesysDiskspace - Perl answers

I want to monitor disk space and send a email if it exceeds the limit.
I m using "use Filesys:: Diskspace" but there is an error in compling.

Expand|Select|Wrap|Line Numbers

2. "Can't locate Filesys/Diskspace.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at disk.pl line 4.
3. BEGIN failed--compilation aborted at disk.pl line 4."

Thanks
Vasu

That is not part of the core modules of perl:

http://perldoc.perl.org/index-modules-F.html

So as Kevin states, you'll have to install it:
http://cpan.perl.org/misc/cpan-faq.h...l_Perl_modules

I'm not certain this'll be a module you want to use though. By browing the cpan description, you'll notice that the module has not been updated since 1999. This might be a problem, but I would consider looking through the other Filesys modules to determine if there is one that might be better, as in maintained more frequently and therefore supports more file systems.

http://search.cpan.org/search?query=Filesys

Ximinez

Ximinez is a disk usage analyzer and comparator. It enables you to take snapshots of disk usage for a given folder, to browse through the snapshots, and to view differences between snapshots. It is aimed towards solving problems like: "Why is my disk suddenly so full when it was OK just recently?" Besides the GUI, the package also includes an optional command line tool, which can be used, for example, in conjunction with a scheduler to take snapshots on a regular basis.

Tags	Systems Administration Filesystems
Licenses	BSD Revised
Operating Systems	OS Independent
Implementation	Python
Translations	Slovenian English

System monitoring tools

• To: chuck <[email protected]>
• Subject: Re: System monitoring tools
• From: Scott Walters <[email protected]>
• Date: Tue, 18 Aug 1998 09:15:10 -0400 (EDT)
• cc: Scott Williams <[email protected]>, John Sechrest <[email protected]>, [email protected]
• In-Reply-To: <http://www.sage.org/mailarchive/sage-members-archive/1998/msg00428.html>
• Sender: [email protected]

On Tue, 18 Aug 1998, chuck wrote:

First off, I really enjoyed your email.

> Frankly, a good automatic monitoring tool should be able to do
> what you do by hand but better.  If an important machine goes
> down or is heading for trouble, I want to know ahead of time.
> Watcher (or SNMP) can alter you when a disk reaches 95%.
> Watcher can note that a partition is filling fast or that a
> process is running away.  Yeah, you should know that machines,
> but you can automate a lot of this (see bottom for a rebuttal of
> this:)

As I mentioned before I just got done implementing a monitoring system
(using Tivoli, arg, which basically just calls shell scripts that I
wrote ;). One 'big picture' thing i formalized in my implementation
was 'reactions'.  I mean, it's great that now I *know* the drive is
filling up, but what now?  For all of our 'events' I automated reactions
to deal with them.  In the example of disk space, a script gets run which
does a find on the filesystem for all files mod in the last 24 hours and
sorts them by size and emails it to the admin on duty.

I am very pleased with the 'reactions' i have put in place, because now,
not only am i aware of things going wrong, but the first steps i would
take are already taken, and I can just analyze instead of doing grunt
work. 

> #################### ALERTING: ####################

I established three different levels that did different things.  Warning,
Severe and Critical.  Warning just an email, severe email and popup,
critical, page email and popup.  This made it very easy to 'classify'
events.  My environment is not very large (OK, it's small), but it has
been a timesaver for me.  

> but it got us bonus points from auditors and data security.

heh, that's cool.

> Can Tivoli(/SNM/Openview) do this?  Well, kinda sorta, not really.
> You have a nice GUI. that makes some people feel good.  You have
> an API (and every SA has time to learn an API to write their
> scripts in C to talk to these tools); you have support
> (generically).  You have someone else to shoulder responsibility;
> vaguely.

bingo.  in Tivoli's defense the underlying architecture (CORBA) is pretty
solid, and you are given the tools to customize easily.  But out of the
box (for Distributed Monitoring) it's useless.  It seems as though these
products are being developed by ppl that are marketing types vs. admins
who have been in the trenches.

> I want something adaptive ("the machines are usually quiet now, but
> today they aren't, please note")  Nothing replaces ME watching the
> machines and even the graphs (oh sendmail's running hotter than
> usual for a sunday).  I want that.  Must I relearn prolog?

I think that is *very* key.  In my documentation of my implementation I
state, "No software product in the world is going to be able to provide
the root of what needs to happen.  Competent system/network architecture
and operations.  Experience is the only thing I have seen which can really
provide that."

For some reason it seems as though IT managers think software can fix all
of their problems when odds are it's not the software or hardware that
needs fixing, it's the ppl.

> Perhaps a SAGe group to cover SA tools like this might be of
> interest (even just a web page as a resource).  Perhaps I volunteer
> (as long as someone else does the pretty pictures); I'll poke at it.

I would be into that.  Email me directly if you would like to brainstorm.
I have seen a lot of details on how to do this and that with monitoring,
but have yet to see a resource that covers the 'big picture' thinking (ie.
create a loghost for your network, use email address roles <-you really
don't want to have to change email addresses is mucho scripts when bob
leaves do you?, etc).

--
Scott Walters
-PacketPusher

"The world speaks IP"

Internal

• Unix df Command

External:

• df (Unix) - Wikipedia, the free encyclopedia
• df(1)- report file system disk space usage - Linux man page

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: November 16, 2020