Grepmail Command

At first thought, grep ought to be able to perform a task like searching mailboxes for specific text. You can search mail files for text, but using grep has at least two disadvantages. First, you may want to retrieve the entire message(s) that matches the pattern, and grep only returns matching lines by default. Second, if any of the mailboxes contain lengthy MIME attachments, searching with grep can produce voluminous output arising from an unlucky false positive within the binary attachment. 

A better tool for this job is grepmail, an open source utility written by David Coppit (see http://grepmail.sourceforge.net for more information). grepmail is designed specifically for searching mail folders. Here is a simple example of its use: 

% grepmail -R -i -l hilton ~/Mail
Mail/conf/acs_w01

I was looking for the phone number of a specific Hilton hotel, which was in a mail message somewhere, but I couldn’t remember where I’d filed it. This command searches for the string “hilton” (-I says to perform a case-insensitive search) in all mail folders under the specified starting directory (-R means recursive), and lists the names of files containing messages that match (-l option). The advantage of this approach is that I can search for the string I remember and find the telephone number even though the two items may be lines apart in the actual message. This command yields the phone number: 

% grepmail -i hilton '!!' | grep -i telephone
Telephone:  619-231-4040

This grepmail command searches for the same string in the mail folder returned by the previous command. This time, grepmail will return the entire message as its output (since -l is omitted). The result is then piped to grep to isolate the phone number. 

Here is a somewhat more complicated command that uses grepmail twice. Its goal is to find messages from user nadia that mention something related to Naples, Italy: 

% grepmail -R -h "^From: .*nadia" ~/Mail | grepmail -b -i \
  "naples|napoli|neapolit"

The first command searches mail headers (-h) for “From” lines including “nadia” somewhere in their text. The second command searches only the body (-b) of the matching messages for the specified strings. 

grepmail has several other useful options: 

-d date — Limit search to messages on the specified date or within the specified date range. The date format is very flexible; see the manual page for details.
-v — Display only non-matching messages.
-u — Display only unique messages.
-M — Don’t search non-text MIME attachments.
-r — Display a report listing each folder searched and the total number of matching messages within it.
-m — Add an X-Mailfolder header to displayed messages; the header’s text will be the path to the message’s mail folder.
-H — Display only the headers of matching messages.

It is also very easy to forward a mail message located in this manner. Here is a simple method:

% grepmail -m -u ... | mail -s subject someone@somewhere

Finally, some people prefer to view the search results from a mail client. This is usually easy to accomplish via a simple script that redirects grepmail’s output to the mailer’s default folder. Several have been created for this purpose: 

• pine: grepine by Cristin Pietsch — http://www.dfki.de/~pietsch/software 
• mutt: grepm by Moritz Barsnick — http://www.barsnick.net/sw/grepm.html 
• VM: vm-grepmail.el by Robert Fenk — http://www.robf.de/Hacking/elisp/vm-grepmail.el 

Search Operations for Software Packages 

Software packages are another item whose contents are hard to search with grep. More specifically, I often want to answer questions like these: 

• Is a specific package installed? 
• What package does a specific file belong to? 
• What packages are available on an individual CD (or other media)? 
• What is included within a package (installed and not)? 

On many systems, one or more of these questions can be answered using the package management tools supplied with the operating system. For example, the following commands can be used to list all currently installed packages on various UNIX systems: 

Linux: rpm -q -a
FreeBSD: pkg_info -a -I
Solaris: pkginfo
HP-UX: swlist
AIX: lslpp -l all 

You can pipe any of these commands to grep to determine whether a specific package is present to find its actual package name. For example, the following command lists all packages related to LDAP installed on a Linux system: 

% rpm -q -a | grep -i ldap
nss_ldap-184-1
openldap-2.0.23-4
openldap-clients-2.0.23-4
openldap-servers-2.0.23-4

This system has the OpenLDAP servers and client utilities installed, as well as the modules that interface LDAP to PAM and to the name service switch file, /etc/nsswitch.

It’s often useful to find out which package a particular file is part of (e.g., when you delete it accidentally and need to restore it). These command forms will indicate which package installed the specified file:

Linux: rpm -q ---whatprovides path
Solaris: pkgchk -l -p path
AIX: lslpp -w path

Here is an example from a Solaris system:

% pkgchk -l -p /etc/init.d/sendmail
Pathname: /etc/init.d/sendmail
Type: editted file
Expected mode: 0744
Expected owner: root
Expected group: sys
Referenced by the following packages:
        SUNWsndmr
Current status: installed

When you want to know what is contained in an installed package, use these commands:

Linux: rpm -q -l name
FreeBSD: pkg_info -L name
Solaris: pkgchk -l name | grep "^Pathname:"
HP-UX: swlist -l file
AIX: lslpp -f name

Here is an example from a FreeBSD system:

% pkg_info -L grub-0.91_1
Information for grub-0.91_1:


Files:
/usr/local/bin/mbchk
/usr/local/info/grub.info
/usr/local/info/multiboot.info
/usr/local/sbin/grub
...

In general, if you want to list the contents of an uninstalled package, you can replace the package name with the path to the package file in the preceding commands. On Linux systems, however, you must precede the package name with the -p option.

Only HP-UX and AIX have easy-to-use commands for listing the packages available on CDs or other media: 

HP-UX: swlist -s path-or-device
AIX: installp -l -d device 

On Linux, FreeBSD, and Solaris systems, you must rely on GUI package management tools to handle this function. On Linux systems, you can use gnorpm and similar packages (as well as yast2 on SuSE Linux systems). Under FreeBSD systems, you can use the sysinstall utility and select the Configure=>Packages menu path. On Solaris systems, the Supplementary Software CD includes a GUI installation tool that starts automatically when the CD is inserted, and it can be used to view the contents of the CD as well. On all three systems, you can also examine the directory containing the package files with ls for a quick listing of what is available. 

Searching Net-SNMP MIBs 

The Simple Network Management Protocol (SNMP) can be used to monitor and reconfigure a wide variety of computer systems and other network devices. The items that can be queried or set are defined in Management Information Bases (MIBs). A MIB is a collection of value and property definitions, and the various items are organized as a tree structure. This hierarchical organizational scheme serves to group related data together. MIB definitions are stored in files and are implemented in the software on the actual computers and devices. The MIB does not hold any data — it is a schema, not a database.

Here is an example MIB item:

iso.org.dod.internet.mgmt.mib-2.system.sysLocation = "Machine Room"

The long string on the left is the setting’s name, and its value is the string to the right of the equals sign. The name is separated into components by periods, and each corresponds to successive levels of the MIB tree. Thus, we can see that the sysLocation node is eight levels from the top of the tree.

Although the MIB is organized as a tree, it is not uniformly populated. The top four levels of the standardized MIB tree exist mainly for historical reasons. Given this rather ad hoc structure, searching the MIB tree for specific items is often essential. However, it is not a job for grep.

Most SNMP implementations provide utilities for examining MIBs. The open source SNMP implementation Net-SNMP is used on Linux and FreeBSD systems (and other UNIX systems, if desired). The tool the package provides to examine the MIB structure is snmptranslate. This command provides information about the MIB structure and its items. For example, you can use it to display a MIB subtree, as in this example:

% snmptranslate -Tp .iso.org.dod.internet.mgmt.mib-2.system
+--system(1)
   |
   +-- -R-- String    sysDescr(1)
   |        Textual Convention: DisplayString
   |        Size: 0..255
   +-- -R-- ObjID     sysObjectID(2)
   +-- -R-- TimeTicks sysUpTime(3)
   +-- -RW- String    sysContact(4)
   |        Textual Convention: DisplayString
   |        Size: 0..255
   ...

I’ve truncated the output after four entries.

snmptranslate can also provide detailed information about a specific MIB item, as in this example using the sysLocation leaf:

% snmptranslate -Td .iso.org.dod.internet.mgmt.mib-2. \
  system.sysLocation
1.3.6.1.2.1.1.6
sysLocation OBJECT-TYPE
  -- FROM       SNMPv2-MIB, RFC1213-MIB
  -- TEXTUAL CONVENTION DisplayString
  SYNTAX        OCTET STRING (0..255)
  DISPLAY-HINT  "255a"
  MAX-ACCESS    read-write
  STATUS        current
  DESCRIPTION   "The physical location of this 
           node (e.g., 'telephone closet, 3rd 
           floor'). If the location is unknown,
           the value is the zero-length string."
::= { iso(1) org(3) dod(6) internet(1) \
  mgmt(2) mib-2(1) system(1) 6 }

However, the most important searching feature — finding the location within the tree of a specific leaf — is not provided automatically by snmptranslate. This command will provide that information for the memTotalReal item:

% snmptranslate -Ts | grep memTotalReal\$
.iso.org.dod.internet.private.enterprises. \
  ucdavis.memory.memTotalReal

This item, the total real memory present on a system, is located at the specified point within the hierarchy. A slightly more complex command can provide both the full location and a description for a MIB leaf:

% snmptranslate -Td 'snmptranslate -Ts | grep memTotalReal\$'

I use it often enough that I’ve defined an alias for this command:

% alias snmpwhat 'snmptranslate -Td `snmptranslate -Ts | grep \!:1\$`'

Filtering Foreign Language Email 

Like many people, I use procmail to preprocess mail messages, including attempting to remove spam. My current recipes work reasonably well for mail messages in Western languages, but they fail for ones in many other languages (e.g., Japanese, Chinese, Russian). Currently, I get 15-20 such spam messages each day. 

Some people deal with this situation by discarding all email from the corresponding countries, but this approach does not work for me as I get legitimate mail from these countries on a regular basis (from non-predictable senders). What I needed was a procmail recipe to identify the foreign characters, which are above the normal ASCII range. The trick here is to get all of these characters into the .procmailrc file. This is easiest to do by entering them on a system/application that supports two-byte characters. The next step is to copy that file in binary mode to the system where procmail is run where its contents can be pasted into the initialization file.

A quick and dirty procmail recipe will look something like this when viewed with most text editors:

:0BH:
* [\200\201\202...\377][\200\201\202...\377][\200\201\202...\377]
$MAILDIR/foreign_spam

For me, three such characters in a row was a good enough first attempt at solving this problem. There are many more elegant solutions available on the Web. One of the best is by Walter Dnes, and it is available at: 

http://www.waltdnes.org/email/chinese/index.html

It takes advantage of procmail’s weighting capabilities to detect messages containing more than 5% non-ASCII characters. 

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019