|Home||Switchboard||Unix Administration||Red Hat||TCP/IP Networks||Neoliberalism||Toxic Managers|
May the source be with you, but remember the KISS principle ;-)
Skepticism and critical thinking is not panacea, but can help to understand the world better
|News||Enterprise Unix Administration||Recommended Books||Recommended Links||Sysadmin Horror Stories||Creative uses of rm||Performing the operation on a wrong server|
|Shadow IT||Project Management||Programmable Keyboards|
|Slackerism||Information Overload||Workagolism and Burnout||Unix Sysadmin Tips||Orthodox Editors||Admin Humor||Sysadmin Health Issues|
Recently I decided to publish several written by me over the years sysadmin utilities (the exact set of utilities I plan to publish is subject to change without notice).
Previously I published on GitHub three utilities from this set: neatperl, neatbash and saferm . They were by-and-large ignored. So here much depends on the positive feedback loop. Without it project will be abandoned soon. As Fred Brooks notes in 1975 in his groundbreaking book "The Mythical Man-Month: Essays on Software Engineering" the difference in effort required to produce the utility "for yourself" and its publishable form can be ten times or more.
The trend now is toward "integrated" configuration management solutions like Ansible ( reinvention of IBM JCL on a new level), which have their own weak spots Some of the utilities listed below can be used as steps in Ansible playbooks you develop.
Also finishing those utilities to the level acceptable for publication is a very labor intensive process. I recently spend 12 hours on "finishing touches" for such a simple utility as dormant_user_stats ;-). Which is a tiny and a very basic utility, essentially a more intelligent, more polished Perl implementation of a three line Bash script:
find /home -maxdepth 1 -type -d -mtime +365 | xargs -L 1 du -sh du -dh /home find /home | wc -l
Instead of basic output dormant_user_stats produces neat table with all kind of additional stats required for deciding whether reclaiming the space is worth the candles burned in the process, and if yes, for which 20% of users in the list above it should be done :-).
It is useful mainly for large installation and for maintainers of web sites like Softpanorama which store the content in the Unix directories tree not in MySQL or other database. In this case knowing what branches are stale and need to be iether updated or archived, saving space and i-nodes, is as important as knowing, which users abandoned the particular server, and the space of their home directories can be reclaimed on the large enterprise servers. When the number of users often exceeds 300 primitive bash script like above can run several hours and you can benefit form some optimization which dormant_user_stats implements (which first of all means more careful filtering).
But in any case such utilities can run several hours, or several days (on petabyte storage), especially if multiple users have multi-terabyte directories and millions of files. I have another utility called dir2tar which does this safely with all kind of verifications in the process and can be used along with dormant_user_stats for the top 20% of dormant users or, more correctly, dormant directories.
Again, those utilities exists, and many of them were used by me for a decade or more, but for publishing they need polishing, time for which is very scarce.
Among around 200 utilities which I have wrote during my carrier, the following might have some value to other sysadmin:
[Published] msync -- rsync wrapper that allow using multiple connections for transferring compressed archives or sets of them orginized in the tree (which are assumed iether consist of a single files to a subdirectory with the archive split into chunks of a certain size, for example 5TB ) . Files are sorted into N piles, where N is specified as parameter, and each pile transmitted vi own TCP connection. useful for transmitted over WAN lines with high latency. I achieved on WAN links with 100Ms latency results comparable with Aspera using 8 channels of transmission.
It organizes files into given number of "piles" and transfers all piles in parallel using for each separate invocation of rsync or tar pipe to the target server. If program is terminated before full transfer took place, on next execution partial transfers are completed using rsync. Each time it will start from the point it was terminated operating like rsync but for the whole directory tree or file list. It scans the content of the remote size and transmit only those files what not yet transmitted.
The script tried to verify whether files were already transferred and if missing files are detected it transfers only missing
files (partially transferred files transferred using rsync)
Detects all "critical" changed file, diffs them with previous version, and produces report. All information by default in stored /var/Dirhist_base (configurable via via config file.) You can specify "watched" files and directories within those folders in a format similar to YAML dictionaries, or Windows 3 ini files.
If any of "watched" files or directories changes the utility can email you the report to selected email addresses, to alert about
those changes. Useful when several sysadmin manage the same server. Can also be used for checking, if changes made were documented
in GIT or other version management system with the utility admpolice.
Very useful for working with remote or critical production server, where a mistake or accidental typo can cost a lot pain/money.
Originally was used for preventing accidental reboots, but later made more universal. It allows to create a set of aliases, sourcing
which will prevent execution of the command if it is submitted in interactive session printing inread the context in which particular
command will be executed (customizable via configuration file.) Operated via concept of dangerous options, set of which can be specified
for each command to alert sysadmin about possible tragic consequences' or a rush. impulsive run of such a command, typical
when sysadmin is stressed and/or is working under tremendous pressure. In such circumstances it is important not to make the
situation worse. So if you type the command reboot the utility will print the HOSTNAME of the server you are on, and
will ask you to resubmit the command. while for a command like find it provides PWD the list of
"dangerous options" used, if any. Useful when working on remote server, which in case you do something nasty might require
you to buy an airline ticket and fly to the destination instead of TCP packets.
It requires the usage of custom prompt function (provided as my_prompt function in the distribution file dir_favorites_shell_functions.sh) which should be referenced in .bash_profile/.bashrc as:
The second function that uses this utility is function go ( also provided in dir_favorites_shell_functions.sh). It displayed list of directories and allows to cd to any of them by putting the number of copying and editing the path. For example:
root@lustwz99: # go 0 ~/.config 1 ~/bin 2 ~/.config 3 /etc/sysconfig 4 /etc/sysconfig/network-scripts 5 /etc/profile.d 6 /var/log 7 /etc --- the divider between static part and dynamic part of favorites; directors below are generated dynamically from the history 8 /tmp/rhel 9 /tmp/rhel/isolinux 10 /mnt/Packages ... ... ... .. Select directory by the number (negative from bottom) or copy and edit the path:Favorites are stored in Bash directory stack accessible by dirs command. and so on as well as via function go also provided in dir_favorites_shell_functions.sh
If it is invoked with option -m also changes directory favorites in MC (Midnight commander) providing dynamic
updates. In this case mc hotlist consist of two parts -- static and dynamic.
Written long ago, during the period when a 4GB USB drive was large. Used mainly for creating a set of HTML pages describing managed servers in some details as a party of creation of inventory. Allows custom fields for putting information not collectable automatically. Can be used via Ansible.
In view of existence of Relax-and-Recover and cheap 256GB Fit drives might not be that important to have, although it collects some dynamic configuration information from the current session (output of ifconfig and other utilities that provide the current map of connections, disk map, etc.) Useful on small cluster, especially if you use SGI as your scheduler.
It incorporates several "no nonsense" checks that make success of the conversion more probable. I experienced around 30% failure rate iin my tests and 10% (or one server out of ten failed to report after the conversion was finished). Serious troubles include but not limited to deletion of hundreds of vital RPMs ( due to my mistake made out of frustration; nit stll no protection from this kind of errors), if safety measures are ignored. So failures due to the lack of pre-conversions checks is not a hypothetic scenario, especially failure on the state of rolling our Oracle RPMs, when the system in "transitional state" and can't be safely rebooted. It is an important safety measure if you convert multiple CentOS or RHEL servers to Oracle Linux and need to convert important production servers. Man page at centos2ol_wrapper. See Converting CentOS to Oracle Linux with centos2ol script for details
Uses pbzip2 for compression by default. Useful if resulting archive is over 100GB as it checks for interruptions in archiving and many other abnormal situations. It also recognized already archived directories. Can work in tandem with the dormant_user_stats. Compression program used can be changed via option -a, for example -a pigz . For obvious reasons for large archives only parallel versions of compression programs are acceptable. Tested on tarballs up to 20TB.
"Primitive", one level free space monitoring program can be written in an hour or so, but intelligent monitoring of free space with the suppression of redundant messages and flexibility as for whom for particular filesystem you need to send emails and what action you need to take is the last critical threshold is breached, is difficult without using some kind of correlation engine.
This utility tries to imitate correlation engine and provides three level of alerts (Warning, Serious, and Critical/Fatal) for free space with the ability to specify a mail list (individual for each monitored filesystem) and an action for the last level (also for each monitored filesystem; action can be arbitrary command or script, for example a shutdown of the server, cleaning some folder, or blocking of user logins).
Emails warning about insufficient disk space sent to the most recent users, or specified mail list. Blocks "spam" emails and sends exactly one email after crossing each threshold. Allow to specify fractions of percentage for the last (critical) threshold, for example 99.99%.
Very useful for working with remote or critical production server, where a mistake or accidental typo can cost a lot pain/money. Originally was used for preventing accidental reboots, but later made more universal. It creates a set of aliases for several "potencially dangerious" utilities(rm, find, chmod, chown, etc), sourcing which will prevent execution of the command if it is submitted in interactive session printing instead the context in which particular command will be executed (customizable via configuration file.)
Operated via concept of dangerous options, set of which can be specified for each command in the config file for this utility. If and of "dangerous options" is detected, the utility will alert sysadmin about possible "tragic consequences" of rush, impulsive run of such a command, typical when a sysadmin is stressed, tired/sleepy/overcafienated, and/or is working under tremendous pressure. Extreme fatigue and exhaustion often leads to SNAFU that makes an existing difficult situation far worse.
Or just distracted or absent minded who failed to concentrate and access the environment in a proper way.
If option LS is specified in config file for this utility and dangerous options are detected the command tried to convent the command into an ls command and execute it to give you better understanding about which file are affected. Sysadmins know that this is one of the best way to prevent SNAFU with find -exec (or find -delete, or chmod -LR something) but seldom follow this recommendation.
In such circumstances it is important not to make the situation worse. So if you type the command reboot the utility will print the HOSTNAME of the server you are on, and will ask you to resubmit the command. while for a command like find it provides PWD the list of "dangerous options" used, if any. Useful when working on remote server, which in case you do something nasty might require you to buy an airline ticket and fly to the destination instead of TCP packets.
Again, one of the most useful features of this command is option LS, which you can specify in config file for any command, and which attempts to change the command such a find with exec option and executing the same command with '-ls' instead. For command like rm, chmod and chown this is simpler: you just need discard all options and provide ls with "naked arguments". But of course the devil is in details as some arguments might contain spaces, or special characters (like -name in find or the first argument in grep) and need to be quoted.
Delay serves as a safety valve that allow you to correct some dangerous typos, automatically specifying absolute path for the backup of system directory (/etc instead of etc), and like.
Useful for computational cluster nodes, especially if some applications suffer from memory leaks.
Help to create a usable history of directories most frequently accessed (favorites). It requires the usage of custom prompt function (provided in dir_favorites_shell_functions.sh.) Favorites are stored in stored in stack accessible by dirs command and consist of two parts -- static (loaded from a file $HOME/.config/dir_favorites.$HOSTNAME ) and dynamic (created from history). The utility also dynamically generates aliases such as cd--, cd---, cd-4, cd-5 and so on as well as via function go also provided in dir_favorites_shell_functions.sh
If it is invoked with option -m also changes directory favorites in MC (Midnight commander) providing dynamic updates. In this case mc hotlist consist of two parts -- static and dynamic.
NOTE: the set of shell function which are necessary for this ustility is provided in dir_favorites_shell_functions.sh
Designed to run mainly from cron. Uses a different, a much simpler approach to the same problem than the etckeeper and extends it to arbitrary number of directires. Does not use GIT or any other version control system as they proved to be of questionable utility for syadmins, unless there are multiple sysadmins on the server.
It does not have the connected with the usage of GIT problem with incorrect assignment of file attributes when reconverting system files.
If it detects changed file it creates a new tar file for each analyzed directory. For example /etc, /root, and /boot
Detects all "critical" changed file, diffs them with previous version, and produces report.
All information by default in stored in /var/Dirhist_base. Directories to watch and files that are considered important are configurable via two config files dirhist_ignore.lst and dirhist_watch.lst which by default are located at the root of the /var/Dirhist_base tree ( as /var/Dirhist_base/dirhist_ignore.lst and /var/Dirhist_base/dirhist_watch.lst )
You can specify any number of watched directories and within each directory any number of watched files and subdirectories. The format used is similar to YAML dictionaries, or Windows 3 ini files. If any of "watched" files or directories changes, the utility can email you the report to selected email addresses, to alert about those changes. Useful when several sysadmin manage the same server. Can also be used for checking, if changes made were documented in GIT or other version management system (this process can be automated using the utility admpolice.)
For more information see dirhist utility
Useful for provisioning multiple servers that use traditional authentication, and not LDAP and for synchronizing user accounts between multiple versions on Linux . Also can be used for "normalizing" servers after acquisition of another company, changing on the fly UID and GID on multiple servers, etc. Can also be used for provisioning computational nodes on small and medium HPC clusters that use traditional authentication instead of DHCP.
Useful for transfer of sets of trees with huge files over WAN links. In case of huge archives they can be split into chunks of a certain size, for example 5TB and orgnized into a directory. Files are sorted into N piles, where N is specified as parameter, and each pile transmitted vi own TCP connection. useful for transmitted over WAN lines with high latency. I achieved on WAN links with 100Ms latency results comparable with Aspera using 8 channels of transmission.
Useful for large RAID5 arrays without spare drive, or other RAID configurations with limited redundancy and critical data stored. Currently works with Dell DRAC only, which should be configured for passwordless ssh login from the server that runs this utility. Detects that a disk in RAID 5 array failed, informs the most recent users (default is those who login during the two months), and then shuts down the server, if not cancelled after the "waiting" period (default is five days).
The utility lists all users who were inactive for the specified number of days (default is 365). Calculates I-nodes usage too. Can execute simple commands for each dormant user (lock or delete account) and generates a text file with the list (one user per line) that can be used for more complex operations.
for more information
Google matched content
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2020 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|You can use PayPal to to buy a cup of coffee for authors of this site|
Last modified: December 30, 2020