May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Side effects of patching

News Sysadmin Horror Stories Recommended Links Creative uses of rm Mistakes made because of the differences between various Unix/Linux flavors Missing backup horror stories Lack of testing complex, potentially destructive, commands before execution of production box Pure stupidity
Locking yourself out Premature or misguided optimization Reboot Blunders  Performing the operation on a wrong server Executing command in a wrong directory Side effects of performing operations on home or application directories Typos in the commands with disastrous consequences Side effects of patching
Multiple sysadmin working on the same box Side effects of patching of the customized server Ownership changing blunders Dot-star-errors and regular expressions blunders Excessive zeal in improving security of the system Unintended consequences of automatic system maintenance scripts LVM mishaps Abuse of privileges
Safe-rm Workaholism and Burnout Coping with the toxic stress in IT environment The Unix Haterís Handbook Tips Horror stories History Humor Etc

Patching horrors are much more common then you would suspect. Some companies (Sun) test patches pretty well. Other (IBM, HP) less well. Some (Novell) introduce important changes via patches which can lead to completely hosed system.

It goes without saying that patches. and, especially, service packs need to be tested on quality server before being applied to production server.  The most deadly situations arise when the system patched has complied Sendmail, Bind or other popular open source package. Such packages are often overwritten during patching with unpredictable results. 

Even without custom components installed applying service packs can be very interesting experience:

The best way to install service packs is using installation DVD (which has an upgrade option). Attempts to do it from Novel website using regular patch mechanism are more risky. 

At the same time expect troubles in any case and reserve enough downtime. Testing in the lab is a necessary preliminary step for production servers. Do not dive blindly. It might not reveals all the problems that you might encounter on a production server but it tremendously helps.

SP3 is actually more like a major upgrade then a service pack. For example, it tries to convert /boot to /by-uud/ format (and often screw /boot partition as a result).  For example all partitions not controlled by LVM will be mounted using /by-id/ scheme, for example:

/dev/disk/by-id/scsi-36001ec90e5265400103d932e4fea1f42-part2 /boot ext3 acl,user_xattr 1 2

After the upgrade most server have the problem with the multipath driver which is installed by default.  It can demonstrate itself  in two ways:

If the server does not use multipath driver the easiest way to fix this problem is to disable it. You can boot into rescue mode, mount /boot partition and disable multipath on /boot/grub/menu.lst by adding multipath=off to the kernel directive. The other way to solve this problem is to manually rebuild initrd. Both fixes are not permanent and will probably be overwritten by the next fixpack (SP3 actually reliably wipes out multipath=off from /boot/grub/menu.lst).

Its unclear what to do if multipath is really used on the server. In this case it might make sense not apply online patches after upgrading to SP3 or stay with SP2 and wait until SP4 will be available. 

Dr. Nikolai Bezroukov

Top Visited
Past week
Past month


Old News ;-)

[Jun 19, 2010] Sysadmin horror story

May 06 2000 |

A story of a novice or inattentive sysadmin wiping out vital parts of a computer system by such commands as "rm -rf /" or "rm /vmunix". It becomes more horrific if there were no readable backups.

Here's the worst thing I have yet done as a sysadmin. I had to upgrade several machines from HP-UX 9 to HP-UX 10. Having done the workstations with reasonable ease, I set aside a Saturday to do the departmental mail server. I installed HP-UX 10 from the CD-ROM HP had sent me with no problems, just a long wait spent reading Usenet on my laptop, made our standard local modifications and checked that email was up again.

Then I had a large number of patches to install from another CD-ROM from HP (both CD-ROMs were part of their Year 2000 Transition Kit). This meant another long wait, and it was quite late in the afternoon by the time they had installed. I thought that since these patches came directly from HP, and the system had been fine before I installed them, it was unlikely that it would suddenly stop working. So I let it reboot and went home without testing it.

On Monday morning, I found out that email had completely stopped working. Eventually I discovered the reason why. On our systems, /usr/sbin/sendmail is a link to /usr/sbin/exim, which is a link to the real exim, which is on the /usr/dpmms filesystem on our mail server, which is NFS-exported to our other machines. I had been careful not to overwrite anything on /usr/dpmms during the install. However, the patch program (swinstall), on being asked to replace /usr/sbin/sendmail, had followed the chain of links and replaced the REAL exim with sendmail! As you can imagine, my users were not happy about the long disruption to email. Fortunately I was able to restore exim from a working backup tape.

Linux kernel 2.6.33 released - openSUSE Forums

Re: Linux kernel 2.6.33 released


dale14846 View Post

Does this mean that the factory kernel that panics during boot will soon be fixed? There's nothing wrong with the 2.6.33 kernel, at least the mainline one. Works here for me. The only thing that's not working is the nvidia driver which won't compile if you get the stable official one (190.53) and not the beta (or apply the patch to the 190.53 driver which lives somewhere on the gentoo bugzilla - the one on the nvidia forum is flawed). In 2.6.33, kernel internals changed so the 190.53 driver is not compatible with the changes, thus fails to compile. It seems the beta driver does support the new kernel version

Now if you get a kernel panic from the factory kernel, that's a different issue. A kernel won't panic if it can't load a module (because it's either not present or can't be loaded, esp if the module is a video driver). There's something else going on with the factory kernel

PS: I stopped using SUSE kernels a while ago, they seem to always introduce some problems and the way they split the kernel in a gazillion different packages, I find ridiculous. I switched to mainline kernel and have been happy ever since (also had much less issues with it). But yeah, this requires some knowledge on how to properly configure and compile mainline kernel, which luckily I have Re: Linux kernel 2.6.33 released

On 02/25/2010 11:26 AM, dale14846 wrote:
> microchip8;2126710 Wrote:
>> 'Linux 2 6 33 - Linux Kernel Newbies'
>> (
>> Compiling now

> Does this mean that the factory kernel that panics during boot will
> soon be fixed?

There was a problem with a long-standing kernel bug that was activated by the
switch to gcc 4.5. If that was the one that got you, then it is fixed, but that
has nothing to do with the usage of 2.6.33-rcX kernels.

[Sept 15, 2003] Fix-It Fatigue By John Foley, George V. Hulme.

Patching became a problem and not only on Microsoft operating systems ;-)

With the Blaster worm seeming to be under control, alleged virus-author Jeffrey Parson under house arrest in Minnesota, and hacker Adrian Lamo under the watchful eye of the feds, business-technology managers may have enjoyed a few hours of peace and quiet last week. But it was short-lived. On Sept. 10, Microsoft issued a security bulletin warning of three new critical vulnerabilities in the Windows operating system, sending systems administrators rushing to patch their computers. It's become an all-too-common scenario--and one that's causing some businesses to re-evaluate their heavy reliance on Microsoft products.

A year-and-a-half after Bill Gates declared that trustworthy computing had become Microsoft's No. 1 priority, the software bugs keep coming. The latest vulnerabilities involve the Remote Procedure Call service in Windows, making it possible for a malicious hacker to take control of a target system, introduce an infectious worm, or launch a denial-of-service attack. A week earlier, Microsoft issued five other warnings, four involving the omnipresent Office applications suite. For the year, the tally stands at 39.

And those are just the holes that have been uncovered by others and reported to Microsoft. In addition, the software vendor is combing through its code, finding holes, and issuing patches without publicizing the flaws. No one knows how many more are yet to be uncovered. "There's no way to wrap your hands around that," says Dan Ingevaldson, engineering manager with security vendor Internet Security Systems Inc.

Some business and technology professionals are running out of patience. "The issues around these vulnerabilities are escalating to the point where it's not just CIOs or CTOs, it's corporate officers, it's boards of directors asking: 'What are we going to do?'" says Ruth Harenchar, CIO of Bowne & Co., which last week scrambled to patch 4,500 Windows PCs and 500 servers in the United States and more overseas. "The situation appears to be getting worse, not better."

The patching work has thrown Bowne & Co.'s technology projects off schedule. Now, the specialty-printing-services company is assessing its options. Among them: redesigning its network around a thin-client model to reduce the number of PCs running Windows and, on other machines, migrating to Linux. "It's getting to be enough of a burden that you have to seriously start thinking about alternatives," Harenchar says.

Raymond James & Associates has assembled a team of IT staffers to manage the constant patching. "Organizations have to mobilize and realize this is going to be a way of life for the foreseeable future," says VP of IS Gene Fredriksen.

The financial-services firm, with offices around the world, last week began the arduous task of patching 10,000 PCs and 1,000 servers. "The pressure is on," Fredriksen says. "Anybody that isn't patched by the weekend is going to have trouble." The fear is that the latest vulnerability leaves Windows computers open to a Blaster-like worm. "There's a very good chance that a worm is going to be developed" to take advantage of the latest security holes, says ISS's Ingevaldson.

"People are getting fed up," says Lloyd Hession, chief information security officer at financial-network provider Radianz, adding that the number of Windows patches is reaching "epic proportions." The situation is causing more than just a few disgruntled customers to re-evaluate how much they use Microsoft products. Says Gartner security analyst John Pescatore, "There's definitely a very large trend towards that."

Recommended Links

Google matched content

Softpanorama Recommended

Top articles


How To Survive With Many Patches Introduction to Quilt



Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy


War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotesSomerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose BierceBernard Shaw : Mark Twain Quotes


Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law


Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOSProgramming Languages History : PL/1 : Simula 67 : C : History of GCC developmentScripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-MonthHow to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Haterís Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2021 by Softpanorama Society. was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

You can use PayPal to to buy a cup of coffee for authors of this site


The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019