Loss of architectural integrity of Unix is now very pronounced in RHEL. Both RHEL6 and RHEL7 although 7 is definitely
worse. And this is not only systemd fiasco. For example,
recently I
spend a day troubleshooting an interesting and unusual problem: one out of 16 identical (both hardware and software wise) blades in a small HPC cluster (and only one) failed to enable bonded interface
on boot and this remained off line. Most sysadmins would think that something is wrong with hardware. For example, Ethernet card on
the blade and/or switch port or even internal enclosure interconnect. I also initially thought his way. But this was not the case
;-)
Tech support from Dell was not able to locate any hardware problem although they did diligently upgraded CMC on enclosure and BIOS and
firmware on the blade. BTW this blade has had similar problems in the past and Dell tech support once even replaced Ethernet card in
it, thinking that it is culprit. Now i know that this was a completely wrong decision on their part, and waist of both time and
money :-), They come to this conclusion by swapping the blade to a different slot and seeing that the problem migrated into
new slot. Bingo -- the card is the root cause. The problem is that it was not. What is especially funny is that
replacing the card did solve the problem for a while. After reading the information provided below you will be as puzzled as me why
it happened.
But after yet another power outage the problem returned.
This time I started to suspect that the card has nothing to do with the problem. After more close examination I discovered that in its infinite wisdom in RHEL 6 Red Hat introduced a package called
biosdevname. The package was developed by Dell (the fact which seriously
undermined
my trust in Dell hardware ;-).
This package renames interfaces to a new set of names, supposedly consistent with their etching on the case of rack servers. It
is useless (or more correctly harmful) for blades. The package is primitive and does not understand that server is a rack
server or blade.
Moreover by doing this supposedly useful remaining this package introduces in 70-persistent-net.rules file a
stealth rule:
KERNEL=="eth*", ACTION=="add", PROGRAM="/sbin/biosdevname -i %k", NAME="%c"
I did not look at the code but from the observed behaviour it looks that in some cases in RHEL 6 (and most probably in RHEL 7
too) the package adds a "stealth" rule to the END (not the beginning but the end
!!!) of /udev/rules.d/70-persistent-net.rules file, which means that if similar rule in 70-persistent-net.rules
exits it is overwritten. Or something similar to this effect.
If you look at Dell knowledge base there are dozens of advisories related to this package (search just for biosdevname).
Which suggests that there some deeply wrong with its architecture.
What I observed that on some (the key word is some, converting the situation in Alice in Wonderland environment) rules for
interfaces listed in 70-persistent-net.rules file simply does
not work if this package is enabled. For example Dell Professional services in their infinite wisdom renamed interfaces back
to eth0-eth4 for Inter X710 4-port 10Gb Ethernet card that we have on some blades. On 15 out of 16 blades in the Dell
enclosure this absolutely wrong idea works perfectly well. But on blades 16 sometimes it does not as a result this blade does
not boot after power outage of reboot. When this happens is unpredictable. Sometimes it boots, but sometimes it does not. And you can't understand what is happening, no matter how hard you try because of stealth
nature of changes introduced by
biosdevname package.
Two interfaces on this blades (as you now suspect eth0 and eth1) were bonded on this blade. After around 6 hours of poking around the problem I discover that despite presence of rule for "eth0-eth4 in
70-persistent-net.rules
file RHEL 6.7 still renames on boot all four interfaces to em0-em4 scheme and naturally bonding fails as eth0 and eth1 interfaces do not exit.
First I decided to deinstall biosdevname package and see what will happen. Did not work
(see below why -- de-installation script in this RPM is incorrect and contains a bug -- it is not enough to remove files
you also need to run
the command update-initramfs -u (Hat tip to Oler).
Searching for "Renaming em to eth" I found a post in which the author recommended disabling this "feature" via adding biosdevname=0 to kernel parameters
/etc/grub.conf
That worked. So two days my life were lost for finding a way to disable this completely unnecessary for blades RHEL "enhancement".
Here is some information about this package
biosdevname
Copyright (c) 2006, 2007 Dell, Inc.
Licensed under the GNU General Public License, Version 2.
biosdevname in its simplest form takes a kernel device name as an argument, and returns the BIOS-given name it "should" be. This is
necessary on systems where the BIOS name for a given device (e.g. the label on the chassis is "Gb1") doesn't map directly
and obviously to the kernel name (e.g. eth0).
The distro-patches/sles10/ directory contains a patch needed to integrate biosdevname into the SLES10 udev ethernet
naming rules.This also works as a straight udev rule. On RHEL4, that looks like:
KERNEL=="eth*", ACTION=="add", PROGRAM="/sbin/biosdevname -i %k", NAME="%c"
This makes use of various BIOS-provided tables:
PCI Confuration Space
PCI IRQ Routing Table ($PIR)
PCMCIA Card Information Structure
SMBIOS 2.6 Type 9, Type 41, and HP OEM-specific types
therefore it's likely that this will only work well on architectures that provide such information in their BIOS.
To add insult to injury this behaviour was demonstrated on only one of 16 absolutely identically configured Dell M630 blades with
identical hardware and absolutely identical (cloned) OS instances. Which makes RHEL kind of "Alice in Wonderland" system. After this experience it is difficult not to hate Red Hat, but
we can do very little to change this situation to the better.
This is just one example. I have more similar stories to
tell
I would like to stress that the fact that the utility that is included in this package does not understand that it the
target for installation is a blade ( there is no etching on blade network interfaces ;-) or server is a pretty typical RHEL
behaviour, despite the fact that the package was developed by Dell. They also install audio packages on boxes that have no audio
card and do a lot of similar things ;-)
If you look at this topic using your favorite search engine (which should be Google anymore ;-) you will find dozens of posts in
which people try to resolve this problem with various levels of competency and success. Such a tremendous waist of time and efforts.
Among best that I have found were:
Softpanorama Recommended
Among best links that I have found were:
Society
Groupthink :
Two Party System
as Polyarchy :
Corruption of Regulators :
Bureaucracies :
Understanding Micromanagers
and Control Freaks : Toxic Managers :
Harvard Mafia :
Diplomatic Communication
: Surviving a Bad Performance
Review : Insufficient Retirement Funds as
Immanent Problem of Neoliberal Regime : PseudoScience :
Who Rules America :
Neoliberalism
: The Iron
Law of Oligarchy :
Libertarian Philosophy
Quotes
War and Peace
: Skeptical
Finance : John
Kenneth Galbraith :Talleyrand :
Oscar Wilde :
Otto Von Bismarck :
Keynes :
George Carlin :
Skeptics :
Propaganda : SE
quotes : Language Design and Programming Quotes :
Random IT-related quotes :
Somerset Maugham :
Marcus Aurelius :
Kurt Vonnegut :
Eric Hoffer :
Winston Churchill :
Napoleon Bonaparte :
Ambrose Bierce :
Bernard Shaw :
Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient
markets hypothesis :
Political Skeptic Bulletin, 2013 :
Unemployment Bulletin, 2010 :
Vol 23, No.10
(October, 2011) An observation about corporate security departments :
Slightly Skeptical Euromaydan Chronicles, June 2014 :
Greenspan legacy bulletin, 2008 :
Vol 25, No.10 (October, 2013) Cryptolocker Trojan
(Win32/Crilock.A) :
Vol 25, No.08 (August, 2013) Cloud providers
as intelligence collection hubs :
Financial Humor Bulletin, 2010 :
Inequality Bulletin, 2009 :
Financial Humor Bulletin, 2008 :
Copyleft Problems
Bulletin, 2004 :
Financial Humor Bulletin, 2011 :
Energy Bulletin, 2010 :
Malware Protection Bulletin, 2010 : Vol 26,
No.1 (January, 2013) Object-Oriented Cult :
Political Skeptic Bulletin, 2011 :
Vol 23, No.11 (November, 2011) Softpanorama classification
of sysadmin horror stories : Vol 25, No.05
(May, 2013) Corporate bullshit as a communication method :
Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000):
the triumph of the US computer engineering :
Donald Knuth : TAoCP
and its Influence of Computer Science : Richard Stallman
: Linus Torvalds :
Larry Wall :
John K. Ousterhout :
CTSS : Multix OS Unix
History : Unix shell history :
VI editor :
History of pipes concept :
Solaris : MS DOS
: Programming Languages History :
PL/1 : Simula 67 :
C :
History of GCC development :
Scripting Languages :
Perl history :
OS History : Mail :
DNS : SSH
: CPU Instruction Sets :
SPARC systems 1987-2006 :
Norton Commander :
Norton Utilities :
Norton Ghost :
Frontpage history :
Malware Defense History :
GNU Screen :
OSS early history
Classic books:
The Peter
Principle : Parkinson
Law : 1984 :
The Mythical Man-Month :
How to Solve It by George Polya :
The Art of Computer Programming :
The Elements of Programming Style :
The Unix Hater’s Handbook :
The Jargon file :
The True Believer :
Programming Pearls :
The Good Soldier Svejk :
The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society :
Ten Commandments
of the IT Slackers Society : Computer Humor Collection
: BSD Logo Story :
The Cuckoo's Egg :
IT Slang : C++ Humor
: ARE YOU A BBS ADDICT? :
The Perl Purity Test :
Object oriented programmers of all nations
: Financial Humor :
Financial Humor Bulletin,
2008 : Financial
Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related
Humor : Programming Language Humor :
Goldman Sachs related humor :
Greenspan humor : C Humor :
Scripting Humor :
Real Programmers Humor :
Web Humor : GPL-related Humor
: OFM Humor :
Politically Incorrect Humor :
IDS Humor :
"Linux Sucks" Humor : Russian
Musical Humor : Best Russian Programmer
Humor : Microsoft plans to buy Catholic Church
: Richard Stallman Related Humor :
Admin Humor : Perl-related
Humor : Linus Torvalds Related
humor : PseudoScience Related Humor :
Networking Humor :
Shell Humor :
Financial Humor Bulletin,
2011 : Financial
Humor Bulletin, 2012 :
Financial Humor Bulletin,
2013 : Java Humor : Software
Engineering Humor : Sun Solaris Related Humor :
Education Humor : IBM
Humor : Assembler-related Humor :
VIM Humor : Computer
Viruses Humor : Bright tomorrow is rescheduled
to a day after tomorrow : Classic Computer
Humor
The Last but not Least Technology is dominated by
two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt.
Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org
was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP)
without any remuneration. This document is an industrial compilation designed and created exclusively
for educational use and is distributed under the Softpanorama Content License.
Original materials copyright belong
to respective owners. Quotes are made for educational purposes only
in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains
copyrighted material the use of which has not always been specifically
authorized by the copyright owner. We are making such material available
to advance understanding of computer science, IT technology, economic, scientific, and social
issues. We believe this constitutes a 'fair use' of any such
copyrighted material as provided by section 107 of the US Copyright Law according to which
such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free)
site written by people for whom English is not a native language. Grammar and spelling errors should
be expected. The site contain some broken links as it develops like a living tree...
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or
referenced source) and are
not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness
of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be
tracked by Google please disable Javascript for this site. This site is perfectly usable without
Javascript.
Last modified: March, 12, 2019
