Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

iLO 3 -- HP engineering fiasco

News HP Servers Recommended Links Command line interface ILO3 TCP ports Firmware upgrade
Initial setup Fiasco: some additional information Fiasco: some preliminary recommendations Rebooting the iLO "Remote Console - in use" Problem Serial console
Security

Using ILO virtual CDROM

 

 Using ILO virtual floppy drive for holding YAST autoyast installation file  How do I reset the HP iLO IP address and password Dell DRAC
Sysadmin Horror Stories Administration of Remote Servers  HP Smart Array P410 controller Humor Random Findings Etc

Introduction

The defining feature of ILO3 in versions up to and including 1.26 is its instability. It is unsafe to use with any product, but especially with large HP servers and blades. The first usable version of ILO 3 is 1.85 (CP026424.scexe) released May 13, 2015, long after server of V7 series became obsolete.

If ILO hangs and lose network connectivity for no obvious reasons, there is no reset button for ILO: you need to power server down, disconnect cables (or remove the blade) and power it up again. Sometimes the console can't connect tot he server/blade and only button and lights icon at the bottom of the screen are active. Sometimes it also loses time/date.

Blades are the most annoying case as there are usually way too many of them. But ILO freezing in remote production rack servers is nothing short of HP-made disaster. You need to shutting down, remove power cables and then reboot server such as DL580 or DL 380 to restore ILO functionality. Imagine the case when a remote site has no night shift personnel or there is no qualified personnel on the site at all and situation does not look too promising. In any case few would argue that HP managed to invent  a very interesting engineering solution for the problem of fixing a crash or freezing of a small supplementary computer by shutting down the main production server and removing power cable. My sincere congratulations to HP team. You are the best  ;-)

Notes:

Instability of a new version of a complex product (although overcomplexity of ILO is a blunder and has no excuse)  is nothing new, but what make this problem a real disaster is absence of any convenient means of rebooting ILO without removing power cords from the box. And this is a real engineering fiasco. After three of four such operations with DL580 or with multiple blades at night you start passionately hate HP and its server products :-)

Instability of a new version of a complex product (although overcomplexity of ILO is a blunder and has no excuse)  is nothing new, but what make this problem a real disaster is absence of any convenient means of rebooting ILO without removing power cords from the box. And this is a real engineering fiasco. After three of four such operations with DL580 or with multiple blades at night you start passionately hate HP and its server products :-)

Note: Version 1.65 was released November 26, 2013. It has better stability and I did not experienced this freezing situation with it yet. 

Documentation looks like a translation  from some foreign language into English ;-)

Seasoned sysadmin do not expect from vendor documentation much. We are trained to work with manuals written by people who do not understand the product and do not care, but in this area HP still stands out.

For example HP ProLiant Integrated Lights-Out 3 v 1.20 User Guide (HP part number 616301-003) in places looks like translation from some other language to English, which is an innovative trend, I would say a real breakthrough,  for a large US company. As a result some quotes can well be submitted to the Onion (p. 85)

To use a physical CD/DVD-ROM drive in your client PC:
1. Select IRC within the Remote Console section. --[There is no IRC section in Remote control section]
2. Select the Virtual Drive tab --[There is no Virtual tab section]

and then select the drive letter of the desired physical CD/DVD-ROM --[There is no drop-menu with drive letters to select from] drive on your client PC from the drop-down menu.

The trick to understand this paragraph is that they are talking about a tab named "Virtual drive" on the remote console screen, not about Remote console/Remote console section of ILO3 main menu :-). Please note that there is no IRC tab on remote console screen either, only "Virtual drive" tab.

Another interesting tidbit is that ILO timeouts no matter what setting you put at Administration/Access/settings. So it takes your current setting for advisory and then does not follow up :-). If you installing OS on a remote server that might have an interesting effect on your mood as logout persistently resets virtual CD/DVD and virtual floppy drive.  I agree that it provide an excellent  security of this feature in a sense of making it impossible to use.  As we all agree that security is important, that well might be implicit HP design goal, although just dropping this feature would be an equally secure and less frustrating solution. .

Additional cost of remote console

HP servers are more expensive then servers of competitors, such as Dell. So it is naturally to expect from them more. And HP does include ILO in all of its servers by default; so it's cost is implicitly reflected in the price if, for example you compare the cost of HP server and equivalent Dell server.

But at some point HP brass became way too greedy and decided to "correct" this situation. The method they found is pretty interesting.

Unlike Dell DRAC, remote console capabilities are not free. HP demands additional license fee (around $300 for license with one year support) for the ability to use remote console with iLO 3. In view of those "disconnects" that ILO 3 suffer from,  I think HP need to pay users instead until they fix the problem :-).

In any case the price structure HP adopted is somewhat questionable. IMHO there should not be software license as this is a specialized appliance, not a general purpose device. To charge $300 for advanced license which nothing more than the ability to use remote console (based on VNC) does not look like a sophisticated business model.

For example Dell charges for hardware (Drac card) which is more logical: $99 for iDRAC6 express hardware, $349 hardware with enterprise version installed and $448 for version with 8GB SD card (this is 2011 prices without corporate discount).  As we see the price of the device is suspiciously close to the price of the advanced license.

And DRAC actually provides some functionality that ILO currently does not have as the flash card size is enough for putting DVD image into it and that allow you to avoid a slow connection problems when booting the server from the image. DRAC the ability to send SMTP alerts. And the ability to send email alerts directly to sysadmin has great value in a typical enterprise environment if we take into account the amount of red type necessary for accomplishing the same via monitoring system such as HP Operations Manager.

Unfortunately it looks like Dell with DRAC 7 got into the same tar pit as HP. It also hangs.  

ILO virtual DVD capabilities

They are really useful for booking an OS.  But ILO virtual DVD capabilities are not very stable due to timeouts and usable mostly for booting installer (after that you can get ISO image from a nearby server via FTP or HTTP).

Beware of using 60 days free advanced ILO license

This is another funny (or not so funny) idiosyncrasy: you can get 60 days free advanced ILO license from HP, but the problem is that when you buy a permanent license, unless this is a license for multiple servers you can't replace temporary license with permanent (at least as of version 1.25; I know because I tried... ).

Over-engineered means unsecure

It looks like ILO3 is over-engineered and those who design and wrote the code have no or little clue how this hardware device is used in real world.  That means that it represents one huge security hole, a hidden gateway to your servers. In additional to those that I mentioned there are known issues, related to certificate handling.

Judging from the experience with using ILO 3 I have little doubt that there are multiple exploits for each version of the device. That makes it a very attractive target for state-supported hackers as in most corporation ILO are not well protected and this vector of attack is typically overlooked.

From the security standpoint ILO 3 represents a hidden backdoor to your server for state-supported hackers. Nothing more, nothing less.

As such it should be secured by using a separate protected by firewall segment. For remote installations usage of VPN should be considered.  Which of course means additional cost and complexity.

But please don't overlook this vector of attack. Due to overcomplexity of the codebase, state-supported hackers can breach built-in security as easy as  knife enters the butter.  If you need to protect corporate assets from this type of hackers in no way you can rely on built-in security features.

Instability Fiasco: some additional information

Versions up to 1.26 demonstrates is a very interesting type of instability. Often if you work with ILO long enough it logs you out. Or you can be logged out due to inactivity, and in early versions (probably up to 1.28) change of inactivity period was disregarded.

After that often you can't log-in again no matter how hard you try without shutting down the server, disconnecting power from the server and rebooting it. In other words under some mysterious circumstances such a log out also frequently switches ILO to a state in which it is not possible to connect to ILO's network port. It remains to visible to nmap:  

Starting Nmap 4.00 ( http://www.insecure.org/nmap/ ) at 2011-10-11 12:07 EDT
Interesting ports on ilo_host.somewhere.com (10.194.181.27):
(The 1669 ports scanned but not shown below are in state: closed)
PORT    STATE    SERVICE
22/tcp  filtered ssh
80/tcp  filtered http
443/tcp filtered https
Inability to connect to the iLO is unacceptable for the product the sole role of which provide ability to control server when something went wrong. This is fiasco or SNAFU in army terms.

Up to version 1.28 the most typical scenario of getting into this state is to let ILO login to timeout and close browser session with it. Here is the trace of what is happening:

# tcpdump -vv dst host 10.194.181.27 or src host 10.194.181.27
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
17:57:34.547496 IP (tos 0x0, ttl  64, id 20690, offset 0, flags [DF], proto: TCP (6), length: 52) ssh_client_host.somecompany.com.38178 > ilo_host.somecompany.com.ssh: F, cksum 0x95d2 (correct), 1380736172:1380736172(0) ack 2766974886 win 12 
17:57:34.554772 IP (tos 0x0, ttl  59, id 38653, offset 0, flags [DF], proto: TCP (6), length: 52) ilo_host.somecompany.com.ssh > ssh_client_host.somecompany.com.38178: ., cksum 0xcb96 (correct), 1:1(0) ack 1 win 33303 
17:57:35.881080 IP (tos 0x0, ttl  64, id 5855, offset 0, flags [DF], proto: TCP (6), length: 60) ssh_client_host.somecompany.com.60838 > ilo_host.somecompany.com.ssh: S, cksum 0x6930 (correct), 1413089547:1413089547(0) win 5840 
17:57:35.888571 IP (tos 0x0, ttl  59, id 38654, offset 0, flags [DF], proto: TCP (6), length: 64) ilo_host.somecompany.com.ssh > ssh_client_host.somecompany.com.60838: S, cksum 0xc490 (correct), 2632478857:2632478857(0) ack 1413089548 win 33304 
17:57:35.888580 IP (tos 0x0, ttl  64, id 5856, offset 0, flags [DF], proto: TCP (6), length: 52) ssh_client_host.somecompany.com.60838 > ilo_host.somecompany.com.ssh: ., cksum 0x8766 (correct), 1:1(0) ack 1 win 12

As we mentioned before, the problem with such behavior is that if this is a remote installation with no local support personnel you are royally f*@$ed ;-).

Note: for HP blades ILO 3 was and still is (as of March 2012 or version 1.28) a disaster. It is unstable and often lock you out if you forget to logoff and timeout kicks in.

Resolution of HO ILO3 fiasco: some preliminary recommendations

Here are some preliminary recommendations:

  1. The first really usable version of  ILO 3 is 1.85 (CP026424.scexe) released May 13, 2015, long after server of V7 series became obsolete. All versions before it including 1.82 (released Jan, 2015) are crap. As of September 2915 this firmware remains a free download from HP site and does not depend whether you have active support contract with HP or not (Bios firmware now requires contract)  

     

  2. Version 1.65 of firmware os another version that is more or less OK (see Firmware upgrade for details). It looks like it fixes possible random server shutdowns problem.

    Looks like this upgrade solves the problem of instability (actually all versions starting from 1.28 were an improvement in comparison with really horrible previous versions).  

    Still on blades you usually can't install it from the SCEXE file. HP recommendation

    To update firmware from the Linux operating system on target server:
    Download the SCEXE file to the target server.

    is bogus. Installation freezes with message "

    Flashing is underway... 1 percent programmed.
    but it does not destroy the flash ROM. You can still reboot ILO and do it via remote interface
    Never try to execute recommended by HP command sh CP016203.scexe on blades. It might fail and then what?
    I would like to stress it again that the message:
    Flashing is underway... 1 percent programmed.

    can be displayed for a long time and then the process timeout:

    FLASH_iLO3 v1.08 for Linux (May  8 2012)
(C) Copyright 2002-2012 Hewlett-Packard Development Company, L.P.
Firmware image: ilo3_150.bin
Current iLO 3 firmware version  1.26; Serial number ILOUSE0504TPM

Component XML filel
CP016203.xml reports firmware version 1.50
This operation will update the firmware on the
iLO 3 in this server with version 1.50.
Continue (y/N)?y
Current firmware is 1.26 (Aug 26 2011 )
Firmware image is 0x801664(8394340) bytes
Committing to flash part...
******** DO NOT INTERRUPT! ********
Flashing is underway...   1 percent programmed. -

    For blades the only reliable way to upgrade firmware is to download the SCEXE file to a client running a Linux operating system. Execute: 

    sh CP016203.scexe --unpack=directory.

    This command will unpack the iLO3 bin into a user specified "directory". If the directory does not exist, the unpacker will attempt to create it. Then move .bin file to your windows client and do upgrade from ILO Web interface. See Uploading .bin file via ILO3 Web interface

    For blades the only reliable way is Download the SCEXE file to a client running a Linux operating system. Execute: 
    sh CP016203.scexe --unpack=directory.

    This command will unpack the iLO3 bin into a user specified "directory". If the directory does not exist, the unpacker will attempt to create it. Then move .bin file to your windows client and do upgrade from ILO Web interface. See Uploading .bin file via ILO3 Web interface
  3. On the first login create at least one other user account in addition to Administrator account. Multiple account make the "sudden freeze" problem less frequent in versions up to 1.26. The highest frequency observed is when there is only one account (Administrator).
     
  4. In Administration/Access setting screen set "Idle connection Timeout drop-down list to "Indefinite". Go to the Administration/Network tab and click apply. I think this setting is activated only on reboot of ILO, while new user setting does not require reboot. This is really Byzantine peace of equipment.
     
  5. Logout explicitly as you finished using ILO.

Initial setup

  1. Restart or power the server on (Important: only removing power from power supplies resets ILO).

  2. Press the F8 key when ILO prompt appears during POST.

  3. Select Network>DNS/DHCP, press the Enter key, and then select DHCP Enable. Press the spacebar to turn off DHCP. Be sure that DHCP Enable is set to Off, and save the changes.

  4. Select Network>NIC>TCP/IP, press the Enter key, and enter the appropriate information in the IP Address, Subnet Mask, and Gateway IP Address fields.

  5. Save the changes.

  6. Additional recommended settings:

Applying firmware upgrade

In the first half of 2011 HP released iLO firmware updates one each second month or so. Later it became more rare but still it creates significant load on the sysadmins, as if they have nothing to do but upgrade produced by HP junk each second month.  Such cavalier attitude toward important part of their server undermines any trust in HP engineering abilities.

You can use iLO Online updates for Windows and linux. Packages have different extensions (.exe for Windows and .scexe for Linux). Update can be applied in two ways:

Since web based firmware update supports only the firmware image file (.BIN file) it is important not to forget to extract the .BIN file from the firmware package first. The .BIN firmware update file is not available as a direct download option at HP.com

Steps

  1. Download iLO online firmware update package
  2. Extract the .BIN file from it. For Linux .scexe file can be extracted using command:
    sh *.scexe --unpack=directory
    For example 
    sh CP015458.scexe --unpack=directory.
  3. Load .bin file it into ILO using ILO browser

iLO Firmware update from USB drive

In case you screw things up you can use HP Smart Update Firmware DVD of the server to update the iLO firmware. To use HP Smart Update Manager on the Firmware Maintenance CD:

Rebooting the iLO

The blunders HP committed would be less biting if there were a possibility to reboot ILO separately from server using some hardware button. But there no such capability for stand-alone HP servers. Please note that there is such capability for blades not not for ILO alone, but together with the blade (see above).

HP  communities this is a very interesting way:

       You can reboot ILO

1) By removing all the power cables from the server.

2) By changing the system maintenance switch 1 on the system board (DL 580). That's convoluted operation:

Power the server OFF.
Disconnect the power cord from the server.
Remove the access panel.
Remove the controller and the riser board.
Push and hold the power button down for about a minute.
Remove and reseat all the memory DIMMS.
Disconnect and reconnect the VGA cable from the server.
Set the System maintenance switch 1 to the ON position.
Power the server back ON.
After the server has completed the Power-On Self Test, power the server OFF (If display is present).
Set the System maintenance switch 1 back to the OFF position.
Power the server back ON.

"Remote Console - in use" Problem

This is another annoying problem. In short sometime you can't connect to remote console getting the message "Remote Console - in use". But it is less severe then previous in recent versions as remote interface is still functional and you can reboot ILO from it:

See also How to reset iLO 2 (HP Integrated Lights-Out) of HP Proliant or ...

New features in iLO 3

Here is some info from readme for version 1.0:

The following features are new with iLO 3 v 1.00:

Faster performance:

Streamlined interface:

  • Enhanced standards support:

    Obsolete features

    • Top Visited
    		 Switchboard
    Latest
    Past week
    Past month

    NEWS CONTENTS

    Old News ;-)

    [Dec 12, 2013] Advisory HP Integrated Lights-Out 3 and HP Integrated Lights-Out 4 - False CPU Overheating Messages May Be Logged

    Any HP ProLiant G7 and Gen8 series servers configured with Intel processors; iLO 3 Firmware Version 1.61 (or earlier) and iLO 4 Firmware Version 1.30 (or earlier).

    RESOLUTION

    To prevent the false messages from occurring, update the iLO firmware as follows:

    Update the iLO 3 firmware to version 1.65 (or later):

    Perform the following steps to obtain the latest iLO 3 firmware version:

    1. Click on the following link:

      http://www.hp.com/support/iLO3

    2. Select the appropriate operating system.
    3. Click on "Firmware - Lights-Out Management."
    4. Locate, download, and install iLO 3 Firmware Version 1.65 (or later).

    Update the iLO 4 firmware to version 1.32 (or later):

    Perform the following steps to obtain the latest iLO 4 firmware version:

    1. Click on the following link:

      http://www.hp.com/support/iLO4

    2. Select the appropriate operating system.
    3. Click on "Firmware - Lights-Out Management."
    4. Locate, download, and install iLO 4 Firmware Version 1.32 (or later).

    Firmware update problem during remote update

    HP Smart Update Manager (HP SUM) - HP Smart Update Manager (SUM) 5.x - Unable to Update HP ProLiant iLO3 Firmware from 1.2x to 1.5.

    If using HP SUM bundled in the HP Support Pack for ProLiant (SPP) in order to update the HP ProLiant iLO3 firmware from 1.2x to 1.50 remotely, the update will fail.

    The installation will report as failed, and the HP SUM error log will show the following:

    ====== HP Smart Update Manager Installation Log ====== Starting ILO firmware update. Initializing connection to ILO 192.168.10.1. Connection established to ILO. Checking the type of ILO. load -source http://192.168.10.20:63000/Rep1/cp016202/3.bin status=1 status_tag=COMMAND SPAWNED Mon Jan 14 11:33:27 2013 Attempting to load iLO3 firmware image. status=2 status_tag=COMMAND PROCESSING FAILED error_tag=COMMAND ERROR-UNSPECIFIED Mon Jan 14 11:33:27 2013 Unable to retrieve a valid iLO3 firmware image. Check file path and login credentials.

    This issue is due to a bug in the iLO3 firmware.

    HP SUM executes the load -source command from iLO3 SSH console, but when the iLO3 web pooler proceeds to download the firmware file, it will fail.

    Therefore, it is an iLO3 issue and not with HP SUM.

    NOTE: This issue does not happen if running HP SUM locally, only when trying remote updates.

    Solution

    This issue does not occur in 1.1x or 1.50 and later firmware. llpadding="0" width="100%">
    NOTE: In order to update iLO3 to 1.50, the iLO3 firmware must be in the 1.2x versions.
    As a workaround, use the following options:
    1. Run the iLO3 firmware update locally on the server. The firmware will be transferred via the iLO3 driver.
    2. If need to update several systems remotely, use the iLO RIBCL XML scripts. Use the script Update_Firmware.xml as a template.

    HP Integrated Lights-Out 3 Firmware upgrade version 1.5

    Type: Firmware - Lights-Out Management
    Version: 1.50 (26 Oct 2012)
    Operating System(s): Red Hat Enterprise Linux 5 Desktop (x86), Red Hat Enterprise Linux 5 Desktop (x86-64), Red Hat Enterprise Linux 5 Server (x86), Red Hat Enterprise Linux 5 Server (x86-64), Red Hat Enterprise Linux 6 Server (x86), Red Hat Enterprise Linux 6 Server (x86-64), SUSE Linux Enterprise Server 10 (AMD64/EM64T), SUSE Linux Enterprise Server 10 (x86), SUSE Linux Enterprise Server 11 (AMD64/EM64T), SUSE Linux Enterprise Server 11 (x86), VMware ESX/ESXi 4.0, VMware ESX/ESXi Server 3.5

    The Command Line Interface (CLI) Command to Create a New User May Not Function in iLO 3 Firmware Version 1.26 or Version 1.28

    SUPPORT COMMUNICATION - CUSTOMER ADVISORY
    Document ID: c03573824

    Version: 1

    Advisory: HP Integrated Lights-Out 3 (iLO 3) - The Command Line Interface (CLI) Command to Create a New User May Not Function in iLO 3 Firmware Version 1.26 or Version 1.28
    NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.
    Release Date: 2012-11-14

    Last Updated: 2012-11-14

    [ Dec 20, 2012 ] Advisory (Revision) HP Integrated Lights-Out 3 (iLO 3) - The iLO 3 ROM-Based Setup Utility May Incorrectly Display the Gateway

    Solution is to upgrade to version 1.5 (October 2012)
    12/10/2012

    After upgrading HP Integrated Lights-Out 3 (iLO 3) Firmware to Version 1.28 and using the iLO 3 ROM-Based Setup Utility (RBSU) to configure a static IP address using the following steps, the next time that the iLO 3 RBSU Setup Utility is loaded, the iLO 3 Gateway IP address may be incorrectly displayed as 0.0.0.0. This is a "display only" issue in iLO 3 Firmware Version 1.28. No other iLO 3 Firmware versions are affected.

    The condition occurs after performing the following steps:

    Restart or power on the server. Press the F8 key when prompted during POST. The iLO 3 RBSU runs. Select Network > DNS/DHCP, press the ENTER key, and then select DHCP Enable . Press the spacebar to turn off DHCP. Be sure that DHCP Enable is set to Off, and save the changes. Select Network > NIC > TCP/IP, press the ENTER key, and enter the appropriate information in the IP Address, Subnet Mask, and Gateway IP Address fields. Save the changes. Exit iLO 3 RBSU. The change takes effect after exiting iLO 3 RBSU. SCOPE Any ProLiant server configuration using HP Integrated Lights-Out 3 (iLO 3) Firmware Version 1.28.

    RESOLUTION This is a display only issue and can be safely ignored.

    HP Integrated Lights-Out 3 (iLO 3) Firmware Version 1.50 (or later) corrects the Gateway IP Address display issue. To access the HP Integrated Lights-Out 3 (iLO 3) driver and software download page, click on the following URL:

    http://www.hp.com/support/iLO3

    OR

    As a workaround, use the iLO 3 web GUI to verify that the Gateway IP address is displaying correctly on the network. The network settings can be checked by looking under Administrator -> Network.

    [Jun 25, 2012] If an HP ProLiant Server Resets Unexpectedly, the Integrated Management Log (IML) Should Be Check

    Document ID: c03370645

    Version: 1

    Notice: HP ProLiant Servers - If an HP ProLiant Server Resets Unexpectedly, the Integrated Management Log (IML) Should Be Checked to Determine if the Source of the Reset Is Indicated

    NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

    Release Date: 2012-06-12

    Last Updated: 2012-06-12

    DESCRIPTION

    If an HP ProLiant server resets unexpectedly, the Integrated Management Log (IML) should be checked to determine if the source of the reset is indicated. This information can help determine if any action needs to be taken.

    The Integrated Management Log (IML) records hardware events and stores them in a formatted table. It records the time of the event and categorizes events in severity levels such as:

    DETAILS

    There are numerous conditions that can result in a server unexpectedly resetting. Whenever such a reset occurs, the Integrated Management Log (IML) should be checked to determine if the cause of the reset is indicated. While the IML will not always indicate the cause of an unexpected reset, it will in most cases.

    The IML can be viewed from the HP Integrated Lights-Out (iLO) web page or using other HP Management Tools. Some of the most commonly used methods are:

    IML Viewer:

    1. Click Start->All Programs -> HP System Tools-> HP ProLiant Integrated Management Log Viewer.
    2. The IML Viewer will open and the IML logs will be displayed on the screen.

    From HP Integrated Lights-Out (iLO):

    1. Open the iLO web interface.
    2. Enter the iLO Login name and Password.
    3. From the left menu, access IML Logs .

    From the System Management Homepage (SMH):

    1. Click Start-> All Programs -> HP Management Agents -> HP System Management Homepage.
    2. Click on Logs . The log screen will be displayed.
    3. Click on Integrated Management Log .

    [Jun 01, 2012] Systematic failure during installation of version 1.29 on HP blades

    Only blades are affected. Standalone servers like , DL360, DL380, DL580 are OK. 
    # ./CP016462_ILO3_1_28.scexe

FLASH_iLO3 v1.06 for Linux (Jan 17 2012)
(C) Copyright 2002-2011 Hewlett-Packard Development Company, L.P.
Firmware image: ilo3_128.bin
Current iLO 3 firmware version  1.26; Serial number ILOUSE0504TPN

Component XML file: CP016462.xml
CP016462.xml reports firmware version 1.28
This operation will update the firmware on the
iLO 3 in this server with version 1.28.
Continue (y/N)?y
Current firmware is 1.26 (Aug 26 2011 )
Firmware image is 0x800000(8388608) bytes
Committing to flash part...
******** DO NOT INTERRUPT! ********
/
Channel Interface transactions (Linux) returns 21!
Channel Interface call status: FIFO empty.
ERROR: received errcode 21
/
Channel Interface transactions (Linux) returns 21!
Channel Interface call status: FIFO empty.

Failed(5-21)!

ERROR: Unable to commit flash. [ilo3_128.bin]

    [May 15, 2012] HP Integrated Lights-Out 3 (iLO 3) Version 1.25 (or Earlier) - iLO Virtual Media Service May Randomly Stop Responding When Attempting to Connect a Virtual Device in Remote Console

    c03316654

    Products: HP ProLiant DL Servers, HP ProLiant BL Server Blades, HP ProLiant Scalable Systems, Insight Control Software, HP ProLiant ML Servers

    Description: iLO Virtual Media Service May Randomly Stop Responding When Attempting to Connect a Virtual Device in Remote Console

    [May 15, 2012] Disabling an iLO 3 NIC May Cause the iLO 3 Link Status to be Incorrectly Displayed as Failed in HP System Management Homepage and HP Systems Insight Manager (HP SIM)

    c03327392

    Products: Software, ProLiant Servers, BladeSystem

    Description: Advisory: HP Integrated Lights-Out 3 (iLO 3) - Disabling an iLO 3 NIC May Cause the iLO 3 Link Status to be Incorrectly Displayed as Failed in HP System Management Homepage and HP Systems Insight Manager (HP SIM)

    [May 15, 2012] iLO 3 Firmware Flash Progress May Intermittently Stop at One Percent or Take Sev

    HP

    Document ID: c03318935

    Version: 1

    Advisory: HP Integrated Lights-Out 3 (iLO 3) - iLO 3 Firmware Flash Progress May Intermittently Stop at One Percent or Take Several Hours to Complete

    NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

    Release Date: 2012-05-07

    Last Updated: 2012-05-07

    DESCRIPTION

    While upgrading the HP Integrated Lights-Out 3 (iLO 3) from any previous version prior to version 1.28 (or later), the firmware flash progress may intermittently stop at one percent or take several hours to complete. This occurs due to idle Secure Shell (SSH) connections that are incorrectly closed and then time-out, causing the iLO 3 CPU to become busy and stall or delay the firmware flash progress.

    An error in the way SSH sessions are shut down causes iLO 3 to assume that the sessions are fully active and constantly transferring data, delaying the firmware flash progress. HP Systems Insight Manager (HP SIM) servers and other SSH management tools located on the same network as iLO 3 may open these SSH connections.

    SCOPE

    Any HP ProLiant server with Integrated Lights-Out 3 (iLO 3) Firmware Version 1.26 (or earlier).

    RESOLUTION

    This is resolved by upgrading Integrated Lights-Out 3 (iLO 3) to firmware version 1.28 (or later).

    Note : The issue may occur while upgrading to version 1.28 (or later) if upgrading from a version of the firmware prior to version 1.28.

    Until the firmware upgrade is installed, reset iLO 3 to clear the SSH connection state prior to flashing the firmware.

    If the firmware flash progress has stopped, connect several simultaneous SSH sessions until no additional SSH connections are allowed. Then close them normally to reset the connection state and allow the flash progress to continue.

    Perform the following steps to obtain the latest iLO 3 firmware version:

    1. Click on the following link:

      http://www.hp.com/support/iLO3

    2. Select the appropriate operating system.
    3. Click "Firmware - Lights-Out Management."
    4. Locate, download, and install iLO 3 Firmware Version 1.28 (or later).

    [May 07, 2012] ILO3 FW 1.26-1.28 Drive Information Shows Only Physical Driver Stat

    For ILO3 with firmware 1.26 and 1.28, web GUI and CLI does not show rebuilding status for Drive information .

    Web GUI and CLI show only OK or Not installed .

    [May 07, 2012] Integrated Lights-Out 3 (iLO 3) FIRMWARE UPGRADE REQUIRED to Prevent Unexpected Server

    IMPORTANT : The Integrated Lights-Out 3 (iLO 3) firmware upgrade provided in the Resolution is required to prevent unexpected shutdowns, false CPU clock throttled messages, or incorrect values displayed via the PPIC command. HP recommends performing this upgrade at the customer's earliest possible convenience. Neglecting to perform the recommended iLO 3 firmware upgrade could result in the potential for subsequent errors to occur.

    On an HP ProLiant SL390s G7 server, after upgrading the Integrated Lights-Out 3 (iLO 3) firmware to version 1.26, the server may experience unexpected shutdowns, false CPU clock throttled messages displayed in the operating system console, or incorrect values displayed via the ProLiant Power Interface Configuration (PPIC) command.

    SCOPE
    Any HP ProLiant SL390s G7 server (2U or 4U) after upgrading the Integrated Lights-Out 3 (iLO 3) firmware to version 1.26.

    RESOLUTION
    To prevent these issues from occurring, upgrade the Integrated Lights-Out 3 (iLO 3) to Firmware Version 1.28 (or later).

    [May 07, 2012] HP Integrated Lights-Out 3 Version 1.25 (or Earlier) Browser Interface and Secure Shell (SSH) May Stop Responding When Attempting to Login to SSH Using a Private SSH-DSA Key Larger Than 2048 Bits

    May 03 2012 | c03315526

    Products: HP ProLiant BL Server Blades, Insight Control Software

    Description: Advisory: HP BladeSystem ProLiant Server Blades - HP Integrated Lights-Out 3 Version 1.25 (or Earlier) Browser Interface and Secure Shell (SSH) May Stop Responding When Attempting to Login to SSH Using a Private SSH-DSA Key Larger Than 2048 Bits

    HP Integrated Lights-Out 3 (iLO 3) - HP Proliant G7 Blade servers - iLO 3 Firmware version 1.25 Server Power On Issue Caused by Overprovisioning of Power

    Resolution is to apply firmware version 1.26.

    HP Proliant G7 Blade servers might exibit the issue where these servers are not powering On by default any longer when they have HP iLO firmare 1.25 applied.

    The issue is that with iLO 3 firmware 1.25 some half height HP Proliant Blade servers get assigned 509 W of power and full height servers get 1800 W of power assigned. Sometimes, it is on an average of 60% more then what it should be.

    See the screenshot below for visual indicator...

    [May 07, 2012] Advisory HP ProLiant Servers - Integrated Lights-Out 3 and 4 (iLO 3-iLO 4) User Guides Incorrectly State That New Certificate Is Always Created When iLO Is Reset

    2012-04-30

    The HP Integrated Lights-Out 3 and 4 (iLO 3/iLO 4) User Guides incorrectly state that a new certificate is always created each time that iLO is reset (Diagnostics -> reset on iLO). For example, the iLO 3 Version 1.20 User Guide states the following on page 166:

    If the iLO 3 self-signed certificate is installed permanently into some browsers and the iLO 3 is reset, you might not be able to log back in to iLO 3 because iLO 3 generates a new self-signed certificate every time it is reset.

    HP ProLiant Integrated Lights-Out 3 Version 1.20 User Guide
    http://h20000.www2.hp.com/bc/docs/support/SupportManual/c02774507/c02774507.pdf

    Any HP ProLiant server with Integrated Lights-Out 3 or 4 (iLO 3/iLO 4)

    [Apr 26, 2012] HP Lights-Out XML PERL Scripting Sample for Linux (ver 4.00.0)

    Products: BladeSystem, Options and Accessories, ProLiant Servers, Software

    OS: Other Legacy OS,Linux

    Description: This is a set of sample XML scripts used to manage the configuration of Integrated Lights-Out (iLO) management processors and to control servers in which iLO devices are in use. Use LOCFG.PL (available in this bundle) to run the XML.

    [Apr 16, 2012] Advisory: HP Integrated Lights-Out 3 (iLO 3) - The iLO 3 ROM-Based Setup Utility May Incorrectly Display the Gateway IP Address as 0.0.0.0 After Upgrading iLO 3 to Firmware Version 1.28 (c03283274)

    Products: HP ProLiant DL Servers, HP ProLiant BL Server Blades, HP ProLiant ML Servers, HP ProLiant Scalable Systems, Insight Control Software

    Description: Advisory: HP Integrated Lights-Out 3 (iLO 3) - The iLO 3 ROM-Based Setup Utility May Incorrectly Display the Gateway IP Address as 0.0.0.0 After Upgrading iLO 3 to Firmware Version 1.28

    Advisory ProLiant Servers Integrated Lights-Out 3 - FIRMWARE UPGRADE RECOMMENDED to Avoid Java and Internet Explorer Unrespon

    Sept 12, 2011

    IMPORTANT : The Integrated Lights-Out 3 firmware upgrade provided in the Resolution is recommended to prevent Java and Internet Explorer from becoming unresponsive when attempting to acquire an already open Integrated Lights-Out 3 Remote Console session by opening the Java Remote Console applet. HP recommends performing this upgrade at the customer's earliest possible convenience. Neglecting to perform the recommended action and not performing the recommended resolution could result in the potential for subsequent errors to occur.

    Attempting to acquire an already open Integrated Lights-Out 3 (iLO 3) Remote Console session by opening the Java Remote Console applet may cause Java and Internet Explorer to stop responding when running iLO 3 firmware version 1.20 (or earlier).

    Any ProLiant server with Integrated Lights-Out 3 Version 1.20 (or earlier).

    [Jan 02, 2012] Connecting to HP iLO using the command line by Emerson Takahashi

    July 17, 2010 | setaOffice

    Just ssh to the IP that you configured the iLO

    emerson@shellcore:~ $ ssh [email protected]
    [email protected]′s password:
    User:Emerson Takahashi logged-in to ILO_TESTLABHP.(192.168.50.118)
    iLO 2 Advanced Evaluation 1.81 at 11:05:47 Jan 15 2010
    Server Name: proliant_g5
    Server Power: On

    hpiLO->

    Since I will access through a text terminal, Linux is configured to use the serial port (configured through the file /boot/grub/grub.conf)

    hpiLO-> help
    status=0
    status_tag=COMMAND COMPLETED

    DMTF SMASH CLP Commands:

    help : Used to get context sensitive help.
    show : Used to show values of a property or contents of a collection target.
    create : Used to create new user account in the name space of the MAP.
    Example: create /map1/accounts1 username= password=
    name=
    group=

    delete : Used to delete user account in the name space of the MAP.
    Example: delete /map1/accounts1/

    load : Used to move a binary image from an URL to the MAP. The URL is
    limited to 80 characters
    Example : load -source http://192.168.1.1/images/fw/iLO2_130.bin

    reset : Used to cause a target to cycle from enabled to disabled and back to enabled.

    set : Used to set a property or set of properties to a specific value.
    start : Used to cause a target to change state to a higher run level.
    stop : Used to cause a target to change state to a lower run level.
    cd : Used to set the current default target.
    Example: cd targetname

    exit : Used to terminate the CLP session.
    version : Used to query the version of the CLP implementation or other CLP
    elements.

    oemhp_ping : Used to determine if an IP address is reachable from this iLO 2.
    Example : oemhp_ping 192.168.1.1 , where 192.168.1.1 is the IP address that you wish
    to ping

    oemhp_loadSSHKey : Used to authorize a SSH Key File from an URL The URL is
    limited to 80 characters
    Example : oemhp_loadSSHKey -source http://UserName:[email protected]/images/SSHkey1.ppk

    HP CLI Commands:

    POWER : Control server power.
    UID : Control Unit-ID light.
    NMI : Generate an NMI.
    VM : Virtual media commands.
    VSP : Invoke virtual serial port.

    Type VSP and you're in. To login as root you need to include the serial port (in this case ttyS1) on your /etc/securetty file or you will be given the error message that your user or password is wrong.

    hpiLO-> VSP

    Starting virtual serial port.
    Press 'ESC (' to return to the CLI Session.

    hpiLO-> Virtual Serial Port active: IO=0x02F8 INT=3

    login as:

    [Nov 01, 2011] What TCP-IP ports does iLO 3 use

    Aug 10, 2010 | HP Communities

    You can look up what ports are used via the iLO 3 web interface. Expand the "Administration" menu on the left, then click on the "Access Settings" link. That screen will tell you the ports used by the various services.

    Here are the defaults:
    SSH 22
    Web (non-SSL) 80
    SSL 443
    IPMI-over-LAN 623
    Remote Console 17990
    Virtual Media 17988

    You might also need to enable other ports if you're using DHCP, DNS, SNTP, SNMP, and/or LDAP from iLO.

    [Oct 31, 2011] c00257375 Best Practices for Integrated Lights-Out and Integrated Lights-Out 2, 3-rd edition

    HP

    For an iLO device to work properly when going across routers using port blocking and/or firewalls, ports 23, 80, 443, and 17988 must be open.

    The directory services LDAP port (636) may be required. The Terminal Services RDP port (3389) may be required.

    Port 23 is for the Telnet ports where the remote and graphical Remote Console is used, port 80 is for HTTP communications, port 443 is required for the HTTPS connection, and port 17988 is for Virtual Media.

    LDAP traffic from a directory server uses random port numbers to enter the iLO device.

    The inability to access the iLO management ports is often confused with incorrect proxy settings. When in doubt, disable proxy in Internet Explorer or Netscape.

    [Oct 31, 2011] Default Port for the remote console for iLO 3

    HP Communities

    Hi Guys,

    We have found that the remote console port defined for iLo3 has changed from being 3389 (standard RDP port) to 17990.

    Can one of you please ask HP about the reasoning about this change and if it will be an issue if we change this to the standard 3389 port. The alternative is that we get NS to open the port 17990 on the firewall then we do not have to manually change every iLO 3 interface for servers in ecom.

    ***************************

    David responded:

    **************************

    I think they're confusing 2 different things. Port #3389 is a standard RDP port and was valid for the "iLO Terminal Services Pass-through" but never was the port for accessing the iLO remote console. Since TS Pass-through is no longer available with iLO3, this doesn't apply.

    [Oct 31, 2011] Opening firewall ports for iLO

    Aug.20, 2009 | NachoTech

    If you want to access an iLO behind a firewall, there are some TCP ports that need to be opened on the firewall to allow all iLO traffic to flow through. Here is a list of the default ports used by iLO, but these can be modified on iLO's Administration… Access… Services… tab.

    ILO FUNCTION           SOCKET TYPE PORT NUMBER 
---------------------- ----------- -----------
Secure Shell (SSH)         TCP        22
Remote Console/Telnet      TCP        23
Web Server Non-SSL         TCP        80
Web Server SSL             TCP        443
Terminal Services          TCP        3389
Virtual Media              TCP        17988
Shared Remote Console      TCP        9300
Console Replay             TCP        17990
Raw Serial Data            TCP        3002

    [Oct 12, 2011] HP Integrated Lights-Out 3 Version: 1.26 (29 Aug 2011)

    Installation:

    To update firmware from the Linux operating system on target server:
    Download the SCEXE file to the target server. Execute: sh CP015458.scexe

    To obtain firmware image for updating via iLO user interface, utilities, or scripting interface:
    Download the SCEXE file to a client running a Linux operating system. Execute: sh CP015458.scexe --unpack=directory.

    This command will unpack the iLO3 bin into a user specified "directory". If the directory does not exist, the unpacker will attempt to create it.

    To use HP Smart Update Manager on the Firmware Maintenance CD:

    [Oct 11, 2011] Reset ILO (Integrated Lights-Out 2) on HP Server

    Stracca Blog

    Recently I had the necessity to reset the ILO interface of an HP Proliant Server.
    I found that you need to connect in ssh (or in telnet) to do it.
    One connect give this commands:

    cd /Map1
    reset

    Here an example:

    User:admin logged-in to ILOGB87451B7E(10.1.1.15)
    iLO 2 Advanced 1.81 at 11:05:47 Jan 15 2010
    Server Name: myserver.mydomain.com
    Server Power: On

    hpiLO-> cd /Map1
    status=0
    status_tag=COMMAND COMPLETED

    /Map1

    hpiLO-> reset
    status=0
    status_tag=COMMAND COMPLETED
    Resetting iLO.

    CLI session stopped

    [Mar 6, 2011] HP Integrated Lights-Out 3 (iLO 3) - Firmware CD Supplemental Update - Online ROM Flash Component for Linux

    Type: Firmware - Lights-Out Management
    Version: 1.28 (6 Mar 2012)

    Products: HP ProLiant DL Servers, HP ProLiant BL Server Blades, HP ProLiant Scalable Systems, HP ProLiant ML Servers, Insight Control Software
    OS: Windows,Linux,Other Legacy OS

    Description: This component provides updated iLO firmware that can be installed directly on supported Linux Operating Systems. This component can also be used to obtain the firmware image for updating via iLO user interface, utilities, or through the scripting...

    Upgrade Requirement:
    Optional - Users should update to this version if their system is affected by one of the documented fixes or if there is a desire to utilize any of the enhanced functionality provided by this version.

    Recommended Links

    Google matched content

    Softpanorama Recommended

    Top articles

    Sites

    HP Integrated Lights-Out - Wikipedia, the free encyclopedia

    HP Integrated Lights-Out 3 (iLO 3) - Manuals - HP Business Support Center

    Proliant Watch Configuring HP Integrated Lights-Out (iLO) - HP Proliant Server

    NachoTech Opening firewall ports for iLO

    Setting up ssh tunnel to access a distant iLO - Tin's Journey

    Random Findings

    Topic: Configuring iLO2 and Debian with serial support

    Starting a text-mode Debian install using the virtual serial port (VSP) via an ssh connection to iLO 2

    no remcons in ilo2?

    iLO 2

    You will have to start an agetty process on the COM2 port in order to use the VSP. RHEL 5 example: 

    S1:2345:respawn:/sbin/agetty 115200 ttyS1 vt100

    In order to see the startup/shutdown messages on the VSP, add the following to the appropriate kernel line in /boot/grub/menu.lst: 

    ...
kernel /vmlinuz-2.6.18-8.el5 ro root=LABEL=/ console=tty0 console=ttyS1,115200
....

    OpenSSH problems with iLO ssh server

    The OpenSSH client on my Ubuntu 7.10 system is incompatible with the ssh server on an iLO system with 1.91 firmware. There may be problems with other OpenSSH/iLO versions. 

    $ ssh -V
OpenSSH_4.6p1 Debian-5build1, OpenSSL 0.9.8e 23 Feb 2007

$ echo "QUIT" | nc iLO 22
SSH-2.0-mpSSH_0.0.1

$ ssh admin@iLO
admin@iLO's password: 
dispatch_protocol_error: type 100 seq 9
dispatch_protocol_error: type 100 seq 10
buffer_get_ret: trying to get more bytes 4 than in buffer 0
buffer_get_int: buffer error

    As a workaround, use PuTTY.

    Using Virtual Media with the Command-Line Interface

    The iLO 2 Virtual Media Java applet does not work well over a low-speed WAN[1]. My Linux ISOs would routinely not boot when using this Java applet.

    As an alternative, ISOs may be loaded from an HTTP server using Virtual Media with the iLO command-line interface.

    Example: 

    hutch@hutch:~$ ssh admin@iLO
admin@iLO's password:
User:admin logged-in to iLO(10.215.14.5)
iLO Advanced 1.42 at 08:37:01 Oct 03 2007
Server Name: CZC7124NST00
Server Power: On

</>hpiLO-> vm cdrom insert http://10.215.0.35/kickstart/boot_isos/5Server-i386_boot.iso

(Note: use IPs when specifying an HTTP server)

</>hpiLO-> vm cdrom get
VM Applet = Disconnected
Boot Option = NO_BOOT
Write Protect = Yes
Image Inserted = Connected
Image URL = http://10.215.0.35/kickstart/boot_isos/5Server-i386_boot.iso

(Note: the "NO_BOOT" means that the system will not boot off the "connected" image)

</>hpiLO-> vm cdrom set boot_once

(Note: The next boot will be from the connected image) 

</>hpiLO-> power reset

    You will likely want to start a Remote Console via the iLO 2's HTTPS interface.

    Virtual Power Options

    From HP Integrated Lights-Out 2 User Guide for Firmware 1.35:

    Retrieved from "http://brandonhutchinson.com/wiki/ILO_Notes"

    Linux-BSD SysAdmin and Oracle DBA Guide How to login from serial port under Linux

    How to login from serial port under Linux

    Environment: Linux server A with at least one com port which is com1
    Objective: enable login through com1 using null modem cable from another machine B which can be Windows or Linux.

    Steps:

    1. connect null modem cable between A and B at com1 port
    2. on Linux server A, vi /etc/inittab and append the followin line:

    s0:2345:respawn:/sbin/agetty 115200 ttyS0 vt100

    3. If machine B is Linux, use minicom to configure the serial port speed as 115200, then connect
    4. If machine B is Windows, use putty to directly connect to serial port, set speed as 115200, then login

    note:

    for HP Proliant DL serial servers, you can also use VSP (Virtual Serial Port) with ILO2, you can basically ssh into ILO IP address with Administrator login. Add the following lines to /etc/inittab

    sx:2345:respawn:/sbin/agetty 115200 ttyS1 vt100

    then run 'init q' to enable it, after that, you can use vsp command to connect to this serial port login.

    FAQ:

    1. how to change speed? - best practise.
    vi /etc/inittab , for example, change 115200 to 9600, then comment out the line first, run 'init q' to re-read file, after that, uncomment it, issue 'init q' again.

    2. how to enable root login through serail port?
    Just add ttyS0 or ttyS1 into /etc/securetty.

    3. How to use HP virtual serial port(VSP) to login for HP Proliant servers

    use ssh connect to ILO ip address:(if there's firewall in between, enable port 22)
    for ILO version 1, just run 'REMCONS'.
    for ILO2 , type 'vsp' to connect to ttyS0 or ttyS1 to get console screen.

    References:

    1. HP Proliant server VSP documentation
    http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00263709/c00263709.pdf

    2. Redhat knowledgebase

    How do I set up a serial terminal and/or console in Red Hat Enterprise Linux?

    http://kbase.redhat.com/faq/docs/DOC-7213

    Useful Commands:
    1. setserial -a /dev/ttyS0

    Posted by Jephe Wu at 9/24/2009 01:57:00 PM

    Labels: console, serial port

    Etc

    Society

    Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

    Quotes

    War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

    Bulletin:

    Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

    History:

    Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

    Classic books:

    The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

    Most popular humor pages:

    Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

    The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

    Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

    FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

    This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

    		 You can use PayPal to to buy a cup of coffee for authors of this site

    Disclaimer:

    The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

    Last modified: March 12, 2019