Suse Troubleshooting

Troubleshooting, problem analysis and root cause determination requires patience, determination, and experience. It is important to fully investigate the problem and collect all relevant data in order to begin troubleshooting on the correct path. Although you may start out on one path and end up on another in order to resolve a complex problem, over time the most important skill will become patience. Keep log as in a process of solving complex problem you can become distracted and forget about important facts or findings. 

Troubleshooting of SLES problems can be a complex process. Also Suse is definitely less popular then Red Hat so Google is less helpful resource and you often need to solve problem in old-fashioned way -- yourself.   It is important to have a baseline of key configuration files see  Baseliners.

Often problem arise as the result of modification of  configuration files in some well organized fashion. So keeping automatic track record by using some configuration management system (or even simple copying of previous version with the timestamp) are really important. If this process is broken and several system administrators are responsible for the server as is typical in large corporations, nasty problems can arise.  In this case communication of  changes is a key.

The following section contains important log files, procedures, and tools used during the troubleshooting process.

The script supportconfig is a standard tool to collect all the relevant for troubleshooting information in Suse. It is recommended to run it periodically from cron at least weekly (daily for important server) so that you have a track record of key configuration files. There is also a tool that helps to analyze the collected information called the Supportconfig Health Check Report Tool (schealth)

15 October 2008 - 3:16pm
Submitted by: jrecord

license: GPLv2 home page url: http://en.opensuse.org/NTSutils 

The Supportconfig Health Check Report Tool (schealth) parses and evaluates the basic-health-check.txt file generated by supportconfig. The tool is based on the concepts outlined in the article, "A Basic Server Health Check with Supportconfig". The schealth output will be most effective to you if you have read and understand the concepts documented in the article.

Installation Instructions

1. schealth is included in the supportutils package along with supportconfig.

Usage

schealth [-hqv]

1. Get a supportconfig tar ball from the server.
2. Extract the supportconfig tar ball.
3. Change to the directory where the basic-health-check.txt file is located.
4. schealth requires the basic-health-check.txt file be in the current working directory.
5. Run schealth.
6. Observe the output. The output is also stored in the basic-health-report.txt file in the current directory.
7. Use the supportconfig output to troubleshoot in more detail any red or yellow flags reported by schealth.

Sample Output

######################################################
Supportconfig Health Check Report Tool v0.95.3
######################################################

Health Check Files                         [  Green  ]
Processes Waiting for Run Queue            [ Yellow  ]
 Last 1 and 15 minutes: 6 > 5 && 6 > 5

Kernel Taint Status                        [  Green  ]
CPU Utilization                            [   Red   ]
 95% meets or exceeds 90%

Interrupts Per Second                      [ Yellow  ]
 8852 meets or exceeds 8000

Context Switches Per Second                [   Red   ]
 16546 meets or exceeds 10000

Free Memory and Disk Swapping              [ Yellow  ]
 Observed: 3 MB <= 4 MB, Swapping: No

Used Disk Space                            [   Red   ]
 Some meet or exceed limits

Red Flags
/var                 96% >= 90%
/boot                93% >= 90%

Yellow Flags
/dev                 81% >= 80%

Uninterruptible Processes                  [ Yellow  ]
 3 meets or exceeds 3

Zombie Processes                           [  Green  ]

######################################################
Status:   Red Flag
Checked:  /var/log/nts_jrecord1_080711_0953/basic-health-check.txt
Report:   /var/log/nts_jrecord1_080711_0953/basic-health-report.txt
######################################################

Suse Log Files

SLES uses the syslog-ng  for logging. System events are written to log files in a way similar to standard syslog with /etc/log/messages as the main file.

 Important Log Files

NOTE: Log files for OES components can normally be located in /var/opt/novell/log.

/proc and /sys Filesystems

When you're troubleshooting hardware-related problems, it is often important to determine exactly what view the kernel has of all hardware devices attached to the server. The /proc filesystem is a virtual filesystem that allows an insight into the running kernel. Many kernel configuration values can be analyzed by viewing the appropriate file within the /proc directory structure.

Starting with the 2.6 kernel, the sysfs filesystem has been added for accessing additional information regarding kernel data structures and attributes. This filesystem is mounted at the /sys directory and can be used to query specific settings of hardware devices recognized by the current kernel. As not all devices have interfaces within the sysfs filesystem, both the /proc and /sys filesystems must be used for low-level device management.

Rescue Mode

See also The SUSE Rescue System

Rescue mode is a method of running Linux from the installation media rather than a damaged SLES installation. This mode is useful for advanced troubleshooting and disaster recovery when the installed operating environment is failing to start up properly. Rescue mode is accessed by following these steps:

mount -t reiserfs <root device (e.g. /dev/sda1)> /mnt

chroot /mnt

At this point, a new BASH shell has been opened within the filesystem of your SLES installation. Additional troubleshooting (reviewing log files, changing passwords, disabling services, and so on) can all be performed prior to rebooting in normal mode.

Troubleshooting Utilities

Troubleshooting Linux-related problems sometimes involves in-depth investigation of the disk, running processes, networking configuration, and countless other topics. Here is a small list of utilities often used in the troubleshooting process.

Using these troubleshooting utilities to track down and resolve issues can be a daunting task. For help with this process, or any technical issue you may face, contact Novell Technical Support (http://www.novell.com/support).

NEWS CONTENTS

• 20120602 : Optimizing and Troubleshooting SUSE Linux Enterprise Server 11, Advanced Technical Training, Course 8256 ( Optimizing and Troubleshooting SUSE Linux Enterprise Server 11, Advanced Technical Training, Course 8256, Jun 02, 2012 )
• 20110724 : Troubleshooting Common Boot Issues ( Troubleshooting Common Boot Issues, Jul 24, 2011 )
• 20100609 : SUSE Broken Don't fear the chroot ! by johnlange ( September 22, 2009 , Blang! )

Old News ;-)

[Jun 02, 2012] Optimizing and Troubleshooting SUSE Linux Enterprise Server 11, Advanced Technical Training, Course 8256

Day 1: Optimization 1

Section 1: Introduction to System Optimization Section 2: CPU Subsystem Optimization Section 3: Memory Subsystem Optimization

Day 2: Optimization 2

Section 4: Storage Subsystem Optimization Section 5: Network Subsystem Optimization Section 6: Application Subsystem Optimization

Day 3: Troubleshooting

Section 1: Troubleshooting Methodology Section 2: Troubleshooting Tools Section 3: Troubleshooting System Startup and Shutdown Section 4: Troubleshooting Kernel Configuration Section 5: Troubleshooting Kernel Panic Dumps

Day 4: Advanced Troubleshooting

Section 1: Introduction to Configuration Backups Section 2: Configuration Backup Fundamentals Section 3: Rapid Recovery

[Jul 24, 2011] Troubleshooting Common Boot Issues

This document (3864925) is provided subject to the disclaimer at the end of this document.

This document is intended as a general guideline for troubleshooting system boot issues. Please read and evaluate the entire document prior to contacting Novell Technical Support.

Symptom: Regardless of the kernel selected to boot (failsafe or default), a kernel panic stops the system from booting.

Error(s): RAMDISK: Couldn't find a valid RAM disk image starting at 0.
VFS: Cannot open a root device "sda2" or unknown-block(0,0)
Please append a correct"root=" boot option
Kernel panic - not syncing: VFS: Cannot open a root device "sda2" or unknown-block(0,0)

Probable Cause: A corrupted or missing initrd.

Resolution:

1. Boot Installed System*.
2. Login as root.
3. Verify that the / (root) and /boot (if used) filesystems are mounted. The mount command should supply sufficient information. If not, comparing its output with the contents of /etc/fstab should.
4. Run mkinitrd.
5. Reboot.

---

Symptom: The system fails to boot and prompts for the root password.

Error(s):
error on stat() /dev/hdb3: No such file or directory
Failed to open the device'/dev/hdb3' : No such file or directory
fsck.reiserfs /dev/hdb3 failed (status 0x8). Run manually!
fsck failed for at least one filesystem (not /).

Probable cause:
Invalid/etc/fstabentry, /dev/hdb3 is a non-existent device.

Resolution:

1. Enter the root password to enter maintenance mode.
2. Remount the root filesystem as read-write:
   

   mount -o rw,remount /

3. Edit /etc/fstab and remove the non-existent device entry. Comparing the output of fdisk -l may provide additional guidance for the non-existent device.
4. [CTRL]+[D] reboots.

---

Symptom: The system simply hangs after POST. The screen is completely blank. The option to Boot Installed System* is not available.

Error(s):
If Rescue System is attempted, and fdisk -l run, no partitions are seen. If parted is used, and check run,Error: Partition doesn't existis returned.

Probable cause:
The MBR has been damaged or corrupted.

Resolution:

1. Boot Installed System*.
2. Login as root.
3. Reinstall GRUB:
   

   grub-install bootdevicepath (e.g. /dev/sda)

4. Reboot.

-or-

1. If Boot Installed System* is unavailable, the most likely probable cause is that the partition table is damaged or corrupt, no recovery is possible unless a previous backup of the partition table is available.
   

---

Symptom: When the system boots, an error message is seen, and the system locks. Sometimes the screen just goes black or the server reboots. Sometimes all that is seen is the grub details screen halted after trying to load the/boot/initrd(see below).

Error(s):
No setup signature found ...
initrd /boot/initrd
[Linux-initrd @ 0x1fc38000, 0x2a7ab8 bytes]

Probable cause:
Damaged or corrupted kernel in/boot.

Resolution:

1. Boot Installed System*.
2. Login as root.
3. Install a valid kernel rpm. This can be had from the selected installation medium (under /suse/arch) or from our website at http://www.novell.com/download (search the patches section for kernel-).
   

   rpm -Uvh --force kernel-type-revision-arch.rpm

4. Reboot.

---

Symptom: The system boots up to System Boot Control: Running /etc/init.d/boot.local, then gracefully reboots.

Error(s):
None.

Probable cause:
Corrupted or misconfigured boot script.

Resolution:

1. At the GRUB menu, type in

   init=/bin/bash
   

   on the Boot Options line.
2. Edit /etc/init.d/boot.local and modify or remove the corrupted or misconfigured line.
3. Reboot.

---

Symptom:
Once exited from a virtual console, the console is not respawned. The console prompt just blinks.

Error(s):
INIT: no more processes left in this runlevel

Probable cause:
Corrupt or misconfigured/etc/inittab

Resolution:

1. Login as root.
2. Edit /etc/inittab and change any tty configuration(s) from once to respawn.
3. Reboot or pkill -1 init.

---

Symptom: The kernel panics after trying to mount the root filesystem.

Error(s):
Waiting for device /dev/sda1 to appear: . ok
rootfs: major=8 minor=1 devn=2049
rootfs: /sys/block/sda/sda1 major=8 minor=1 devn=2049
mount: unknown filesystem type 'swap'
umount: /dev: device is busy
Kernel panic - not syncing: Attempted to kill init!
Kernel panic: VFS: Unable to mount root fs on sda1

Probable cause:
Corrupt or misconfigured/boot/grub/menu.lst.

Resolution:

1. Boot Installed System*.
2. Login as root.
3. Edit /boot/grub/menu.lst.
4. Modify the kernel parameter root= to point to the correct root partition. fdisk -l should provide sufficient guidance to determine the root filesystem.
   
5. Reboot.

---

Symptom:
The system boots to the GRUB prompt (grub>).

Error(s):
None.

Probable cause:
Corrupt or missing/boot/grub/menu.lstfile.

Resolution:

1. Boot Installed System* -or- if sufficiently familiar with GRUB, manually boot the system.
2. Login as root.
3. Verify the existence of /boot/grub/menu.lst.
   If existing, but misnamed, rename it.
   If corrupt, delete it and Repair Installed System** (just the Boot Loader Configuration check should be sufficient).
   If missing, Repair Installed System** (just the Boot Loader Configuration check should be sufficient).
   
4. Reboot.

---

Symptom:
The system boots, but filesystems are not mounted. Many mount-related errors are seen during boot.

Error(s):
Mostly mount-related error messages are seen during boot.
startproc: mount returned not-zero exit status
startproc: /proc not mounted, failed to mount: No such file or directory failed

Probable cause:
The mount binary is either corrupt or missing.

Resolution:

1. Either Repair Installed System** -or - the following.
   
2. Boot into rescue mode (Rescue System) from the selected installation medium.
   
3. Login as root.
4. Manually mount the root filesystem (i.e., mount /dev/sda2 /mnt). fdisk -l should provide sufficient information to determine the correct partition.
5. Copy in a valid mount binary from the Rescue System.
6. Reboot.

---

Symptom:
The system boots, but login fails.

Error(s):
INIT: no more processes left in this runlevel
INIT: /etc/inittab[xx]: missing action field

Probable cause:
Corrupt or misconfigured/etc/inittab

Resolution:

1. Login as root.
2. Edit /etc/inittab and change any tty configuration(s) to include an action (once or respawn) in the action field (third column).
3. Reboot or pkill -1 init.

---

Symptom:
The system boots, but only to a GRUB screen, then hangs.

Error(s):
GRUB Hard Disk Error

Probable Cause:
As the full GRUB prompt is not achieved, the problem lies somewhere in GRUB stage1. The /boot/grub/stage1file may be missing or corrupted.

Solution:

1. Boot Installed System*.
2. Login as root.
3. Copy /usr/lib/grub/stage1 to/boot/grub/stage1.
4. Reinstall GRUB:
   

   grub-install bootdevicepath(e.g., /dev/sda).

5. Reboot.

-or-

1. Boot Installed System*.
2. Login as root.
3. Identify the installed version of GRUB:
   

   rpm -q grub

4. Remove the installed version. E.g.,
   

   rpm -ev --nodeps grub-0.97-16.1

5. Reinstall the grub package. This can be had from the selected installation medium (under /suse/arch) or from our website at http://www.novell.com/download (search the patches section for grub).
   

   rpm -Uvh grub-version.rpm.

6. Reinstall GRUB:
   

   grub-install bootdevicepath(e.g.,/dev/sda).

---

Symptom:
The system either doesn't boot, or boots, but some modules aren't loaded and/or some devices are undetected.

Error(s):
FATAL: Error insertingmodulename(modulepath): Unknown symbol in module, or unknown parameter (see dmesg).
modulename: Unknown symbol symbolname

Probable cause:
Occasionally, when modules are updated, the modules dependency file/lib/modules/kernelversion/modules.depis improperly configured or corrupted prior to updating the initial ramdisk.

Resolution:

1. Boot Installed System*.
2. Login as root.
3. Run depmod to regenerate the modules dependency file.
4. Run mkinitrd to generate a new initial RAM disk image.
5. Reboot.

---

Symptom:
EVMS is used for the root filesystem. The system doesn't boot, with errors related to finding the root filesystem.

Error(s):
Waiting for device /dev/evms/lvm2/system/root to appear: ... not found

Probable cause:
The initial RAM disk image lacks EVMS support.

Resolution:

1. Boot the rescue system and enter the shell.
2. Probe EVMS information:
   

   echo "probe" | evms -s

3. Query devices:
   

   echo "q:d" | evms -s

4. Query volumes:
   

   echo "q:v" | evms -s
   

   This should display the name your root container.
5. Mount the root filesystem (note the comma in the mount command):
   

   mkdir -p /old
   echo "mount:/dev/evms/lvm2/system/root,/old" | evms -s

6. The contents of the old root filesystem should be visible now:
   

   ls -l /old/

7. Enter the environment of the installed system:
   

   chroot /old

8. Generate a new inital RAM disk, with EVMS support:
   

   /sbin/mkinitrd -f evms

9. Exit the environment of the installed system through [CTRL]+[D].
10. Reboot.

Status

Top Issue

Additional Information

* Boot Installed System is the process of using a SuSE Linux Enterprise Server installation medium to boot the installed system. These are the steps:

1. Boot the system off of the selected installation medium (CD/DVD in most cases). This medium should be the same (or later) revision level as the installed system. I.e., if the installed system is SLES9SP2, the installation medium should be SLES9SP2 or later.
2. At Welcome screen, Installation should be selected in place of Boot from Hard Disk.
3. Select the desired Language.
4. Accept the License Agreement(s) (if prompted).
   
5. At the Installation Mode screen, select Boot Installed System. On SLES10 and later, click on the [Other] button to see these options.

**Repair Installed System is a process similar to Boot Installed System and provides a more automatic repair process. In some cases, when a broader approach to fixing the issue is needed (sledgehammer rather than scalpel), Repair Installed System is the desired process. These are the steps:

1. Boot the system off of the selected installation medium (CD/DVD in most cases). This medium should be the same (or later) revision level as the installed system. I.e., if the installed system is SLES9SP2, the installation medium should be SLES9SP2 or later.
2. At Welcome screen, Installation should be selected in place of Boot from Hard Disk.
3. Select the desired Language.
4. Accept the License Agreement(s) (if prompted).
   
5. At the Installation Mode screen, select Repair Installed System. On SLES10 and later, click on the [Other] button to see these options.

[Jun 09, 2010] SUSE Broken Don't fear the chroot ! by johnlange

September 22, 2009 | Blang!

SUSE hasn't let me down very often but recently I had a bad experience while applying some updates to an OpenSUSE laptop. There were quite a few updates so I undocked the laptop so I could relax while they downloaded.

For reasons that I have not yet resolved, the wirless networking became unstable and as a result, the updates had to be aborted.

Unfortunately, a new kernel was part of the updates and when the laptop rebooted it was in a bad state. X windows wouldn't start and critically, there were no network drivers for the new kernel. To make matters worse, OpenSUSE does not keep the old kernels in /boot (why is that?) so there was nothing to fall back on.

With nothing left to do, it was time to try rescue mode and in a few short steps I had the system fully working again. Here is what I did:

Step 1: boot to rescue mode (duh).

Step 2: mount your hard disk partitions under /mnt in the same layout they would be normally. For example:

# mount /dev/sda2 /mnt

# mount /dev/sda1 /mnt/boot

… etc.

Step 3: Next we need to make sure we have access to all the important system resources.

# mount --bind /proc /mnt/proc
# mount --bind /sys /mnt/sys
# mount --bind /dev /mnt/dev

Step 4: We're ready to chroot into our new environment.

# chroot /mnt

Step 5: We are now running on our system just as if we had booted to it and we can perform repairs. In my case all I needed to do was complete the updates:

# zypper up

I rebooted and everything was back to normal.

Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: March 12, 2019