Softpanorama

May the source be with you, but remember the KISS principle ;-)
Home Switchboard Unix Administration Red Hat TCP/IP Networks Neoliberalism Toxic Managers
(slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and  bastardization of classic Unix

Troubleshooting InfiniBand connection issues using OFED tools

Intel article:

Submitted by Peter Hartman (... on

The Open Fabrics Enterprise Distribution (OFED) package has many debugging tools available as part of the standard release. This article describes the use of those tools to troubleshoot the hardware and firmware of an InfiniBand fabric deployment.

First, the /sys/class sub-system should be checked to verify that the hardware is up and connected to the InfiniBand fabric. The following command will show the InfiniBand hardware modules recognized by the system:

ls /sys/class/infiniband

This example will use the module mlx4_0, which is typical for Mellanox ConnectX* series of adapters. If this, or a similar module, is not found, refer to the documentation that came with the OFED package on starting the OpenIB drivers.

Next, check the state of the InfiniBand port: cat /sys/class/infiniband/mlx4_0/ports/1/state

This command should return “ACTIVE” if the hardware is initialized, and the subnet manager has found the port and added the port to the InfiniBand fabric. If this command returns “INIT” the hardware is initialized, but the subnet manager has not added the port to the fabric yet.

If necessary, start the subnet manager:
/etc/init.d/opensmd start
Once the port on the head node is in the “ACTIVE” state, check the state of the InfiniBand port on all the compute nodes to ensure that all of the Infiniband hardware on the compute nodes has been initialized, and the subnet manager has added all of the compute nodes ports on to the fabric. This article will use the pdsh tool to run the command on all nodes:
 
pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/state
All nodes should report “ACTIVE”. If a node reports it cannot find the file, ensure the OpenIB drivers is loaded on that node. Refer to the documentation that came with the OFED package on starting the OpenIB drivers.

Once all of the compute nodes report that port 1 is “ACTIVE”, verify the speed on each port using the following commands:
 
cat /sys/class/infiniband/mlx4_0/ports/1/rate
pdsh –a cat /sys/class/infiniband/mlx4_0/ports/1/rate
This is a good first check for a bad cable or connection. Each port should report the same speed. For example, the output for double data rate (DDR) InfiniBand cards will be similar to “20 Gb/sec (4X DDR)”.

Once the above basic checks are complete, more in-depth troubleshooting can be performed. The main OFED tool for troubleshooting performance and connection problems is ibdiagnet. This tool runs multiple tests, as specified on the command line during the run, to detect errors related to the subnet, bad packets, and bad states. These errors are some of the more common seen during initial setup of Infiniband fabrics.

Run ibdiagnet with the following command line options:
ibdiagnet –pc –c 1000
The output will be similar to this:
Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2
-W- Topology file is not specified.
Reports regarding cluster links will use direct routes.
Loading IBDM from: /usr/lib64/ibdm1.2
-W- A few ports of local device are up.
Since port-num was not specified (-p option), port 1 of device 1 will be
used as the local port.
-I- Discovering ... 17 nodes (1 Switches & 16 CA-s) discovered.


-I---------------------------------------------------
-I- Bad Guids/LIDs Info
-I---------------------------------------------------
-I- No bad Guids were found

-I---------------------------------------------------
-I- Links With Logical State = INIT
-I---------------------------------------------------
-I- No bad Links (with logical state = INIT) were found

-I---------------------------------------------------
-I- PM Counters Info
-I---------------------------------------------------
-I- No illegal PM counters values were found

-I---------------------------------------------------
-I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list)
-I---------------------------------------------------

-I---------------------------------------------------
-I- IPoIB Subnets Check
-I---------------------------------------------------
-I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00
-W- No members found for group

-I---------------------------------------------------
-I- Bad Links Info
-I- Errors have occurred on the following links
(for errors details, look in log file /tmp/ibdiagnet.log):
-I---------------------------------------------------
Link at the end of direct route "1,5"
----------------------------------------------------------------
-I- Stages Status Report:
STAGE Errors Warnings
Bad GUIDs/LIDs Check 0 0
Link State Active Check 0 0
Performance Counters Report 0 0
Partitions Check 0 0
IPoIB Subnets Check 0 1
Link Errors Check 0 0

Please see /tmp/ibdiagnet.log for complete log
----------------------------------------------------------------
-I- Done. Run time was 9 seconds.
The warning “No members found for group” can safely be ignored.

In this example, a bad link was found: “Link at the end of direct route “1,5”.” "1,5" refers to the LID numbers associated with the individual ports. The following commands can be used to identify the LID numbers associated with each port:
 
cat /sys/class/infiniband/mlx4_0/ports/1/lid
pdsh –a /sys/class/infiniband/mlx4_0/ports/1/lid

This command generates a list of LIDs associated with nodes. In the output of the above command, locate the entries for 0x1 and 0x5. 0x1 is likely the head node. For errors of this type, reseat or replace the InfiniBand cable connecting the node corresponding to LID 0x5.

Finally, run ibdiagnet once more time to verify there are no errors, and then to check the error state of each port. Each test should pass.
ibdiagnet –pc –c 1000
ibcheckerrors.
Top updates
Bulletin Latest Past week Past month
Google Search

NEWS CONTENTS

Old News ;-)

Basic Infiniband troubleshooting by Van Roekel, Magdalene J,

Nov 21, 2013

There are several utilities that are part of the OFED stack that are useful for troubleshooting. They are

  1. ibdiagnet --> useful for scanning the fabric. This will use direct routes and takes a while to run. 
  2. ibnetdiscover --> This is used to generate the fabric topology. This is most useful when a standard topology is known. I have a topology file in /share/apps/Helium_IB.topo that I just updated. The idea would be that identifying a non-functioning node could be done by generating a new topology and doing a diff of it with the standard.
  3. iblinkinfo --> This reports on all of the links of the fabric and is pretty quick to run. This tool actually provide the easiest way to identify a downed link but it requires deducing it from neighbor links. Pipe this through 'less' and look for empty CA names with a state other than "Down". From the CA names around it, the down node should be easily discernible.
  4. ibcheckerrors --> self explanatory. This utility is somewhat noisy and yet not always very informative.
  5. ibqueryerrors --> This is the one that I like to use. Run it as follows
ibqueryerrors -c -s XmtWait

The errors to look for are SymbolErrors but the presence of SymbolErrors is not fatal. It is really the rate of increase of the errors that is what you want to look for. A high rate is likely to be fatal while a slow rate is still within the IB specs. Still, we have a couple of nodes that I would like to eventually change cables for.

Rebooted nodes will cause errors to be logged in the counters. After a node reboots, the ibclearerrors command should be run. It is also necessary to clear the errors when trying to ascertain the rates of error generation.

There are cases where the infiniband driver crashes. This causes the node to get in a state where it has to be power cycled because the modules can not be unloaded and a clean shutdown is not possible. There is a cron job that monitors the fabric and will pick this up and will send an email alert when one of the nodes is not responding. To determine which node it is requires running diagnostics on the fabric. To facilitate this, there is a reference topology file and a device map file in /share/apps/infiniband. To determine what has changed in the topology, run the following command from the /share/apps/infiniband directory

sudo /usr/sbin/ibnetdiscover | /usr/sbin/ibdiscover.pl | grep HCA

That will output the current topology followed by the differences from the reference. The map file is used to reference the node GUIDs to a human friendly host name. The nodes listed at the end are the ones that are different from the reference. They should be cross checked with host names that are known to be down.

Example output of a topology change

Delta change in topo (change between successive runs)

Link change: Local/Remote Port 1/ 1 Local/Remote GUID: 8f10500202e66/2c903004bd692
Locations: Local/Remote
"Voltaire 4036 # IBL17"

"compute-10-248 HCA-1"

Once the node that is broken is determined, it will need to be power cycled. It is a good idea to verify the host determined from the above is the correct one. One way would be to look through the output of iblinkinfo.

To power cycle the node use ipmitool on the head node, in the /root directory.

ipmitool -f /root/ipmipass -U admin -H compute-10-248.ilo power cycle
Make sure to use the ".ilo" extension to the host name.

Once the node is back up the queue instances will need to be enabled.

qmod -e *@compute-10-248
 

Recommended Links

Top Visited
Bulletin Latest Past week Past month
Google Search


Etc

Society

Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers :   Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism  : The Iron Law of Oligarchy : Libertarian Philosophy

Quotes

War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda  : SE quotes : Language Design and Programming Quotes : Random IT-related quotes :  Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce :  Bernard Shaw : Mark Twain Quotes

Bulletin:

Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 :  Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method  : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law

History:

Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds  : Larry Wall  : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS :  Programming Languages History : PL/1 : Simula 67 : C : History of GCC development :  Scripting Languages : Perl history   : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history

Classic books:

The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month :  How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite

Most popular humor pages:

Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor

The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D

Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.

This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...

 You can use PayPal to to buy a cup of coffee for authors of this site

Disclaimer:

The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.

Last modified: November, 16, 2014