|
|
Home | Switchboard | Unix Administration | Red Hat | TCP/IP Networks | Neoliberalism | Toxic Managers |
| (slightly skeptical) Educational society promoting "Back to basics" movement against IT overcomplexity and bastardization of classic Unix | |||||||
|
|
Once you have the right packages installed and IB interfaces are visible, the next thing is to get a basic system setup in place.
The Infiniband stack is both kernel based and user based. In order to use any given piece of hardware, you need both the kernel driver for that hardware, and the user space driver for that hardware. In the case of mlx4 hardware (which is a two part kernel driver), that means you need the core mlx4 kernel driver (mlx4_core) and also the infiniband mlx4 driver (mlx4_ib).
|
|
When using Infiniband, it is best to make sure you have the openib package installed. Then, these drivers will be automatically loaded into the kernel for you if you enable the openidb service. That will take care of the kernel space component (you don't need to tell openibd what hardware you have, it figures it out for itself). You can tweak the kernel space modules that are loaded by editing /etc/ofed/openib.conf to suit your needs.
If you attempt to run an infiniband application and it says you have no hardware, then that means you either haven't enabled the openibd service or you really don't have any hardware ;-) However, if it says it can't find a driver for your hardware, then it's talking about the user space driver, not the kernel space driver.
For the user space component, you need the core infiniband hardware library, libibverbs, installed and you also need hardware specific library packages installed. The common hardware specific library packages are: libmthca, libmlx4, libipathverbs, libcxgb3, libehca, libnes. Once these packages are installed, the libibverbs library automatically loads whichever ones are necessary in order to support the hardware the kernel detected in your machine.
Once you have the proper kernel space and user space hardware drivers loaded, you are almost ready to start rudimentary testing.
The next thing you need is to select one machine to be the subnet manager for your infiniband network. This machine will need to have the opensm package installed and its service script, opensmd, enabled. The default settings will work for most people, however, if you have multiple, redundant infiniband network fabrics, then you will need to configure more than one machine to start opensm as it will only attach to one fabric each time it is run, and you will need to configure the additional instances of opensm to bind to the proper port so that it manages the redundant network fabric instead of the default fabric. You can edit /etc/ofed/opensm.conf to change what port opensm binds to.
Once you have the opensm machine selected, and you've started the machine with both the openibd and opensmd services enabled, you should have a functional infiniband fabric. An easy way to test this is to make sure that the libibverbs-utils package is installed and run ibv_devinfo and ibv_devices to see what infiniband/iwarp devices the system thinks are present. Assuming that your devices are found and ibv_devinfo shows your port state to be active, then you are ready to run programs on the infiniband fabric.
You can create tcp/ip interfaces over the infiniband network (IPoIB). To do so, you will need to create the ifcfg-ib0 (and possibly ifcfg-ib1) file in /etc/sysconfig/network-scripts. IPoIB interface types have not been added to our system-config-network tool, hence the need to manually create the files. In addition, IPoIB interfaces can not support dhcp, so they must be statically configured.
A sample ifcfg-ib0 file looks like this:
DEVICE=ib0
TYPE=Infiniband
BOOTPROTO=static
BROADCAST=192.168.0.255
IPADDR=192.168.0.1
NETMASK=255.255.255.0
NETWORK=192.168.0.0
ONBOOT=yes
In the case that you have two IB ports plugged into the same infiniband fabric (aka, on the same subnet,
not each port on its own subnet) and that you also have IPoIB enabled on both ports, then in order to
avoid possible confusion over why things sometimes work and sometimes don't when using IPoIB interface
addresses to initiate connections between machines, it is best to add the following lines to your
/etc/sysctl.conf file:
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.ib0.arp_ignore=1
net.ipv4.conf.ib1.arp_ignore=1
@ib_user - memlock 8192or
* - memlock 8192The value used above is a sample value. You can set the limit to -1 to remove the limit entirely. The actual amount of locked memory your application will need depends on how many connections it is going to open and how large of a message queue it allocates for each connection plus memory for the actual read/write buffers it sends. All RDMA memory must be locked in to physical memory so that the infiniband/iwarp hardware can safely access the memory via DMA.
Now, putting that network to use is another matter ;-)
|
|
Switchboard | ||||
| Latest | |||||
| Past week | |||||
| Past month | |||||
 |
 |
 |
Red Hat Customer Portal
According to Does Red Hat Enterprise Linux support FDR Infiniband devices? article, QDR (Quad Data Rate) Mellanox hardware is supported in Red Hat Enterprise Linux 6 and FDR (Fourteen Data Rate) hardware is supported since Red Hat Enterprise Linux version 6 update 3.The following should be made to install Infiniband support for Mellanox hardware on Red Hat Enterprise Linux 6:
1) Install the packages required and reboot:
[root@serv]# yum groupinstall "Infiniband Support" [root@serv]# yum install infiniband-diags perftest qperf opensm [root@serv]# chkconfig rdma on [root@serv]# chkconfig opensm on [root@serv]# shutdown -r now2) Check that IB ports are available (your output should differ):
[root@serv]# for i in `ls /sys/class/infiniband/*/ports/*/state`; do echo $i; cat $i; done /sys/class/infiniband/mlx4_0/ports/1/state 4: ACTIVE /sys/class/infiniband/mlx4_0/ports/2/state 1: DOWN3) Get current IB HCA state and topology (your output should differ):
(shows IB host nodes in topology)
[root@serv]# ibhosts(reports link info for all links in the fabric)
[root@serv]# iblinkinfo Switch 0x0002c9020041e0b8 Infiniscale-IV Mellanox Technologies: 2 1[ ] ==( 4X 2.5 Gbps Down/ Polling)==> [ ] "" ( ) ...skip... 2 31[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 16 1[ ] "serv-2 HCA-1" ( ) 2 32[ ] ==( 4X 10.0 Gbps Active/ LinkUp)==> 17 1[ ] "serv HCA-1" ( )4) (optional) Run some testing commands to ensure connectivity (your output and commands arguments should differ):
[root@serv]# ibtracert 16 17 [root@serv]# ibping -S -v [root@serv]# ibping -c 30 165) (optional) Disable MSI if you have 'bnx2' network cards:
[root@serv]# cat /etc/modprobe.d/bnx2-msi.conf options bnx2 disable_msi=1Please, see additional details in How do I disable MSI for the Broadcom bnx2 kernel module on Red Hat Enterprise Linux 6 ?
6) Furter configuration steps
Please, see the article How can I get started using Infiniband on Red Hat Enterprise Linux ? after you have installerd drivers and ensured Infiniband connectivity.
Society
Groupthink : Two Party System as Polyarchy : Corruption of Regulators : Bureaucracies : Understanding Micromanagers and Control Freaks : Toxic Managers : Harvard Mafia : Diplomatic Communication : Surviving a Bad Performance Review : Insufficient Retirement Funds as Immanent Problem of Neoliberal Regime : PseudoScience : Who Rules America : Neoliberalism : The Iron Law of Oligarchy : Libertarian Philosophy
Quotes
War and Peace : Skeptical Finance : John Kenneth Galbraith :Talleyrand : Oscar Wilde : Otto Von Bismarck : Keynes : George Carlin : Skeptics : Propaganda : SE quotes : Language Design and Programming Quotes : Random IT-related quotes : Somerset Maugham : Marcus Aurelius : Kurt Vonnegut : Eric Hoffer : Winston Churchill : Napoleon Bonaparte : Ambrose Bierce : Bernard Shaw : Mark Twain Quotes
Bulletin:
Vol 25, No.12 (December, 2013) Rational Fools vs. Efficient Crooks The efficient markets hypothesis : Political Skeptic Bulletin, 2013 : Unemployment Bulletin, 2010 : Vol 23, No.10 (October, 2011) An observation about corporate security departments : Slightly Skeptical Euromaydan Chronicles, June 2014 : Greenspan legacy bulletin, 2008 : Vol 25, No.10 (October, 2013) Cryptolocker Trojan (Win32/Crilock.A) : Vol 25, No.08 (August, 2013) Cloud providers as intelligence collection hubs : Financial Humor Bulletin, 2010 : Inequality Bulletin, 2009 : Financial Humor Bulletin, 2008 : Copyleft Problems Bulletin, 2004 : Financial Humor Bulletin, 2011 : Energy Bulletin, 2010 : Malware Protection Bulletin, 2010 : Vol 26, No.1 (January, 2013) Object-Oriented Cult : Political Skeptic Bulletin, 2011 : Vol 23, No.11 (November, 2011) Softpanorama classification of sysadmin horror stories : Vol 25, No.05 (May, 2013) Corporate bullshit as a communication method : Vol 25, No.06 (June, 2013) A Note on the Relationship of Brooks Law and Conway Law
History:
Fifty glorious years (1950-2000): the triumph of the US computer engineering : Donald Knuth : TAoCP and its Influence of Computer Science : Richard Stallman : Linus Torvalds : Larry Wall : John K. Ousterhout : CTSS : Multix OS Unix History : Unix shell history : VI editor : History of pipes concept : Solaris : MS DOS : Programming Languages History : PL/1 : Simula 67 : C : History of GCC development : Scripting Languages : Perl history : OS History : Mail : DNS : SSH : CPU Instruction Sets : SPARC systems 1987-2006 : Norton Commander : Norton Utilities : Norton Ghost : Frontpage history : Malware Defense History : GNU Screen : OSS early history
Classic books:
The Peter Principle : Parkinson Law : 1984 : The Mythical Man-Month : How to Solve It by George Polya : The Art of Computer Programming : The Elements of Programming Style : The Unix Hater’s Handbook : The Jargon file : The True Believer : Programming Pearls : The Good Soldier Svejk : The Power Elite
Most popular humor pages:
Manifest of the Softpanorama IT Slacker Society : Ten Commandments of the IT Slackers Society : Computer Humor Collection : BSD Logo Story : The Cuckoo's Egg : IT Slang : C++ Humor : ARE YOU A BBS ADDICT? : The Perl Purity Test : Object oriented programmers of all nations : Financial Humor : Financial Humor Bulletin, 2008 : Financial Humor Bulletin, 2010 : The Most Comprehensive Collection of Editor-related Humor : Programming Language Humor : Goldman Sachs related humor : Greenspan humor : C Humor : Scripting Humor : Real Programmers Humor : Web Humor : GPL-related Humor : OFM Humor : Politically Incorrect Humor : IDS Humor : "Linux Sucks" Humor : Russian Musical Humor : Best Russian Programmer Humor : Microsoft plans to buy Catholic Church : Richard Stallman Related Humor : Admin Humor : Perl-related Humor : Linus Torvalds Related humor : PseudoScience Related Humor : Networking Humor : Shell Humor : Financial Humor Bulletin, 2011 : Financial Humor Bulletin, 2012 : Financial Humor Bulletin, 2013 : Java Humor : Software Engineering Humor : Sun Solaris Related Humor : Education Humor : IBM Humor : Assembler-related Humor : VIM Humor : Computer Viruses Humor : Bright tomorrow is rescheduled to a day after tomorrow : Classic Computer Humor
The Last but not Least Technology is dominated by two types of people: those who understand what they do not manage and those who manage what they do not understand ~Archibald Putt. Ph.D
Copyright © 1996-2021 by Softpanorama Society. www.softpanorama.org was initially created as a service to the (now defunct) UN Sustainable Development Networking Programme (SDNP) without any remuneration. This document is an industrial compilation designed and created exclusively for educational use and is distributed under the Softpanorama Content License. Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
FAIR USE NOTICE This site contains copyrighted material the use of which has not always been specifically authorized by the copyright owner. We are making such material available to advance understanding of computer science, IT technology, economic, scientific, and social issues. We believe this constitutes a 'fair use' of any such copyrighted material as provided by section 107 of the US Copyright Law according to which such material can be distributed without profit exclusively for research and educational purposes.
This is a Spartan WHYFF (We Help You For Free) site written by people for whom English is not a native language. Grammar and spelling errors should be expected. The site contain some broken links as it develops like a living tree...
|
|
You can use PayPal to to buy a cup of coffee for authors of this site |
Disclaimer:
The statements, views and opinions presented on this web page are those of the author (or referenced source) and are not endorsed by, nor do they necessarily reflect, the opinions of the Softpanorama society. We do not warrant the correctness of the information provided or its fitness for any purpose. The site uses AdSense so you need to be aware of Google privacy policy. You you do not want to be tracked by Google please disable Javascript for this site. This site is perfectly usable without Javascript.
Last modified: November, 16, 2014
