Tuning the run-time characteristics of Infiniband communications

Table of contents:

1. What OpenFabrics-based components does Open MPI have?
2. Does Open MPI support iWARP?
3. Does Open MPI support RoCE (RDMA over Converged Ethernet)?
4. I have an OFED-based cluster; will Open MPI work with that?
5. Where do I get the OFED software from?
6. Isn't Open MPI included in the OFED software package? Can I install another copy of Open MPI besides the one that is included in OFED?
7. What versions of Open MPI are in OFED?
8. Why are you using the name "openib" for the BTL name?
9. Is the mVAPI-based BTL still supported?
10. How do I specify to use the OpenFabrics network for MPI messages?
11. But wait -- I also have a TCP network. Do I need to explicitly disable the TCP BTL?
12. How do I know what MCA parameters are available for tuning MPI performance?
13. I'm experiencing a problem with Open MPI on my OpenFabrics-based network; how do I troubleshoot and get help?
14. What is "registered" (or "pinned") memory?
15. I'm getting errors about "error registering openib memory"; what do I do?
16. How can a system administrator (or user) change locked memory limits?
17. I'm still getting errors about "error registering openib memory"; what do I do?
18. Open MPI is warning me about limited registered memory; what does this mean?
19. I'm using Mellanox ConnectX HCA hardware and seeing terrible latency for short messages; how can I fix this?
20. How much registered memory is used by Open MPI? Is there a way to limit it?
21. How do I get Open MPI working on Chelsio iWARP devices?
22. I'm getting "ibv_create_qp: returned 0 byte(s) for max inline data" errors; what is this, and how do I fix it?
23. My bandwidth seems [far] smaller than it should be; why? Can this be fixed?
24. How do I tune small messages in Open MPI v1.1 and later versions?
25. How do I tune large message behavior in Open MPI the v1.2 series?
26. How do I tune large message behavior in the Open MPI v1.3 (and later) series?
27. How does the mpi_leave_pinned parameter affect large message transfers?
28. How does the mpi_leave_pinned parameter affect memory management?
29. How does the mpi_leave_pinned parameter affect memory management in Open MPI v1.2?
30. How does the mpi_leave_pinned parameter affect memory management in Open MPI v1.3?
31. How can I set the mpi_leave_pinned MCA parameter?
32. I got an error message from Open MPI about not using the default GID prefix. What does that mean, and how do I fix it?
33. What subnet ID / prefix value should I use for my OpenFabrics networks?
34. How do I set my subnet ID?
35. In a configuration with multiple host ports on the same fabric, what connection pattern does Open MPI use?
36. I'm getting lower performance than I expected. Why?
37. I get bizarre linker warnings / errors / run-time faults when I try to compile my OpenFabrics MPI application statically. How do I fix this?
38. Can I use system(), popen(), or fork() in an MPI application that uses the OpenFabrics support?
39. My MPI application sometimes hangs when using the openib BTL; how can I fix this?
40. Does InfiniBand support QoS (Quality of Service)?
41. Does Open MPI support InfiniBand clusters with torus/mesh topologies?
42. How do I tell Open MPI which IB Service Level to use?
43. How do I run Open MPI over RoCE?
44. Does Open MPI support XRC?
45. How do I specify the type of receive queues that I want OMPI to use?
46. Does Open MPI support FCA?
47. Does Open MPI support MXM?

Open MPI supports the verbs and (starting in version 1.2) udapl APIs from the OpenFabrics Alliance (OFA) software stack. The OpenFabrics software stack supports both InfiniBand and iWARP networks. Note that the OpenFabrics Alliance used to be known as the OpenIB project -- so if you're thinking "OpenIB", you're thinking the right thing. Open MPI's support of the OpenFabrics stack is provided through the multiple different components; see the table below.

Open MPI also supported the Mellanox VAPI (mVAPI) software stack through the v1.2 series. See this FAQ entry for more details on VAPI support.

For more details on how to run Open MPI over RoCE, see this FAQ entry.

The "Download" section of the OpenFabrics web site has links for the various OFED releases.

Additionally, several InfiniBand vendors have their own releases of the OFED stack, customized for their hardware and/or support requirements. Consult with your IB vendor for more details.

shell$ mpirun --mca btl openib,self ...

shell$ mpirun --mca btl openib,self,sm ...

See this FAQ entry for more details on selecting which MCA plugins are used at run-time.

Finally, note that if the openib component is available at run time, Open MPI should automatically use it by default (ditto for self). Hence, it's usually unnecessary to specify these options on the mpirun command line. They are typically only used when you want to be absolutely positively definitely sure to use the specific BTL.

No. See this FAQ entry for more details.

# Show the openib BTL parameters
shell$ ompi_info --param btl openib

 

1. Which OpenFabrics version are you running? Please specify where you got the software from (e.g., from the OpenFabrics community web site, from a vendor, or it was already included in your Linux distribution).

    

2. What distro and version of Linux are you running? What is your kernel version?

    

3. Which subnet manager are you running? (e.g., OpenSM, a vendor-specific subnet manager, etc.)

    

4. What is the output of the ibv_devinfo command on a known "good" node and a known "bad" node? (NOTE: there must be at least one port listed as "PORT_ACTIVE" for Open MPI to work. If there is not at least one PORT_ACTIVE port, something is wrong with your OpenFabrics environment and Open MPI will not be able to run).

    

5. What is the output of the ifconfig command on a known "good" node and a known "bad" node? (mainly relevant for IPoIB installations) Note that some Linux distributions do not put ifconfig in the default path for normal users; look for it in /sbin/ifconfig or /usr/sbin/ifconfig.

    

6. If running under Bourne shells, what is the output of the "ulimit -l" command?
   If running under C shells, what is the output of the "limit | grep memorylocked" command?
   (NOTE: If the value is not "unlimited", this FAQ entry and this FAQ entry).

    

Gather up this information and see this page about how to submit a help request to the user's mailing list.

1. Linux kernel module parameters that control the amount of available registered memory are set too low; see this FAQ entry.
2. System / user needs to increase locked memory limits: see this FAQ entry and this FAQ entry.

 

1. Assuming that the PAM limits module is being used (see full docs for the Linux PAM limits module), the system-level default values are controlled by putting a file in /etc/security/limits.d/ (or directly editing the /etc/security/limits.conf file on older systems). Two limits are configurable:

    

   • Soft: The "soft" value is how much memory is allowed to be locked by user processes by default. Set it by creating a file in /etc/security/limits.d/ (e.g., 95-openfabrics.conf) with the line below (or, if your system doesn't have a /etc/security/limits.d/ directory, add a line directly to /etc/security/limits.conf):

      

     * soft memlock <number>

      

     where "<number>" is the number of bytes that you want user processes to be allowed to lock by default (presumably rounded down to an integral number of pages). "<number>" can also be "unlimited".

      

   • Hard: The "hard" value is the maximum amount of memory that a user process can lock. Similar to the soft lock, add it to the file you added to /etc/security/limits.d/ (or editing /etc/security/limits.conf directly on older systems):

      

     * hard memlock <number>

      

     where "<number>" is the maximum number of bytes that you want user processes to be allowed to lock (presumably rounded down to an integral number of pages). "<number>" can also be "unlimited".

    

2. Per-user default values are controlled via the ulimit (or limit in csh) command. The default amount of memory allowed to be locked will correspond to the "soft" limit set in /etc/security/limits.d/ (or limits.conf -- see above); users cannot use ulimit (or limit in csh) to set their amount to be more than the hard limit in /etc/security/limits.d (or limits.conf).

   Users can increase the default limit by adding the following to their shell startup files for Bourne style shells (sh, bash):

    

   ulimit -l unlimited

    

   Or for C style shells (csh, tcsh):

    

   limit memorylocked unlimited

    

   This effectively sets their limit to the hard limit in /etc/security/limits.d (or limits.conf). Alternatively, users can set a specific number instead of "unlimited," but this has limited usefulness unless a user is aware of exactly how much locked memory will require (which is difficult to know since Open MPI manages locked memory behind the scenes).

   It is important to realize that this must be set in all shells where Open MPI processes using OpenFabrics will be run. For example, if you are using rsh or ssh to start parallel jobs, it will be necessary to set the ulimit in your shell startup files so that it is effective on the processes that are started on each node.

   More specifically -- it may not be sufficient to simply execute the following, because the ulimit may not be in effect on all nodes where Open MPI processes will be run:

    

   shell$ ulimit -l unlimited shell$ mpirun -np 2 my_mpi_application

    

Ensure that the limits you've set (see this FAQ entry) are actually being used. There are two general cases where this can happen:

1. Your memory locked limits are not actually being applied for interactive and/or non-interactive logins.
2. You are starting MPI jobs under a resource manager / job scheduler that is either explicitly resetting the memory limited or has daemons that were (usually accidentally) started with very small memory locked limits.

That is, in some cases, it is possible to login to a node and not have the "limits" set properly. For example, consider the following post on the Open MPI User's list:

In this case, the user noted that the default configuration on his Linux system did not automatically load the pam_limits.so upon rsh-based logins, meaning that the hard and soft limits were not set.

There are also some default configurations where, even though the maximum limits are begin set system-wide in limits.d (or limits.conf on older systems), something during the boot procedure sets the default limit back down to a low number (e.g., 32k). In this case, you may need to override this limit on a per-user basis (described this FAQ entry), or effectively system-wide by putting "ulimit -l unlimited" (for Bourne-like shells) in a strategic location, such as:

• /etc/init.d/sshd (or wherever the script is that starts up your SSH daemon) and restarting the SSH daemon
• In a script in /etc/profile.d, or wherever system-wide shell startup scripts are located (e.g., /etc/profile and /etc/csh.cshrc)

Also, note that resource managers such as SLURM, Torque/PBS, LSF, etc. may affect OpenFabrics jobs in two ways:

1. Make sure that the resource manager daemons are started with unlimited memlock limits (which may involve editing the resource manager daemon startup script, or some other system-wide location that allows the resource manager daemon to get an unlimited limit of locked memory). Otherwise, jobs that are started under that resource manager will get the default locked memory limits, which are far too small for Open MPI.

   The files in limits.d (or the limits.conf file) does not usually apply to resource daemons! The limits.s files usually only applies to rsh or ssh-based logins. Hence, daemons usually inherit the system default of maximum 32k of locked memory (which then gets passed down to the MPI processes that they start). To increase this limit, you typically need to modify daemons' startup scripts to increase the limit before they drop root privliedges.

    

2. Some resource managers can limit the amount of locked memory that is made available to jobs. For example, SLURM has some fine-grained controls that allow locked memory for only SLURM jobs (i.e., the system's default is low memory lock limits, but SLURM jobs can get high memory lock limits). See these FAQ items on the SLURM web site for more details: propagating limits and using PAM.

Finally, note that some versions of SSH have problems with getting correct values from /etc/security/limits.d/ (or limits.conf) when using privilege separation. You may notice this by ssh'ing into a node and seeing that your memlock limits are far lower than what you have listed in /etc/security/limits.d/ (or limits.conf) (e.g., 32k instead of unlimited). Several web sites suggest disabling privilege separation in ssh to make PAM limits work properly, but others imply that this may be fixed in recent versions of Open SSH.

If you do disable privilege separation in ssh, be sure to check with your local system administrator and/or security officers to understand the full implications of this change. See this Google search link for more information.

In newer hardware:
    max_reg_mem = (2^log_num_mtt) * (2^log_mtts_per_seg) * PAGE_SIZE

In older hardware: 
    max_reg_mem = num_mtt * (2^log_mtts_per_seg) * PAGE_SIZE

Mellanox has advised the Open MPI community to increase the log_num_mtt value (or num_mtt value), not the log_mtts_per_seg value (even though an IBM article suggests increasing the log_mtts_per_seg value).

It is recommended that you adjust log_num_mtt (or num_mtt) such that your max_reg_mem value is at least twice the amount of physical memory on your machine (setting it to a value higher than the amount of physical memory present allows the internal Mellanox driver tables to handle fragmentation and other overhead). For example, if a node has 64 GB of memory and a 4 KB page size, log_num_mtt should be set to 24 and (assuming log_mtts_per_seg is set to 1). This will allow processes on the node to register:

max_reg_mem = (2^24) * (2^1) * (4 kB) = 128 GB

NOTE: Starting with OFED 2.0, OFED's default kernel parameter values should allow registering twice the physical memory size.

[Mellanox Hermon]
vendor_id = 0x2c9,0x5ad,0x66a,0x8f1,0x1708
vendor_part_id = 25408,25418,25428
use_eager_rdma = 1
mtu = 2048

shell$ ompi_info --param btl openib

See this FAQ entry for information on how to set MCA parameters at run-time.

For the Chelsio T3 adapter, you must have at least OFED v1.3.1 and Chelsio firmware v6.0. Download the firmware from service.chelsio.com and put the uncompressed t3fw-6.0.0.bin file in /lib/firmware. Then reload the iw_cxgb3 module and bring up the ethernet interface to flash this new firmware. For example:

# Note that the URL for the firmware may change over time
shell# cd /lib/firmware
shell# wget http://service.chelsio.com/drivers/firmware/t3/t3fw-6.0.0.bin.gz
[...wget output...]
shell# gunzip t3fw-6.0.0.bin.gz
shell# rmmod iw_cxgb3 cxgb3
shell# modprobe iw_cxgb3

# This last step may happen automatically, depending on your
# Linux distro (assuming that the ethernet interface has previously
# been properly configured and is ready to bring up).  Substitute the
# proper ethernet interface name for your T3 (vs. ethX).
shell# ifup ethX

If all goes well, you should see a message similar to the following in your syslog 15-30 seconds later:

kernel: cxgb3 0000:0c:00.0: successful upgrade to firmware 6.0.0

Open MPI will work without any specific configuration to the openib BTL. Users wishing to performance tune the configurable options may wish to inspect the receive queue values. Those can be found in the "Chelsio T3" section of mca-btl-openib-hca-params.ini.

[0,1,0][btl_openib_endpoint.c:889:mca_btl_openib_endpoint_create_qp] 
ibv_create_qp: returned 0 byte(s) for max inline data

Open MPI, by default, uses a pipelined RDMA protocol. Additionally, in the v1.0 series of Open MPI, small messages use send/receive semantics (instead of RDMA -- small message RDMA was added in the v1.1 series). For some applications, this may result in lower-than-expected bandwidth. However, Open MPI also supports caching of registrations in a most recently used (MRU) list -- this bypasses the pipelined RDMA and allows messages to be sent faster (in some cases).

For version the v1.1 series, see this FAQ entry for more information about small message RDMA, its effect on latency, and how to tune it.

To enable the "leave pinned" behavior, set the MCA parameter mpi_leave_pinned to 1. For example:

shell$ mpirun --mca mpi_leave_pinned 1 ...

NOTE: The mpi_leave_pinned parameter was broken in Open MPI v1.3 and v1.3.1 (see this announcement). mpi_leave_pinned functionality was fixed in v1.3.2.

This will enable the MRU cache and will typically increase bandwidth performance for applications which reuse the same send/receive buffers.

NOTE: The v1.3 series enabled "leave pinned" behavior by default when applicable; it is usually unnecessary to specify this flag anymore.

Open MPI uses a few different protocols for large messages. Much detail is provided in this paper.

The btl_openib_flags MCA parameter is a set of bit flags that influences which protocol is used; they generally indicate what kind of transfers are allowed to send the bulk of long messages. Specifically, these flags do not regulate the behavior of "match" headers or other intermediate fragments.

The following flags are available:

 

• Use send/receive semantics (1): Allow the use of send/receive semantics.
• Use PUT semantics (2): Allow the sender to use RDMA writes.
• Use GET semantics (4): Allow the receiver to use RDMA reads.

   

Open MPI defaults to setting both the PUT and GET flags (value 6).

Open MPI uses the following long message protocols:

 

1. RDMA Direct: If RDMA writes or reads are allowed by btl_openib_flags and the sender's message is already registered (either by use of the mpi_leave_pinned MCA parameter or if the buffer was allocated via MPI_ALLOC_MEM), a slightly simpler protocol is used:

    

    

   1. Send the "match" fragment: the sender sends the MPI message information (communicator, tag, etc.) to the receiver using copy in/copy out semantics. No data from the user message is included in the match header.
   2. Use RDMA to transfer the message:
      • If RDMA reads are enabled and only one network connection is available between the pair of MPI processes, once the receiver has posted a matching MPI receive, it issues an RDMA read to get the message, and sends an ACK back to the sender when the transfer has completed.
      • If the above condition is not met, then RDMA writes must be enabled (or we would not have chosen this protocol). The receiver sends an ACK back when a matching MPI receive is posted and the sender issues an RDMA write across each available network link (i.e., BTL module) to transfer the message. The RDMA write sizes are weighted across the available network links. For example, if two MPI processes are connected by both SDR and DDR IB networks, this protocol will issue an RDMA write for 1/3 of the entire message across the SDR network and will issue a second RDMA write for the remaining 2/3 of the message across the DDR network. The sender then sends an ACK to the receiver when the transfer has completed.

      NOTE: Per above, if striping across multiple network interfaces is available, only RDMA writes are used. The reason that RDMA reads are not used is solely because of an implementation artifact in Open MPI; we didn't implement it because using RDMA reads only saves the cost of a short message round trip, the extra code complexity didn't seem worth it for long messages (i.e., the performance difference will be negligible).

       

   Note that the user buffer is not unregistered when the RDMA transfer(s) is(are) completed.

    

    

2. RDMA Pipeline: If RDMA Direct was not used and RDMA writes are allowed by btl_openib_flags and the sender's message is not already registered, a 3-phase pipelined protocol is used:

    

    

   1. Send the "match" fragment: the sender sends the MPI message information (communicator, tag, etc.) and the first fragment of the user's message using copy in/copy out semantics.
   2. Send "intermediate" fragments: once the receiver has posted a matching MPI receive, it sends an ACK back to the sender. The sender and receiver then start registering memory for RDMA. To cover the cost of registering the memory, several more fragments are sent to the receiver using copy in/copy out semantics.
   3. Transfer the remaining fragments: once memory registrations start completing on both the sender and the receiver (see the paper for details), the sender uses RDMA writes to transfer the remaining fragments in the large message.

       

   Note that phases 2 and 3 occur in parallel. Each phase 3 fragment is unregistered when its transfer completes (see the paper for more details).

   Also note that one of the benefits of the pipelined protocol is that large messages will naturally be striped across all available network interfaces.

   The sizes of the fragments in each of the three phases are tunable by the MCA parameters shown in the figure below (all sizes are in units of bytes):

    

   

    

3. Send/Receive: If RDMA Direct and RDMA Pipeline were not used, copy in/copy out semantics are used for the whole message (note that this will happen even if the SEND flag is not set in btl_openib_flags):

    

   1. Send the "match" fragment: the sender sends the MPI message information (communicator, tag, etc.) and the first fragment of the user's message using copy in/copy out semantics.
   2. Send remaining fragments: once the receiver has posted a matching MPI receive, it sends an ACK back to the sender. The sender then uses copy in/copy out semantics to send the remaining fragments to the receiver.

       

   This protocol behaves the same as the RDMA Pipeline protocol when the btl_openib_min_rdma_size value is infinite.

    

The Open MPI v1.3 (and later) series generally use the same protocols for sending long messages as described for the v1.2 series, but the MCA parameters for the RDMA Pipeline protocol were both moved and renamed (all sizes are in units of bytes):

The change to move the "intermediate" fragments to the end of the message was made to better support applications that call fork(). Specifically, there is a problem in Linux when a process with registered memory calls fork(): the registered memory will physically not be available to the child process (touching memory in the child that is registered in the parent will cause a segfault or other error). Because memory is registered in units of pages, the end of a long message is likely to share the same page as other heap memory in use by the application. If this last page of the large message is registered, then all the memory in that page -- to include other buffers that are not part of the long message -- will not be available to the child. By moving the "intermediate" fragments to the end of the message, the end of the message will be sent with copy in/copy out semantics and, more importantly, will not have its page registered. This increases the chance that child processes will be able to access other memory in the same page as the end of the large message without problems.

Some notes about these parameters:

 

• btl_openib_rndv_eager_limit defaults to the same value as btl_openib_eager_limit (the size for "small" messages). It is a separate parameter in case you want/need different values.
• The btl_openib_min_rdma_size parameter was an absolute offset into the message; it was replaced by btl_openib_rdma_pipeline_send_length, which is a length.

   

Note that messages must be larger than btl_openib_min_rdma_pipeline_size (a new MCA parameter to the v1.3 series) to use the RDMA Direct or RDMA Pipeline protocols. Messages shorter than this length will use the Send/Receive protocol (even if the SEND flag is not set on btl_openib_flags).

NOTE: The mpi_leave_pinned parameter was broken in Open MPI v1.3 and v1.3.1 (see this announcement). mpi_leave_pinned functionality was fixed in v1.3.2.

When mpi_leave_pinned is set to 1, Open MPI aggressively tries to pre-register user message buffers so that the RDMA Direct protocol can be used. Additionally, user buffers are left registered so that the de-registration and re-registration costs are not incurred if the same buffer is used in a future message passing operation.

NOTE: Starting with Open MPI v1.3, mpi_leave_pinned is automatically set to 1 by default when applicable. It is therefore usually unnecessary to set this value manually.

NOTE: The mpi_leave_pinned MCA parameter has some restrictions on how it can be set starting with Open MPI v1.3.2. See this FAQ entry for details.

Leaving user memory registered when sends complete can be extremely beneficial for applications that repeatedly re-use the same send buffers (such as ping-pong benchmarks). Additionally, the fact that a single RDMA transfer is used and the entire process runs in hardware with very little software intervention results in utilizing the maximum possible bandwidth.

Leaving user memory registered has disadvantages, however. Bad Things happen if registered memory is free()ed, for example -- it can silently invalidate Open MPI's cache of knowing which memory is registered and which is not. The MPI layer usually has no visibility on when the MPI application calls free() (or otherwise frees memory, such as through munmap() or sbrk()). Open MPI has implemented complicated schemes that intercept calls to return memory to the OS. Upon intercept, Open MPI examines whether the memory is registered, and if so, unregisters it before returning the memory to the OS.

These schemes are best described as "icky" and can actually cause real problems in applications that provide their own internal memory allocators. Additionally, only some applications (most notably, ping-pong benchmark applications) benefit from "leave pinned" behavior -- those who consistently re-use the same buffers for sending and receiving long messages.

It is for these reasons that "leave pinned" behavior is not enabled by default. Note that other MPI implementations enable "leave pinned" behavior by default.

Also note that another pipeline-related MCA parameter also exists: mpi_leave_pinned_pipeline. Setting this parameter to 1 enables the use of the RDMA Pipeline protocol, but simply leaves the user's memory registered when RDMA transfers complete (eliminating the cost of registering / unregistering memory during the pipelined sends / receives). This can be beneficial to a small class of user MPI applications.

NOTE: The mpi_leave_pinned parameter was broken in Open MPI v1.3 and v1.3.1 (see this announcement). mpi_leave_pinned functionality was fixed in v1.3.2.

When mpi_leave_pinned is set to 1, Open MPI aggressively leaves user memory registered with the OpenFabrics network stack after the first time it is used with a send or receive MPI function. This allows Open MPI to avoid expensive registration / deregistration function invocations for each send or receive MPI function.

NOTE: The mpi_leave_pinned MCA parameter has some restrictions on how it can be set starting with Open MPI v1.3.2. See this FAQ entry for details.

However, registered memory has two drawbacks:

• There is only so much registered memory available
• User applications may free the memory, thereby invalidating Open MPI's internal table of what memory is already registered

The second problem can lead to silent data corruption or process failure. As such, this behavior must be disallowed. Note that the real issue is not simply freeing memory, but rather returning registered memory to the OS (where it can potentially be used by a different process). Open MPI has two methods of solving the issue:

• Using an internal memory manager; effectively overriding calls to malloc(), free(), mmap(), munmap(), etc.
• Telling the OS to never return memory from the process to the OS.

How these options are used differs between Open MPI v1.2 (and earlier) and Open MPI v1.3 (and later).

Be sure to read this FAQ entry first.

Open MPI 1.2 and earlier on Linux used the ptmalloc2 memory allocator linked into the Open MPI libraries to handle memory deregistration. On Mac OS X, it uses an interface provided by Apple for hooking into the virtual memory system, and on other platforms no safe memory registration was available. The ptmalloc2 code could be disabled at Open MPI configure time with the option --without-memory-manager, however it could not be avoided once Open MPI was built.

ptmalloc2 can cause large memory utilization numbers for a small number of applications and has a variety of link-time issues. Therefore, by default Open MPI did not use the registration cache, resulting in lower peak bandwidth. The inability to disable ptmalloc2 after Open MPI was built also resulted in headaches for users.

Open MPI v1.3 handles leave pinned memory management differently.

NOTE: The mpi_leave_pinned parameter was broken in Open MPI v1.3 and v1.3.1 (see this announcement). mpi_leave_pinned functionality was fixed in v1.3.2.

Be sure to read this FAQ entry first.

NOTE: The mpi_leave_pinned MCA parameter has some restrictions on how it can be set starting with Open MPI v1.3.2. See this FAQ entry for details.

With Open MPI 1.3, Mac OS X uses the same hooks as the 1.2 series, and most operating systems do not provide pinning support. However, the pinning support on Linux has changed. ptmalloc2 is now by default built as a standalone library (with dependencies on the internal Open MPI libopen-pal library), so that users by default do not have the problematic code linked in with their application. Further, if OpenFabrics networks are being used, Open MPI will use the mallopt() system call to disable returning memory to the OS if no other hooks are provided, resulting in higher peak bandwidth by default.

To utilize the independent ptmalloc2 library, users need to add -lopenmpi-malloc to the link command for their application:

shell$ mpicc foo.o -o foo -lopenmpi-malloc

Linking in libopenmpi-malloc will result in the OpenFabrics BTL not enabling mallopt() but using the hooks provided with the ptmalloc2 library instead.

To revert to the v1.2 (and prior) behavior, with ptmalloc2 folded into libopen-pal, Open MPI can be built with the --enable-ptmalloc2-internal configure flag.

When not using ptmalloc2, mallopt() behavior can be disabled by disabling mpi_leave_pined:

shell$ mpirun --mca mpi_leave_pinned 0 ...

Because mpi_leave_pinned behavior is usually only useful for synthetic MPI benchmarks, the never-return-behavior-to-the-OS behavior was resisted by the Open MPI developers for a long time. Ultimately, it was adopted because a) it is less harmful than imposing the ptmalloc2 memory manager on all applications, and b) it was deemed important to enable mpi_leave_pinned behavior by default since Open MPI performance kept getting negatively compared to other MPI implementations that enable similar behavior by default.

NOTE: The mpi_leave_pinned parameter was broken in Open MPI v1.3 and v1.3.1 (see this announcement). mpi_leave_pinned functionality was fixed in v1.3.2.

As with all MCA parameters, the mpi_leave_pinned parameter (and mpi_leave_pinned_pipeline parameter) can be set from the mpirun command line:

shell$ mpirun --mca mpi_leave_pinned 1 ...

Prior to the v1.3 series, all the usual methods to set MCA parameters could be used to set mpi_leave_pinned.

However, starting with v1.3.2, not all of the usual methods to set MCA parameters apply to mpi_leave_pinned. Due to various operating system memory subsystem constriaints, Open MPI must react to the setting of the mpi_leave_pinned parameter in each MPI process before MPI_INIT is invoked. Specifically, some of Open MPI's MCA parameter propagation mechanisms are not activated until during MPI_INIT -- which is too late for mpi_leave_pinned.

As such, only the following MCA parameter-setting mechanisms can be used for mpi_leave_pinned and mpi_leave_pinned_pipeline:

• Command line: See the example above.
• Environment variable: Setting OMPI_MCA_mpi_leave_pinned to 1 before invoking mpirun.

To be clear: you cannot set the mpi_leave_pinned MCA parameter via Aggregate MCA parameter files or normal MCA parameter files. This is expected to be an acceptable restriction, however, since the default value of the mpi_leave_pinned parameter is "-1", meaning "determine at run-time if it is worthwhile to use leave-pinned behavior." Specifically, if mpi_leave_pinned is set to -1, if any of the following are true when each MPI processes starts, then Open MPI will use leave-pinned bheavior:

• Either the environment variable OMPI_MCA_mpi_leave_pinned or OMPI_MCA_mpi_leave_pinned_pipeline is set to a positive value (note that the "mpirun --mca mpi_leave_pinned 1 ..." command-line syntax simply results in setting these environment variables in each MPI process).
• Any of the following files / directories can be found in the filesystem where the MPI process is running:
  • /sys/class/infiniband
  • /dev/open-mx
  • /dev/myri[0-9]

Note that if either the environment variable OMPI_MCA_mpi_leave_pinned or OMPI_MCA_mpi_leave_pinned_pipeline is set to to "-1", then the above indicators are ignored and Open MPI will not use leave-pinned behavior.

WARNING: There are more than one active ports on host '%s', but the
default subnet GID prefix was detected on more than one of these
ports.  If these ports are connected to different physical OFA
networks, this configuration will fail in Open MPI.  This version of
Open MPI requires that every physically separate OFA subnet that is
used between connected MPI processes must have different subnet ID
values.

See this FAQ entry for instructions on how to set the subnet ID.

Here's a more detailed explanation:

Since Open MPI can utilize multiple network links to send MPI traffic, it needs to be able to compute the "reachability" of all network endpoints that it can use. Specifically, for each network endpoint, Open MPI calculates which other network endpoints are reachable.

In OpenFabrics networks, Open MPI uses the subnet ID to differentiate between subnets -- assuming that if two ports share the same subnet ID, they are reachable from each other. If multiple, physically separate OFA networks use the same subnet ID (such as the default subnet ID), it is not possible for Open MPI to tell them apart and therefore reachability cannot be computed properly.

See this FAQ entry for instructions on how to set the subnet ID.

Measuring performance accurately is an extremely difficult task, especially with fast machines and networks. Be sure to read this FAQ entry for many suggestions on benchmarking performance.

Pay particular attention to the discussion of processor affinity and NUMA systems -- running benchmarks without processor affinity and/or on CPU sockets that are not directly connected to the bus where the HCA is located can lead to confusing or misleading performance results.

Read both this FAQ entry and this FAQ entry in their entirety.

The answer is, unfortunately, complicated. Be sure to also see this FAQ entry as well.

 

• If you have a Linux kernel before version 2.6.16: no. Some distros may provide patches for older versions (e.g, RHEL4 may someday receive a hotfix).

   

• If you have a version of OFED before v1.2: sort of. Specifically, newer kernels with OFED 1.0 and OFED 1.1 may generally allow the use of system() and/or the use of fork() as long as the parent does nothing until the child exits.

   

• If you have a Linux kernel >= v2.6.16 and OFED >= v1.2 and Open MPI >= v1.2.1: yes. Open MPI v1.2.1 added two MCA values relevant to arbitrary fork() support in Open MPI:

   

  • btl_openib_have_fork_support: This is a "read-only" MCA value, meaning that users cannot change it in the normal ways that MCA parameter values are set. It can be queried via the ompi_info command; it will have a value of 1 if this installation of Open MPI supports fork(); 0 otherwise.

     

  • btl_openib_want_fork_support: This MCA parameter can be used to request conditional, absolute, or no fork() support. The following values are supported:
    1. Negative values: try to enable fork support, but continue even if it is not available.
    2. Zero: Do not try to enable fork support.
    3. Positive values: Try to enable fork support and fail if it is not available.

   

Hence, you can reliably query OMPI to see if it has support for fork() and force OMPI to abort if you request fork support and it doesn't have it.

This feature is helpful to users who switch around between multiple clusters and/or versions of Open MPI; they can script to know whether the OMPI that they're using (and therefore the underlying IB stack) has fork support. For example:

#!/bin/sh 

have_fork_support=`ompi_info --param btl openib --parsable | grep have_fork_support:value | cut -d: -f7`
if test "$have_fork_support" = "1"; then
    # Happiness / world peace / birds are singing
else
    # Despair / time for Haagen Daas
fi

Alternatively, you can skip querying and simply try to run your job:

shell$ mpirun --mca btl_openib_want_fork_support 1 --mca btl openib,self ...

Which will abort if OMPI's openib BTL does not have fork support.

All this being said, even if Open MPI is able to enable the OpenFabrics fork() support, it does not mean that your fork()-calling application is safe.

• In general, if your application calls system() or popen(), it will likely be safe.

   

• However, note that arbitrary fork() support is not supported in the OpenFabrics software stack. If you use fork() in your application, you must not touch any registered memory before calling some form of exec() to launch another process. Doing so will cause an immediate seg fault / program crash.

  It is important to note that memory is registered on a per-page basis; it is therefore possible that your application may have memory co-located on the same page as a buffer that was passed to an MPI communications routine (e.g., MPI_Send() or MPI_Recv()) or some other internally-registered memory inside Open MPI. You may therefore accidentally "touch" a page that is registered without even realizing it, thereby crashing your application.

  This is unfortunately no way around this issue; it was intentionally designed into the OpenFabrics software stack. Please complain to the OpenFabrics Alliance that they should really fix this problem!

See Open MPI ticket #1224 for further information.

NOTE: This FAQ entry only applies to the v1.2 series. This functionality is not required for v1.3 and beyond because of changes in how message passing progress occurs. Specifically, this MCA parameter will only exist in the v1.2 series.

For details on how to tell Open MPI which IB Service Level to use, please see this FAQ entry.

For details on how to tell Open MPI to dynamically query OpenSM for IB Service Level, please refer to this FAQ entry.

NOTE: 3D-Torus and other torus/mesh IB topologies are supported as of version 1.5.4.

shell$ mpirun --mca btl openib,self,sm \
    --mca btl_openib_ib_service_level N ...

shell$ mpirun --mca btl openib,self,sm \
    --mca btl_openib_ib_path_record_service_level 1 ...

shell$ mpirun --mca btl openib,self,sm --mca btl_openib_cpc_include rdmacm ...

(or set these MCA parameters in other ways)

How do I tell Open MPI to use a specific RoCE VLAN?

When a system administrator configures VLAN in RoCE, every VLAN is assigned with its own GID. The QP that is created by the rdmacm CPC uses this GID as a Source GID. When OMPI (or any other application for that matter) posts a send to this QP, the driver checks the source GID to determine which VLAN the traffic is supposed to use, and marks the packet accordingly.

Note that InfiniBand SL (Service Level) is not involved in this process - marking is done in accordance with local kernel policy.

To control which VLAN will be selected, use the btl_openib_ipaddr_include/exclude MCA parameters and provide it with the required IP/netmask values. For example, if you want to use a VLAN with IP 13.x.x.x:

shell$ mpirun --mca btl openib,self,sm --mca btl_openib_cpc_include rdmacm \
    --mca btl_openib_ipaddr_include "13.0.0.0/8" ...

NOTE: VLAN selection in the OMPI v1.4 series works only with version 1.4.4 or later.

See this FAQ entry for instructions how to tell Open MPI to use XRC receive queues.

You can use the btl_openib_receive_queues MCA parameter to specify the exact type of the receive queues for the OMPI to use. This can be advantageous, for example, when you know the exact sizes of messages that your MPI application will use -- Open MPI can internally pre-post receive buffers of exactly the right size. See this paper for more details.

The btl_openib_receive_queues parameter takes a colon-delimited string listing one or more receive queues of specific sizes and characteristics. For now, all processes in the job must use the same string. You can specify three kinds of receive queues:

• P : Per-Peer Receive Queues
• S : Shared Receive Queues (SRQ)
• X : eXtended Reliable Connection queues (XRC, supported since OFED 1.4 with Mellanox ConnectX family HCAs)

The default value of the btl_openib_receive_queues MCA parameter is sometimes equivalent to the following command line:

shell$ mpirun ... --mca btl_openib_receive_queues P,128,256,192,128:S,2048,256,128,32:S,12288,256,128,32:S,65536,256,128,32 ...

In particular, note that XRC is (currently) not used by default.

NOTE: Open MPI chooses a default value of btl_openib_receive_queues based on the type of OpenFabrics network device that is found. The text file $openmpi_packagedata_dir/mca-btl-openib-device-params.ini (which is typically $openmpi_installation_prefix_dir/share/openmpi/mca-btl-openib-device-params.ini) contains a list of default values for different OpenFabrics devices. See that file for further explanation of how default values are chosen.

Per-Peer Receive Queues

Per-peer receive queues require between 1 and 5 parameters:

1. Buffer size in bytes: mandatory
2. Number of buffers: optional; defaults to 8
3. Low buffer count watermark: optional; defaults to (num_buffers / 2)
4. Credit window size: optional; defaults to (low_watermark / 2)
5. Number of buffers reserved for credit messages: optional; defaults to ((num_buffers*2 - 1) / credit_window)

Example: P,128,256,128,16

1. 128 byte buffers
2. 256 buffers to receive incoming MPI messages
3. When the number of available buffers reaches 128, re-post 128 more buffers to reach a total of 256
4. If the number of available credits reaches 16, send an explicit credit message to the sender
5. Defaulting to ((256*2) - 1) / 16 = 31; this many buffers are reserved for explicit credit messages

Shared Receive Queues

Shared Receive Queues can take between 1 and 4 parameters:

1. Buffer size in bytes: mandatory
2. Number of buffers: optional; defaults to 16
3. Low buffer count watermark: optional; defaults to (num_buffers / 2)
4. Maximum number of outstanding sends a sender can have: optional; defaults to (low_watermark / 4)

Example: S,1024,256,128,32

1. 1024 byte buffers
2. 256 buffers to receive incoming MPI messages
3. When the number of available buffers reaches 128, re-post 128 more buffers to reach a total of 256
4. A sender will not send to a peer unless it has less than 32 outstanding sends to that peer

XRC Queues

XRC queues take the same parameters as SRQs. Note that if you use any XRC queues, then all of your queues must be XRC. Therefore, to use XRC, specify the following:

shell$ mpirun --mca btl openib... --mca btl_openib_receive_queues X,128,256,192,128:X,2048,256,128,32:X,12288,256,128,32:X,65536,256,128,32 ...

NOTE: the rdmacm CPC is not supported with XRC. Also, XRC cannot be used when btls_per_lid > 1.

FCA (which stands for Fabric Collective Accelerator) is a Mellanox MPI-integrated software package that utilizes CORE-Direct technology for implementing the MPI collectives communications.
 

You can find more information about FCA on the product web page. FCA is available for download here: http://www.mellanox.com/products/fca

Building Open MPI 1.5.x or later with FCA support

By default, FCA is installed in /opt/mellanox/fca. Use the following configure option to enable FCA integration in OMPI:

shell$ ./configure --with-fca=/opt/mellanox/fca ...

Verifying the FCA Installation

To verify that Open MPI is built with FCA support, use the following command:

shell$ ompi_info --param coll fca | grep fca_enable

A list of FCA parameters will be displayed if Open MPI has FCA support.

How do I tell Open MPI to use FCA?

shell$ mpirun  --mca coll_fca_enable 1 ...

By default, FCA will be enabled only with 64 or more ranks. To turn on FCA for an arbitrary number of ranks ( N ), please use the following MCA parameters:

shell$ mpirun --mca coll_fca_enable 1 --mca coll_fca_np <N> ...

The MXM stand-alone package is available for download here: http://www.mellanox.com/products/mxm

MXM requirements

• Mellanox OFED 1.5.3 or later
• Open MPI 1.6 or later

Building Open MPI with MXM support

By default, MXM is installed in /opt/mellanox/mxm. Use the following configure option to enable MXM support in OMPI:

shell$ ./configure --with-mxm=/opt/mellanox/mxm ...

Enabling MXM in Open MPI 1.6.1 or later

Running Open MPI job with MXM:

shell$ mpirun --mca mtl mxm ...

MXM advantages kick in with large Open MPI jobs only. By default, MXM is automatically selected by Open MPI when the job has at least 128 ranks. To enable MXM for an arbitrary number of ranks ( N ), please use the following parameter:

shell$ mpirun --mca mtl_mxm_np <N> ...

To activate MXM for any job, run the following command:

shell$ mpirun --mca mtl_mxm_np 0 ...

NOTE: In Open MPI 1.6 MXM will always be selected, regardless of the number of ranks.