The Verbs Fabric Provider
The verbs provider enables applications using OFI to be run over any verbs
hardware (Infiniband, iWarp, etc). It uses the Linux Verbs API for network
transport and provides a translation of OFI calls to appropriate verbs API
calls. It uses librdmacm for communication management and libibverbs for other
control and data transfer operations.
The verbs provider supports a subset of OFI features.
FI_EP_MSG, FI_EP_RDM (Experimental support FI_MSG, FI_TAGGED, FI_RMA
FI_MSG, FI_RMA, FI_ATOMIC and shared receive contexts.
FI_MSG, FI_TAGGED, FI_RMA
Verbs provider requires applications to support the following modes:
- FI_LOCAL_MR for all applications.
- FI_RX_CQ_DATA for applications that want to use RMA.
Applications must take responsibility of posting receives for any incoming
- FI_CONTEXT for applications making uses of the experimental
Supported addressing formats include FI_SOCKADDR, FI_SOCKADDR_IN,
Verbs provider supports FI_PROGRESS_AUTO: Asynchronous operations make forward
Verbs provider supports FI_INJECT, FI_COMPLETION, FI_REMOTE_CQ_DATA.
Verbs provider support the following messaging ordering on the TX side:
- Read after Read
- Read after Write
- Read after Send
- Write after Write
- Write after Send
- Send after Write
- Send after Send
Verbs provider supports the fork system call by default. See the limitations
section for restrictions. It can be turned off by setting the FI_FORK_UNSAFE
environment variable to "yes". This can improve the performance of
memory registrations but it also makes the use of fork unsafe.
Only FI_MR_BASIC mode is supported. Adding regions via s/g list is not
supported. Generic fi_mr_regattr is not supported. No support for binding
memory regions to a counter.
Only FI_WAIT_FD wait object is supported. Wait sets are not supported.
Application has to make sure CQs are not overrun as this cannot be detected by
The following features are not supported in verbs provider:
FI_NAMED_RX_CTX, FI_DIRECTED_RECV, FI_TRIGGER, FI_MULTI_RECV, FI_RMA_EVENT,
Scalable endpoints, FABRIC_DIRECT
Counters, FI_SOURCE, FI_TAGGED, FI_PEEK, FI_CLAIM, fi_cancel, fi_ep_alias,
shared TX context, cq_readfrom operations.
The RDM support for verbs have the following limitations:
- Supports iovs of only size 1.
- Wait objects are not supported.
- Not thread safe.
The support for fork in the provider has the following limitations:
- Fabric resources like endpoint, CQ, EQ, etc. should not be
used in the forked process.
- The memory registered using fi_mr_reg has to be page
aligned since ibv_reg_mr marks the entire page that a memory region
belongs to as not to be re-mapped when the process is forked
The verbs provider checks for the following environment variables.
: The prefix or the full name of the network interface
associated with the IB device (default: ib)
: The number of pre-registered buffers for
buffered operations between the endpoints, must be a power of 2 (default: 8).
: The maximum size of a buffered operation
(bytes) (default: platform specific).
: The segment size for zero copy protocols
: The number of entries to be read from
the verbs completion queue at a time (default: 8).
: The wake up timeout of the helper thread
(usec) (default: 100).
: The operation code that will be used for
eager messaging. Only IBV_WR_SEND and IBV_WR_RDMA_WRITE_WITH_IMM are
supported. The last one is not applicable for iWarp. (default: IBV_WR_SEND)
The fi_info utility would give the up-to-date information on environment
variables: fi_info -p verbs -e