Manual Pages for UNIX Operating System command usage for man ibp

Devices                                                   ibp(7D)



NAME
     ibp - Infiniband IPoIB device driver

SYNOPSIS
     /dev/ibp*


DESCRIPTION
     The ibp driver implements the IETF IP over Infiniband proto-
     col  and provides IPoIB service for all IBA ports present in
     the system. For more information about  managing  the  data-
     links created by the ibp driver, see dladm(1M) manual page.


     The ibp driver  is  a  multi-threaded,  loadable,  clonable,
     STREAMS  hardware  driver supporting the connectionless Data
     Link Provider Interface, dlpi(7P)).


     The ibp driver provides  basic  support  for  both  the  IBA
     Unreliable Datagram Queue Pair hardware and the IBA Reliable
     Connected Queue Pair hardware. Functions include QP initial-
     ization, frame transmit and receive, multicast and promiscu-
     ous  mode support, and statistics reporting.


     By default, Connected Mode will be used by the each IB link.
     This  behavior can be modified by changing the linkmode pro-
     perty of the data link. See  the  EXAMPLES  section  of  the
     dladm(1M) manual page for information .


     Because ibp over connected mode attempts to use a large  MTU
     (65520 bytes), applications should adapt to the large MTU to
     get better performance, for example, adopting  a  large  TCP
     window size.


     Use the cloning, character-special device /dev/ibp to access
     all ibp devices installed within the system.


     The ibp driver is dependent on GLD, a loadable kernel module
     that provides the ibp driver with the DLPI and STREAMS func-
     tionality required of a LAN driver. Except as noted  in  the
     Application  Programming Interface section of this man page,
     see gld(7D) for more details on the primitives supported  by
     the    driver.    The    GLD    module    is    located   at
     /kernel/misc/sparcv9/gld  on   64   bit   systems   and   at
     /kernel/misc/gld on 32 bit systems.





SunOS 5.11          Last change: 14 Apr 2010                    1






Devices                                                   ibp(7D)



     The ibp driver expects  certain  configuration  of  the  IBA
     fabric prior to operation (which also implies the SM must be
     active and managing the fabric). Specifically, the IBA  mul-
     ticast group representing the IPv4 limited broadcast address
     255.255.255.255 (also defined as broadcast-GID in IETF docu-
     ments)  should  be created prior to initializing the device.
     IBA properties (including mtu, qkey and sl) of this group is
     used  by  the driver to create any other IBA multicast group
     as instructed by higher  level  (IP)  software.  The  driver
     probes  for  the  existance  of  this  broadcast-GID  during
     attach(9E).

APPLICATION PROGRAMMING INTERFACE (DLPI)
     The values returned by the driver in the DL_INFO_ACK  primi-
     tive in response to your DL_INFO_REQ are:

         o    Maximum  SDU  is  the  MTU  associated   with   the
              broadcast-GID group, less the 4 byte IPoIB header.

         o    Minimum SDU is 0.

         o    dlsap address length is 22.

         o    MAC type is DL_IB.

         o    The sap length value is -2,  meaning  the  physical
              address  component  is followed immediately by a 2-
              byte sap component within the DLSAP address.

         o    Broadcast address value is the MAC address consist-
              ing  of the 4 bytes of QPN 00:FF:FF:FF prepended to
              the IBA multicast address of the broadcast-GID.

              Due to the nature of link  address  definition  for
              IPoIB,  the  DL_SET_PHYS_ADDR_REQ DLPI primitive is
              not supported.

              In the transmit case for streams that have been put
              in raw mode via the DLIOCRAW ioctl, the DLPI appli-
              cation must prepend the 20 byte  IPoIB  destination
              address  to the data it wants to transmit over-the-
              wire. In the receive case, applications receive the
              IP/ARP  datagram along with the IETF defined 4 byte
              header.

WARNING
     This section describes warning messages that might  be  gen-
     erated  by  the driver. Please note that while the format of
     these messages can be modified in future versions, the  same
     general information is provided.





SunOS 5.11          Last change: 14 Apr 2010                    2






Devices                                                   ibp(7D)



     While joining IBA multicast groups corresponding to IP  mul-
     ticast groups as part of multicast promiscuous operations as
     required by IP multicast routers,  or  as  part  of  running
     snoop(1M),  it  is  possible  that  joins  to some multicast
     groups can fail due to inherent resource constraints in  the
     IBA  components.  In  such cases, warning message similar to
     the following appear  in  the  system  log,  indicating  the
     interface on which the failure occurred:

       NOTICE: ibp: Could not get list of IBA multicast groups
       NOTICE: ibp: IBA promiscuous mode missed multicast group
       NOTICE: ibp: IBA promiscuous mode missed new multicast gid



     Also, if the IBA SM indicates that multicast trap support is
     suspended  or unavailable, the system log contains a message
     similar to:

       NOTICE: ibp: IBA multicast support degraded due to
       unavailability of multicast traps



     And when the SM indicates trap support is restored:

       NOTICE: ibp: IBA multicast support restored due to
       availability of multicast traps



     Additionally, if the IBA link transitions to an  unavailable
     state  (that is, the IBA link state becomes Down, Initialize
     or Armed) and then becomes active again, the driver tries to
     rejoin  previously  joined  groups  if  required. Failure to
     rejoin multicast groups triggers messages such as:

       NOTICE: ibp: Failure on port up to rejoin multicast gid



     If the corresponding HCA port is in  the  unavailable  state
     defined  above  when  initializing  an  ibp  interface using
     ifconfig(1M), a message is emitted by the driver:

       NOTICE: ibp: Port is not active



     Further, as described above, if  the  broadcast-GID  is  not
     found,  or  the  associated  MTU is higher than what the HCA
     port can support, the following messages are printed to  the



SunOS 5.11          Last change: 14 Apr 2010                    3






Devices                                                   ibp(7D)



     system log:

       NOTICE: ibp: IPoIB broadcast group absent
       NOTICE: ibp: IPoIB broadcast group MTU 4096 greater than port's
       maximum MTU 2048



     In  all  cases  of  these  reported  problems  when  running
     ifconfig(1M),  it  should  be  checked  that  IBA cabling is
     intact, an SM is running on the fabric, and  the  broadcast-
     GID  with appropriate properties has been created in the IBA
     partition.


     The MTU of Reliable Connected mode can be  larger  than  the
     MTU of Unreliable Datagram mode.


     When Reliable Connected mode  is  enabled,  ibp  still  uses
     Unreliable  Datagram  mode to transmit and receive multicast
     packets. If the payload size (excluding 4 byte IPoIB header)
     of  a multicast packet is larger than the IP link MTU speci-
     fied by the broadcast group, ibp drops it. A message appears
     in the system log when drops occur:

       NOTICE: ibp: Reliable Connected mode is on. Multicast packet
       length ( > ) is too long to send




     If only one side has enabled Reliable Connected  mode,  com-
     munication  falls  back to datagram mode. The connected mode
     instance uses Path MTU discovery to automatically adjust the
     MTU  of a unicast packet if an MTU difference exists. Before
     Path MTU discovery reduces the MTU for a  specific  destina-
     tion,  several packets whose size exceeds the MTU of Unreli-
     able Datagram mode is dropped.

CONFIGURATION
     The IPoIB service comes preconfigured on all  HCA  ports  in
     the  system. To turn the service off, or back on after turn-
     ing it off, refer to documentation in cfgadm_ib(1M).

EXAMPLES
     Example 1 Using ibd

       # Below example modify the 'linkmode' to ud

       # dladm show-linkprop pffff.ibp0 | grep linkmode
       LINK        PROPERTY   PERM VALUE  DEFAULT    POSSIBLE



SunOS 5.11          Last change: 14 Apr 2010                    4






Devices                                                   ibp(7D)



       pffff.ibp0  linkmode    rw   cm       cm        cm,ud

       # dladm set-linkprop -p linkmode=ud pffff.ibp0

       # dladm show-linkprop pffff.ibp0 | grep linkmode
       LINK        PROPERTY   PERM VALUE  DEFAULT    POSSIBLE
       pffff.ibp0  linkmode    rw   ud      cm        cm,ud
       #


FILES
     /dev/ibp*                  Special character device


     /kernel/drv/ib.conf        Configuration file to start IPoIB
                                service


     /kernel/drv/sparcv9/ibp    64-bit SPARC device driver


     /kernel/drv/amd64/ibp      64-bit x86 device driver


     /kernel/drv/ibp            32-bit x86 device driver


SEE ALSO
     cfgadm(1M), dladm(1M), ifconfig(1M),  syslogd(1M),  gld(7D),
     ib(7D),  kstat(7D),  streamio(7I),  dlpi(7P), attributes(5),
     attach(9E)

NOTES
     IBP is  a  GLD-based  driver  and  provides  the  statistics
     described by gld(7D). Valid received packets not accepted by
     any stream (long) increases when IBP transmits broadcast  IP
     packets. This happens because the infiniband hardware copies
     and loops back the  transmitted  broadcast  packets  to  the
     source.  These packets are discarded by GLD and are recorded
     as unknowns.















SunOS 5.11          Last change: 14 Apr 2010                    5