	    Open Fabrics Enterprise Distribution (OFED)
          ConnectX driver (mlx4) in OFED 1.4.2 Release Notes
			  
  		   	July 2009


===============================================================================
Table of Contents
===============================================================================
1. Overview
2. Supported Firmware Versions
3. VPI (Virtual Process Interconnect)
4. Infiniband new features and bug fixes since OFED 1.3.1
5. Infiniband (mlx4_ib) new features and bug fixes since OFED 1.4
6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
7. New features and bug fixes since OFED 1.4.1
8. Known Issues
9. mlx4 Available Parameters

===============================================================================
1. Overview
===============================================================================
mlx4 is the low level driver implementation for the ConnectX adapters designed
by Mellanox Technologies. The ConnectX can operate as an InfiniBand adapter,
as an Ethernet NIC, or as a Fibre Channel HBA. The driver in OFED 1.4 supports
Infiniband and Ethernet NIC configurations. To accommodate the supported
configurations, the driver is split into three modules:
    
- mlx4_core
	Handles low-level functions like device initialization and firmware
	commands processing. Also controls resource allocation so that the
	InfiniBand and Ethernet functions can share the device without
	interfering with each other.
- mlx4_ib
	Handles InfiniBand-specific functions and plugs into the InfiniBand
	midlayer
- mlx4_en
	A new 10G driver named mlx4_en was added to drivers/net/mlx4.
	It handles Ethernet specific functions and plugs into the netdev mid-layer.

===============================================================================
2. Supported Firmware Versions
===============================================================================
- This release was tested with FW 2.6.000.
- The minimal version to use is 2.3.000. 
- To use both IB and Ethernet (VPI) use FW version 2.6.0

===============================================================================
3. VPI (Virtual Protocol Interconnect) 
===============================================================================
VPI enables ConnectX to be configured as an Ethernet NIC and/or an Infiniband
adapter.
o Overview:  
  The VPI driver is a combination of the Mellanox ConnectX HCA Ethernet and 
  Infiniband drivers.
  It supplies the user with the ability to run Infiniband and Ethernet 
  protocols on the same HCA (separately or at the same time).
  For more details on the Ethernet driver see MLNX_EN_README.txt.
o Firmware:
  The VPI driver works with FW 25408 version 2.6.000 or higher.
  One needs to use INI files that allow different protocols over same HCA.
o Installing the mlnx_en driver:
  To install the OFED 1.4 with the Ethernet driver make sure that "mlx4_en=y" 
  in file ofed.conf
o Loading drivers:
  To load the Ethernet driver one should set "MLX4_EN_LOAD=yes"
  in file /etc/infiniband/openib.conf. 
  If "MLX4_EN_LOAD" is not marked as "yes", the Ethernet driver can be loaded
  by running "/sbin/modprobe mlx4_en".
o Port type management:
  By default both ConnectX ports are initialized as Infiniband ports.
  If you wish to change the port type use the connectx_port_config script after
  the driver is loaded.
  Running "/sbin/connectx_port_config -s" will show current port configuration 
  for all ConnectX devices.
  Port configuration is saved in file: /etc/infiniband/connectx.conf.
  This saved configuration is restored at driver restart only if done via
  "/etc/init.d/openibd restart".
  
  Possible port types are:
    "eth"   - Always Ethernet.
    "ib"    - Always Infiniband.
    "auto"  - Link sensing mode - detect port type based on the attached 
              network type. If no link is detected, the driver retries link
	      sensing every few seconds.
    
  Port link type can be configured for each device in the system at run time
  using the "/sbin/connectx_port_config" script.
  
  This utility will prompt for the PCI device to be modified (if there is only
  one it will be selected automatically).
  At the next stage the user will be prompted for the desired mode for each port. 
  The desired port configuration will then be set for the selected device.
  Note: This utility also has a non interactive mode:
  "/sbin/connectx_port_config [[-d|--device <PCI device ID>] -c|--conf <port1,port2>]".

- The following configurations are supported by VPI:
	Port1 = eth   Port2 = eth
	Port1 = ib    Port2 = ib
	Port1 = auto  Port2 = auto
	Port1 = ib    Port2 = eth
	Port1 = ib    Port2 = auto
	Port1 = auto  Port2 = eth

  Note: the following options are not supported:
	Port1 = eth   Port2 = ib
	Port1 = eth   Port2 = auto
	Port1 = auto  Port2 = ib
  
	
===============================================================================
4. Infiniband new features and bug fixes since OFED 1.3.1
===============================================================================
Features that are enabled with FW 2.5.0 only:
- Send with invalidate and Local invalidate send queue work requests.
- Resize CQ support.

Features that are enabled with FW 2.6.0 only:
- Fast register MR send queue work requests.
- Local DMA L_Key.
- Raw Ethertype QP support (one QP per port) -- receive only.

Non FW dependent features:
- Allow 4K messages for UD QPs.
- Allocate/free fast register MR page lists.
- More efficient MTT allocator.
- RESET->ERR QP state transition no longer supported (IB Spec 1.2.1).
- Pass congestion management class MADs to the HCA.
- Enable firmware diagnostic counters available via sysfs.
- Enable LSO support for IPOIB.
- IB_EVENT_LID_CHANGE is generated more appropriately.
- Fixed race condition between create QP and destroy QP (bugzilla 1389)


===============================================================================
5. Infiniband new features and bug fixes since OFED 1.4
===============================================================================
- Enable setting via module param (set_4k_mtu) 4K MTU for ConnectX ports.
- Support optimized registration of huge pages backed memory.
  With this optimization, the number of MTT entries used is significantly
  lower than for regular memory, so the HCA will access registered memory with
  fewer cache misses and improved performance.
  For more information on this topic, please refer to Linux documentation file:
  Documentation/vm/hugetlbpage.txt
- Do not enable blueflame sends if write combining is not available  
- Add write combining support for for PPC64, and thus enable blueflame sends.
- Unregister IB device before executing CLOSE_PORT.
- Notify and exit if the kernel module used does not support XRC. This is done 
  to avoid libmlx4 compatibility problem.
- Added a module parameter (log_mtts_per_seg) for number of MTTs per segment.
  This enable to register more memory with the same number of segments.


===============================================================================
6. Eth (mlx4_en) new features and bug fixes since OFED 1.4
===============================================================================
6.1 Changes and New Features
----------------------------
- Added Tx Multi-queue support which Improves multi-stream and bi-directional
  TCP performance.
- Added IP Reassembly to improve RX bandwidth for IP fragmented packets.
- Added linear skb support which improves UDP performance.
- Removed the following module parameters:
   - rx/tx_ring_size
   - rx_ring_num - number of RX rings
   - pprx/pptx - global pause frames
   The parameters above are controlled through the standard Ethtool interface.

Bug Fixes
---------
- Memory leak when driver is unloaded without configuring interfaces first.
- Setting flow control parameters for one ConnectX port through Ethtool
  impacts the other port as well.
- Adaptive interrupt moderation malfunctions after receiving/transmitting
  around 7 Tera-bytes of data.
- Firmware commands fail with bad flow messages when bringing an interface up.
- Unexpected behavior in case of memory allocation failures.

===============================================================================
7. New features and bug fixes since OFED 1.4.1
===============================================================================
- Added support for new device ID: 0x6764: MT26468 ConnectX EN 10GigE PCIe gen2

===============================================================================
8. Known Issues
===============================================================================
- mlx4_en driver is not supported on PPC64 and IA64
- The mlx4_en module uses a Linux implementation for Large Receive Offload
  (LRO) in kernel 2.6.24 and later. These kernels require installing the
  "inet_lro" module.
- The SQD feature is not supported:
- To load the driver on machines with 64KB default page size UAR bar must be
  enlarged. 64KB page size is the default of PPC with RHEL5 and Itanium with
  SLES 11 or when 64KB page size enabled.
  Perform the following three steps:
  1. Add the following line in the firmware configuration (INI) file under the
     [HCA] section:
       log2_uar_bar_megabytes = 5
  2. Burn a modified firmware image with the changed INI file.
  3. Reboot the system.
- Ethernet in MLNX_OFED 1.4 is not supported for the following OPNs:
    MHQH29-XTC
    MHGH29-XTC
    MHGH29-XSC
    MHGH28-XTC
    MHGH28-XTC
    MHGH28-XSC
    MHGH28-XSC
    MHEH28-XTC
    MHEH28-XSC
    MHQH19-XTC

    Attempting to use these cards as NICs will yield the following error
    in /var/log/messages:
    mlx4_core 0000:0d:00.0: command 0x9 failed: fw status = 0x8
    <date & time> mtlx008 kernel: mlx4_en 0000:0d:00.0: Failed Initializing port
    <date & time> mtlx008 kernel: mlx4_en 0000:0d:00.0: Failed starting port:1

    As a workaround, use the MLNX_EN driver instead of OFED for Linux.

     
===============================================================================
9. mlx4 Available Parameters
===============================================================================
In order to set mlx4 parameters, add the following line(s) to /etc/modpobe.conf:
   options mlx4_core parameter=<value>
      and/or   
   options mlx4_ib   parameter=<value>
      and/or   
   options mlx4_en   parameter=<value>

mlx4_core parameters:
   set_4k_mtu:          attempt to set 4K MTU to all ConnectX ports (default 0)
   msi_x:		attempt to use MSI-X if nonzero (default 1)
   enable_qos:		Enable Quality of Service support in the HCA if > 0, (default 0) 
   block_loopback	Block multicast loopback packets if > 0  (default: 1)
   internal_err_reset:	Reset device on internal errors if non-zero (default 1)
   debug_level:		Enable debug tracing if > 0 (default 0)
   log_num_qp:		log maximum number of QPs per HCA (default is 17; max is 20)
   log_num_srq:		log maximum number of SRQs per HCA (default is 16; max is 20)
   log_rdmarc_per_qp:	log number of RDMARC buffers per QP (default is 4; max is 7)
   log_num_cq:		log maximum number of CQs per HCA (default is 16 max is 19)
   log_num_mcg:		log maximum number of multicast groups per HCA (default is 13; max is 21)
   log_num_mpt:		log maximum number of memory protection table entries per HCA 
   			(default is 17; max is 20)
   log_num_mtt:		log maximum number of memory translation table segments per HCA
   			(default is 20; max is 20)
   log_num_mac:		log maximum number of MACs per ETH port (1-7) (int)
   log_num_vlan:	log maximum number of VLANs per ETH port (0-7) (int)
   log_mtts_per_seg     Log2 number of MTT entries per segment (1-5; default is 3)
   use_prio:		Enable steering by VLAN priority on ETH ports (0/1, default 0) (bool)

mlx4_ib parameters:
   debug_level: 	Enable debug tracing if > 0 (default 0)

mlx4_en parameters:
   rss_xor:             Use XOR hash function for RSS 0 (default is xor)
   rss_mask:            RSS hash type bitmask (default is 0xf)
   num_lro:             Number of LRO sessions per ring or disabled (0) (default is 32)
   pfctx:               Priority based Flow Control policy on TX[7:0]. 
   			Per priority bit mask (default is 0)
   pfcrx:               Priority based Flow Control policy on RX[7:0]. 
   			Per priority bit mask (default is 0)
   inline_thold:        threshold for using inline data (default is 128)
