The Broadcom bnx2x NIC driver for VMware ESXi when installed on a ESXi 5.5 host is often an overlooked component when working with VMware NSX.
If you have one of the Broadcom NICs which supports VXLAN offload and uses the bnx2x driver, it is important to choose the correct driver version. Not all versions are created equal!
If you look through the release notes provided with all versions of the bnx2x driver download on the VMware website, it will show you a full history of changes and enhancements made to the driver.
An important change was made in version 1.78.56v[50,55].3 (Oct. 1, 2013)
And then it looks like Broadcom had the VXLAN offloading certified for ESX5.5 and the following change was made in version 2.710.70v[50,55].3 (Oct. 7, 2014)
So this means that if you have the bnx2x driver 1.78.56v[50,55].3 or newer, but older than 2.710.70.v[50,55].3, VXLAN offloading in ESXi 5.5 is disabled in the driver by default due to it not being certified.
I have seen customer environments where the bnx2x drivers installed on the ESXi 5.5 hosts falls between the 2 versions where VXLAN was disabled by default which has caused some unexpected VXLAN performance results.
So if you find that your driver version falls within the range where VXLAN offloading was disabled by default and you are looking at implementing VMware NSX, you will need to ensure you have a driver version which has VXLAN offloading certified & enabled.
Update – 1st June 2015: To check to see if VXLAN offload has been disabled by default in any driver version, you can use the following command, which should show you what the default is.
If updating the driver is not an option due to other reasons, you can also manually enable VXLAN offloads. First you need to see what options (if any) are already set.
And as you can see, there are no options set in the output above. So you can enabled it as follows:
But what happens if there is another option already set, like RSS.
The command would be as follows:
You will need to reboot the host for the changes to take affect, as apparently the driver cannot be unloaded whilst in use to apply the settings.
Once the host has rebooted, verify the setting has “stuck”