Enable SR-IOV on Broadcom NICs and configure VF count

On a test-setup we're evaluating Broadcom NetXtreme-E (2x100G) interfaces. The networking will be wired up on the system using openvswitch (ovs) as it's easy to handle and OVS can utilize SR-IOV and hardware offloading.

An issue we had, is that during testing, the Broadcom cards didn't offer sriov configuration (running Firmware 227.0.131.0). Upgrading to firmware 228.1.111.0 did show SRIOV but didn't detect the carrier - so we had two choices that did not work for us.

This post will cover how to enable SR-IOV on the 227.0.131.0 firmware.

Configure SR-IOV using bnxtnvm

Broadcom provides an application to configure the network cards. In our case we wanted specifically to enable SR-IOV. This can be done using the setoption command.

bnxtnvm --dev=enp129s0f0np0 setoption=enable_sriov#1

In our case we've been using Ubuntu Server but this should also apply to other distributions as the setting is adjusting a firmware setting in the device rather than a setting in the operating system.

Applying the change requires a reboot.

Adjust number of virtual functions

In many scenarios the default value of eight (8) virtual functions might not be sufficent (in our case we're planning for virtualization, this requires far more VFs).

Increasing the number of VFs is also straight forward.

bnxtnvm --dev=enp129s0f0np0 setoption=num_vf_per_pf:0#64
bnxtnvm --dev=enp129s0f0np0 setoption=num_vf_per_pf:1#64

The number of virtual functions can/must be configured per function (= per physical function which maps to a physical port). As we've a dual port card, we need to set it for each port.

Applying this change also requires a reboot.

Get a list of all options

The bnxtnvm utility provide a list to get all options and show help about these - this is quite awesome.

# list all options
bnxtnvm --dev=enp129s0f0np0 optionhelp=?

Name                              Scope
----------------                  ---------
an_protocol                       Port
autodetect_speed_exclude_mask     Port
afm_rm_resc_strategy              Device
bw_reservation                    Function
bw_reservation_valid              Function
bw_limit                          Function
bw_limit_valid                    Function
bw_in_percent                     Function
cos_precedence_order              Device
...

# get details for an option
bnxtnvm --dev=enp129s0f0np0 optionhelp=num_vf_per_pf
Name                   : num_vf_per_pf
Description            : Number of VFs per PF: Configures VF per PF relationship when in SR-IOV mode.
                         The given value should be a multiple of 8.
Option Type            : Multi Instance Type
Max Instance Indexes   : 0 to 15
Valid values           : 0 to 256

Having said that, it's obviously best practice to only settings if you need to do so :-)