You've successfully subscribed to Nuvotex Blog
Great! Next, complete checkout for full access to Nuvotex Blog
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.
Success! Your billing info is updated.
Billing info update failed.

Enable SR-IOV on Broadcom NICs and configure VF count

Running broadcom network cards using SR-IOV might require configuration of the cards firmware. This also applies for example if you want to adjust the total number of virtual functions.

Daniel Nachtrub
Daniel Nachtrub

On a test-setup we're evaluating Broadcom NetXtreme-E (2x100G) interfaces. The networking will be wired up on the system using openvswitch (ovs) as it's easy to handle and OVS can utilize SR-IOV and hardware offloading.

An issue we had, is that during testing, the Broadcom cards didn't offer sriov configuration (running Firmware Upgrading to firmware did show SRIOV but didn't detect the carrier - so we had two choices that did not work for us.

This post will cover how to enable SR-IOV on the firmware.

Configure SR-IOV using bnxtnvm

Broadcom provides an application to configure the network cards. In our case we wanted specifically to enable SR-IOV. This can be done using the setoption command.

bnxtnvm --dev=enp129s0f0np0 setoption=enable_sriov#1
configure SR-IOV using setoption

In our case we've been using Ubuntu Server but this should also apply to other distributions as the setting is adjusting a firmware setting in the device rather than a setting in the operating system.

Applying the change requires a reboot.

Adjust number of virtual functions

In many scenarios the default value of eight (8) virtual functions might not be sufficent (in our case we're planning for virtualization, this requires far more VFs).

Increasing the number of VFs is also straight forward.

bnxtnvm --dev=enp129s0f0np0 setoption=num_vf_per_pf:0#64
bnxtnvm --dev=enp129s0f0np0 setoption=num_vf_per_pf:1#64
set number of VFs

The number of virtual functions can/must be configured per function (= per physical function which maps to a physical port). As we've a dual port card, we need to set it for each port.

Applying this change also requires a reboot.

Get a list of all options

The bnxtnvm utility provide a list to get all options and show help about these - this is quite awesome.

# list all options
bnxtnvm --dev=enp129s0f0np0 optionhelp=?

Name                              Scope
----------------                  ---------
an_protocol                       Port
autodetect_speed_exclude_mask     Port
afm_rm_resc_strategy              Device
bw_reservation                    Function
bw_reservation_valid              Function
bw_limit                          Function
bw_limit_valid                    Function
bw_in_percent                     Function
cos_precedence_order              Device

# get details for an option
bnxtnvm --dev=enp129s0f0np0 optionhelp=num_vf_per_pf
Name                   : num_vf_per_pf
Description            : Number of VFs per PF: Configures VF per PF relationship when in SR-IOV mode.
                         The given value should be a multiple of 8.
Option Type            : Multi Instance Type
Max Instance Indexes   : 0 to 15
Valid values           : 0 to 256
get options and details

Having said that, it's obviously best practice to only settings if you need to do so :-)


Daniel Nachtrub

Kind of likes computers. Linux foundation certified: LFCS / CKA / CKAD / CKS. Microsoft certified: Cybersecurity Architect Expert & Azure Solutions Architect Expert.