freebsd 12.3 / pfsense 2.6 on hyper-v with low throughput

Having upgraded some pfsense nodes running on Hyper-V (Windows Server 2022) we've had quite some issues accessing some services provided by these nodes.

An example is a webservice for an internal system that spent 30-60 seconds delivering pages that have been delivered in less than 5 seconds before. Because the service is backed by a rather large database (timescaledb with around 1.5TB of data) we expected the backend to be root of all evil.

After a while, analysis revealed the following

tcpdump -i eth0 "icmp"
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:30:34.711816 IP _gateway > n-mon-zxsrv0: ICMP 10.50.24.18 unreachable - need to frag (mtu 1500), length 176
12:30:34.711816 IP _gateway > n-mon-zxsrv0: ICMP 10.50.24.18 unreachable - need to frag (mtu 1500), length 176
12:30:34.822246 IP _gateway > n-mon-zxsrv0: ICMP 10.50.24.18 unreachable - need to frag (mtu 1500), length 176
12:30:35.924826 IP _gateway > n-mon-zxsrv0: ICMP 10.50.24.18 unreachable - need to frag (mtu 1500), length 176
12:30:36.314943 IP _gateway > n-mon-zxsrv0: ICMP 10.50.24.18 unreachable - need to frag (mtu 1500), length 176
12:30:36.437240 IP _gateway > n-mon-zxsrv0: ICMP 10.50.24.18 unreachable - need to frag (mtu 1500), length 176

So, we've small packets but need fragmentation?

FreeBSD 12.3 and rsc

After some research we found out that the issue comes from a combination of two circumstances:

• freebsd12.3 running as a Hyper-V guest
• Hyper-V VM Switch having SoftwareRsc enabled (provided by Windows Server 2019 and newer)

The combination of those results in incorrect behavior.

The not solution

It would be nice to just tell the guest to disable RSC (using a sysctl for example). Unfortunately the freebsd hyper-v nic driver does not allow this - yet.

The issue is known and will be fixed in newer releases (âš™ D34507 added changes for rsc handler (freebsd.org)), for now we need to go another way.

The workaround

As there's currently no real solution, we can only apply a workaround by disabling RSC on the whole VMSwitch - yes, on the switch itself.

So this affects not only the freebsd guests, it affects all. Depending on your environment you may decide to perform either a downgrade of the freebsd guest (if possible) or disable RSC until a fix is availabe.

Set-VMSwitch -Name <VMSwitchName> -EnableSoftwareRsc $false

Applying this won't require a reboot nor will it affect your network workload, it will just take a few seconds to be applied in the background.

Hopefully a fix is released soon, so that we can reenable rsc again.

If you're interested on why you should want to leave RSC enabled: Receive Segment Coalescing (RSC) in the vSwitch