FreeBSD / pfSense slow network with Hyper-V

For some reason, it took me way too long to find information about this issue – so I’m writing another article that will hopefully help some people.

IT Ticket # 1337

Problem Description: Network performance through pfSense is extremely slow with Hyper-V

Software Versions: pfSense 2.6.0 and Windows Server 2022 (August 2022 patches)

Steps Taken Already: Confirmed hardware TCP offload is disabled; confirmed hardware checksum disabled; confirmed MTUs were correct.

Solution: Read on for the solution (hint: it’s a known driver bug, and for now only a workaround exists)

Why am I using pfSense and Hyper-V?

I like pfSense and use it teaching my students. Firewall concepts are universal, pfSense is powerful, and Hyper-V is a very capable hypervisor. This semester, I moved from pfSense 2.5.2 to 2.6.0 for both the lab infrastructure and the student systems. After the 2.6.0 installation, I immediately noticed the network performance through the firewall was atrocious! Speed tests were around 5 Megabits per second – how did we ever survive with DSL 😉

I moved from pfSense 2.5.2 to 2.6.0 […] speed tests were around 5 Megabits per second

I followed the usual steps: Google search “pfsense is slow with Hyper-V”. Lots of posts about disabling hardware TCP offloading, hardware checksum, and so on – but these are already disabled so not the problem.

Next was to verify the speed when directly connected to the WAN (it was 400+ Mbps), verify the speed from the firewall itself (still slow), and finally delete and recreate all vSwitches and VMs – no change.

There was a detour into MTUs (which was not the issue), and after 5 hours of searching and debugging I was no further ahead… until I somehow crafted the correct search query that found this gem on another Daniel’s blog. Huge thanks to them for posting this and saving my bacon.

The solution was out there… months ago

In that article from April 2022, the author describes that a bug exists between the latest FreeBSD 12.3 and Hyper-V’s implementation of software Receive Segment Coalescing (RSC) in Virtual Switches.

The impact? Fragmentation of small frames, which otherwise don’t need fragmentation, resulting is poor performance.

All up to date it seems

The bug didn’t exist in pfSense 2.5.2 because it used FreeBSD 12.2.

In the latest pfSense 2.6.0, it is using FreeBSD 12.3.

More specifically, the FreeBSD 12.3 driver for the Hyper-V virtual network adapter (hnX devices) needs an update to disable RSC offloading (March 2022). At the time of writing, it is unclear if this driver update is available in the FreeBSD 12.x stream.

The Workaround

The workaround is simple, but not ideal: disable software RSC for the Virtual Switch in Hyper-V.

It is a quick, reversible command – and incurs no downtime or network interruption!

Get-VMSwitch
Set-VMSwitch -Name <VirtualSwitchName> -EnableSoftwareRsc $false

Use the first command to list your virtual switches and their names, then apply the second command with the correct Virtual Switch Name to disable software RSC.

Results

Within seconds, the switch disables software RSC and the network performance returns.

Make sure to apply the command to all Virtual Switches used by pfSense

I have tabulated my own results here:

pfSense Version / Software RSCEnabledDisabled
2.5.2410 Mbps440 Mbps
2.6.06.8 Mbps430 Mbps
Test Results – Before and After Disabling Software RSC

In terms of raw throughput, the difference is night and day. There is even a modest improvement with the previous FreeBSD release, though we are probably within the accuracy tolerance of online speed tests.

Final Thoughts

I’m not sure why I had a hard time finding this information. We can all agree that Google has a knack for including “popular” results, not necessarily accurate results. Hopefully this article helps someone else with this issue!

Tags: , ,

Monday, September 26th, 2022 networking, software, work

Leave a Reply