FreeBSD / pfSense slow network with Hyper-V
For some reason, it took me way too long to find information about this issue – so I’m writing another article that will hopefully help some people.
IT Ticket # 1337
Problem Description: Network performance through pfSense is extremely slow with Hyper-V
Software Versions: pfSense 2.6.0 and Windows Server 2022 (August 2022 patches)
Steps Taken Already: Confirmed hardware TCP offload is disabled; confirmed hardware checksum disabled; confirmed MTUs were correct.
Solution: Read on for the solution (hint: it’s a known driver bug, and for now only a workaround exists)
Why am I using pfSense and Hyper-V?
I like pfSense and use it teaching my students. Firewall concepts are universal, pfSense is powerful, and Hyper-V is a very capable hypervisor. This semester, I moved from pfSense 2.5.2 to 2.6.0 for both the lab infrastructure and the student systems. After the 2.6.0 installation, I immediately noticed the network performance through the firewall was atrocious! Speed tests were around 5 Megabits per second – how did we ever survive with DSL 😉
I moved from pfSense 2.5.2 to 2.6.0 […] speed tests were around 5 Megabits per second
I followed the usual steps: Google search “pfsense is slow with Hyper-V”. Lots of posts about disabling hardware TCP offloading, hardware checksum, and so on – but these are already disabled so not the problem.
Next was to verify the speed when directly connected to the WAN (it was 400+ Mbps), verify the speed from the firewall itself (still slow), and finally delete and recreate all vSwitches and VMs – no change.
There was a detour into MTUs (which was not the issue), and after 5 hours of searching and debugging I was no further ahead… until I somehow crafted the correct search query that found this gem on another Daniel’s blog. Huge thanks to them for posting this and saving my bacon.
The solution was out there… months ago
In that article from April 2022, the author describes that a bug exists between the latest FreeBSD 12.3 and Hyper-V’s implementation of software Receive Segment Coalescing (RSC) in Virtual Switches.
The impact? Fragmentation of small frames, which otherwise don’t need fragmentation, resulting is poor performance.
The bug didn’t exist in pfSense 2.5.2 because it used FreeBSD 12.2.
In the latest pfSense 2.6.0, it is using FreeBSD 12.3.
More specifically, the FreeBSD 12.3 driver for the Hyper-V virtual network adapter (hnX devices) needs an update to disable RSC offloading (March 2022). At the time of writing, it is unclear if this driver update is available in the FreeBSD 12.x stream.
The Workaround
The workaround is simple, but not ideal: disable software RSC for the Virtual Switch in Hyper-V.
It is a quick, reversible command – and incurs no downtime or network interruption!
Get-VMSwitch
Set-VMSwitch -Name <VirtualSwitchName> -EnableSoftwareRsc $false
Use the first command to list your virtual switches and their names, then apply the second command with the correct Virtual Switch Name to disable software RSC.
Results
Within seconds, the switch disables software RSC and the network performance returns.
I have tabulated my own results here:
pfSense Version / Software RSC | Enabled | Disabled |
2.5.2 | 410 Mbps | 440 Mbps |
2.6.0 | 6.8 Mbps | 430 Mbps |
In terms of raw throughput, the difference is night and day. There is even a modest improvement with the previous FreeBSD release, though we are probably within the accuracy tolerance of online speed tests.
Final Thoughts
I’m not sure why I had a hard time finding this information. We can all agree that Google has a knack for including “popular” results, not necessarily accurate results. Hopefully this article helps someone else with this issue!