{"id":1901,"date":"2022-09-26T10:36:40","date_gmt":"2022-09-26T14:06:40","guid":{"rendered":"https:\/\/blog.danjoannis.com\/?p=1901"},"modified":"2024-04-11T19:46:29","modified_gmt":"2024-04-11T23:16:29","slug":"freebsd-pfsense-slow-network-with-hyper-v","status":"publish","type":"post","link":"https:\/\/blog.danjoannis.com\/?p=1901","title":{"rendered":"FreeBSD \/ pfSense slow network with Hyper-V"},"content":{"rendered":"\n
\"\"<\/a><\/figure>\n\n\n\n

For some reason, it took me way too long to find information about this issue – so I’m writing another article that will hopefully help some people.<\/p>\n\n\n\n

IT Ticket # 1337<\/strong><\/p>\n\n\n\n

Problem Description<\/strong>: Network performance through pfSense is extremely slow with Hyper-V<\/p>\n\n\n\n

Software Versions<\/strong>: pfSense 2.6.0 and Windows Server 2022 (August 2022 patches)<\/p>\n\n\n\n

Steps Taken Already<\/strong>: Confirmed hardware TCP offload is disabled; confirmed hardware checksum disabled; confirmed MTUs were correct.<\/p>\n\n\n\n

Solution<\/strong>: Read on for the solution (hint: it’s a known driver bug, and for now only a workaround exists)<\/span><\/p>\n\n\n\n\n\n\n\n

Why am I using pfSense and Hyper-V?<\/h2>\n\n\n\n

I like pfSense and use it teaching my students. Firewall concepts are universal, pfSense is powerful, and Hyper-V is a very capable hypervisor. This semester, I moved from pfSense 2.5.2 to 2.6.0 for both the lab infrastructure and the student systems. After the 2.6.0 installation, I immediately noticed the network performance through the firewall was atrocious! Speed tests were around 5 Megabits<\/em> per second – how did we ever survive with DSL \ud83d\ude09<\/p>\n\n\n\n

\n

I moved from pfSense 2.5.2 to 2.6.0 […] speed tests were around 5 Megabits<\/em> per second<\/p>\n<\/blockquote>\n\n\n\n

I followed the usual steps: Google search “pfsense is slow with Hyper-V”. Lots of posts about disabling hardware TCP offloading, hardware checksum, and so on – but these are already disabled so not the problem.<\/p>\n\n\n\n

Next was to verify the speed when directly connected to the WAN (it was 400+ Mbps), verify the speed from the firewall itself (still slow), and finally delete and recreate all vSwitches and VMs – no change.<\/p>\n\n\n\n

There was a detour into MTUs (which was not the issue), and after 5 hours of searching and debugging I was no further ahead… until I somehow crafted the correct search query that found this gem on another Daniel’s blog<\/a>. Huge thanks to them for posting this and saving my bacon.<\/p>\n\n\n\n

The solution was out there… months ago<\/h2>\n\n\n\n

In that article from April 2022, the author describes that a bug exists between the latest FreeBSD 12.3 and Hyper-V’s implementation of software Receive Segment Coalescing (RSC) in Virtual Switches.<\/p>\n\n\n\n

The impact? Fragmentation of small frames, which otherwise don’t need fragmentation, resulting is poor performance.<\/p>\n\n\n

\n
\"\"<\/a>
All up to date it seems<\/figcaption><\/figure>\n<\/div>\n\n\n

The bug didn’t exist in pfSense 2.5.2 because it used FreeBSD 12.2.<\/p>\n\n\n\n

In the latest pfSense 2.6.0, it is using FreeBSD 12.3.<\/p>\n\n\n\n

More specifically, the FreeBSD 12.3 driver for the Hyper-V virtual network adapter (hnX devices) needs an update to disable RSC offloading (March 2022).<\/a> At the time of writing, it is unclear if this driver update is available in the FreeBSD 12.x stream.<\/p>\n\n\n\n

The Workaround<\/h2>\n\n\n\n

The workaround is simple, but not ideal: disable software RSC for the Virtual Switch in Hyper-V.<\/p>\n\n\n\n

It is a quick, reversible command – and incurs no downtime or network interruption!<\/p>\n\n\n\n

Get-VMSwitch\nSet-VMSwitch -Name <VirtualSwitchName> -EnableSoftwareRsc $false<\/code><\/pre>\n\n\n\n
Use the first command to list your virtual switches and their names, then apply the second command with the correct Virtual Switch Name to disable software RSC.<\/p>\n\n\n\n
Results<\/h2>\n\n\n\n
Within seconds, the switch disables software RSC and the network performance returns. <\/p>\n\n\n\n
\"\"<\/a>
Make sure to apply the command to all Virtual Switches used by pfSense<\/figcaption><\/figure>\n\n\n\n
I have tabulated my own results here:<\/p>\n\n\n\n
pfSense Version \/ Software RSC<\/strong><\/em><\/td>Enabled<\/strong><\/td>Disabled<\/strong><\/td><\/tr>
2.5.2<\/strong><\/td>410 Mbps<\/td>440 Mbps<\/td><\/tr>
2.6.0<\/strong><\/td>6.8 Mbps<\/td>430 Mbps<\/td><\/tr><\/tbody><\/table>
Test Results – Before and After Disabling Software RSC<\/figcaption><\/figure>\n\n\n\n
In terms of raw throughput, the difference is night and day. There is even a modest improvement with the previous FreeBSD release, though we are probably within the accuracy tolerance of online speed tests.<\/p>\n\n\n\n
Final Thoughts<\/h2>\n\n\n\n
I’m not sure why I had a hard time finding this information. We can all agree that Google has a knack for including “popular” results, not necessarily accurate results. Hopefully this article helps someone else with this issue!<\/p>\n","protected":false},"excerpt":{"rendered":"
For some reason, it took me way too long to find information about this issue – so I’m writing another article that will hopefully help some people. IT Ticket # 1337 Problem Description: Network performance through pfSense is extremely slow with Hyper-V Software Versions: pfSense 2.6.0 and Windows Server 2022 (August 2022 patches) Steps Taken […]<\/p>\n","protected":false},"author":1,"featured_media":1910,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[285,9,138],"tags":[346,347,109],"class_list":["post-1901","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-networking","category-software","category-work","tag-firewall","tag-performance","tag-server"],"_links":{"self":[{"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=\/wp\/v2\/posts\/1901","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1901"}],"version-history":[{"count":6,"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=\/wp\/v2\/posts\/1901\/revisions"}],"predecessor-version":[{"id":1948,"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=\/wp\/v2\/posts\/1901\/revisions\/1948"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=\/wp\/v2\/media\/1910"}],"wp:attachment":[{"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1901"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1901"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.danjoannis.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1901"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}