[RESOLVED] FreeBSD Router, ping from vms to the internet work but TCP/UDP doesn't

Community Support Network
1

Hello,

I’m new to opennebula and I’m facing a really weird issue with a setup I’ve done a thousand times on multiple hypervisors, which makes me think the issue is on my opennebula setup.

I’ve restricted the setup to the minimum :

  • one Debian 11 kvm host
  • 2 bridges, (public / lan) and one vxlan interface for front/kvm communications
# bridge that holds kvm node public ip and used by public vms
auto brpublic0
iface brpublic0 inet static
    address <redacted>
    gateway <redacted>
    hwaddress ether <redacted>
    bridge-ports enp1s0f0
    bridge-stp off
    bridge-fd 0

# one service network (vxlan - scaleway rpn2)
# - communication between front-end/kvm nodes
auto enp1s0f1.2022
iface enp1s0f1.2022 inet static
    address 10.200.200.21/24

# one lan network
# - communication between vms
auto brlan0
iface brlan0 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
  • one virtual network in bridge mode bound to brlan0
  • one virtual network in bridge mode bound to brpublic0

Now for the vms :

  • one FreeBSD 13 (from the opennebula marketplace) with 2 nics (on the 2 Virtual networks)
    • Communication with the internet works fine (pings + tcp/udp)
    • Communication on the lan works fine
    • ip forwarding enabled
    • Smallest possible pf configuration for nat
scrub in all 
nat on vtnet1 inet from any to any -> (vtnet1)
  • the other one is a Debian 12 vm (from the opennebula marketplace) with one nic on brlan0
    • Communication with the router works fine (pings + tcp/udp)
    • pings to the internet work fine
    • X tcp/udp to the internet doesn’t work at all

I don’t really understand what’s going on.

  • the router can ping/curl/dig whatever on the internet and everything works
  • from the client VM, traceroute -Tn -p 53 9.9.9.9 works
  • but then everytime I try resolvectl query google.com 9.9.9.9 the request times out.

I thought it had something to do with packet sizes but ping -s 65000 9.9.9.9 works fine from the client VM so it’s not that. I tried to lower the mtus all the way (client → router → host bridges) to 1300 but it didn’t work either.

I did try the same setup with a VRouter instead of the freebsd vm but it didn’t work either.

I am not using any security group anywhere.

Here is a bunch of tcpdump traffic, as you can see the pings are coming back but udp dns queries aren’t. (REDACTED is the public IP of my router)

 00:00:05.925771 IP REDACTED > 9.9.9.9: ICMP echo request, id 52693, seq 1, length 64
 00:00:05.926975 IP 9.9.9.9 > REDACTED: ICMP echo reply, id 52693, seq 1, length 64
 00:00:06.927447 IP REDACTED > 9.9.9.9: ICMP echo request, id 52693, seq 2, length 64
 00:00:06.928719 IP 9.9.9.9 > REDACTED: ICMP echo reply, id 52693, seq 2, length 64
 00:00:07.929295 IP REDACTED > 9.9.9.9: ICMP echo request, id 52693, seq 3, length 64
 00:00:07.930558 IP 9.9.9.9 > REDACTED: ICMP echo reply, id 52693, seq 3, length 64
 00:00:08.930968 IP REDACTED > 9.9.9.9: ICMP echo request, id 52693, seq 4, length 64
 00:00:08.932188 IP 9.9.9.9 > REDACTED: ICMP echo reply, id 52693, seq 4, length 64
 00:00:09.932722 IP REDACTED > 9.9.9.9: ICMP echo request, id 52693, seq 5, length 64
 00:00:09.933998 IP 9.9.9.9 > 212.83.138.214: ICMP echo reply, id 52693, seq 5, length 64
 00:00:10.010870 IP REDACTED.50592 > 9.9.9.9.53: 22283+ A? 3.debian.pool.ntp.org. (39)
 00:00:10.010889 IP REDACTED.50592 > 9.9.9.9.53: 35383+ AAAA? 3.debian.pool.ntp.org. (39)
 00:00:10.934617 IP REDACTED > 9.9.9.9: ICMP echo request, id 52693, seq 6, length 64
 00:00:10.935891 IP 9.9.9.9 > REDACTED: ICMP echo reply, id 52693, seq 6, length 64
 00:00:11.936629 IP REDACTED > 9.9.9.9: ICMP echo request, id 52693, seq 7, length 64
 00:00:11.937909 IP 9.9.9.9 > REDACTED: ICMP echo reply, id 52693, seq 7, length 64
 00:00:15.016169 IP REDACTED.50592 > 9.9.9.9.53: 22283+ A? 3.debian.pool.ntp.org. (39)
 00:00:15.016185 IP REDACTED.50592 > 9.9.9.9.53: 35383+ AAAA? 3.debian.pool.ntp.org. (39)

Can someone help me narrow the issue ?

Thanks a lot !

Z

2

Hello,

I’ve been able to resolve this one this morning, if anyone passes by : this was an issue with freebsd and virtio drivers, freebsd activates some hardware offloading by default which is known to have an issue with virtio drivers. I don’t really know what I did with the vrouter appliance but I may have misconfigured stuff and deleted everything out of frustration :-).

Here is some Opnsense (freebsd based firewall/router) documentation on these settings Settings — OPNsense documentation and here is the 3 lines to add to /boot/loader.conf to resolve the issue :

hw.vtnet.tso_disable="1"
hw.vtnet.lro_disable="1"
hw.vtnet.csum_disable="1"

Thanks for the product !!

Z