An unusual suggestion:
vxlans rely on UDP MultiCast to limit layer 2 broadcasting.
This works nicely on relatively simple network topologies, or if you have nice, powerful switches that support lots of multicast groups. However, a multi-tennant environment with an allocation of vxlans-per-client is likely to generate a lot of vxlans, and on OpenNebula each vxlan is assigned to it’s own Multicast group.
With millions of vxlans available, this doesn’t scale brilliantly.
Suggestion 1: Best for scale-out
Allow consolidation of vxlans into Multicast groups. This could be grouped by “tennant”, constraining multicast traffic only to hosts with that tennant’s VMs running (even if the specific vxlan isn’t required, this would still significantly reduce the number of multicast groups, as well as still constraining traffic).
Suggestion 2: Best for smaller tennants on larger networks
‘head-end’ replication: in a software-defined environment, this is a highly elegant solution - set a target number of vxlans per vtep, and then as VMs are deployed / migrate the OpenNebula management plane updates the bridging information across the cluster eg:
bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:2::1
bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:3::100
Whilst this isn’t optimal for vxlans with hundreds of participating hosts (each host must send each frame to each listed host) in a multi-tennant environment the tennants traffic is likely to be constrained to a smaller number of hosts.
This removes the need for switches to actively partake in IGMP snooping and general Multicast behaviour - instead UDP traffic is sent point-to-point between hosts. I accept this adds significant load to the host interface (frames = traffic x number of participating hosts) but with 40Gbs and 100Gbs networking now approaching the commodity end of the market, this seems less and less of an issue than managing and fault-finding dynamic multicast groups.
It might also prove to be an extremely powerful tool for anyone building AWS environments - as I believe it would allow the construction of virtual networks between AWS hosts.