When I configure a second network interface on CentOS7 (with OpenNebula), when the first one is already bridged, I get locked out of the machine

Community Support
1
  • CentOS 7.4
  • OpenNebula 5.4.6
  • both the front end (Sunstone) and back end (KVM) are on the same physical server
  • server has 2 network interfaces

I have followed carefully the install guide of OpenNebula and it all went very well, everything is working, etc. The problem comes when I decided to use the second network interface, as well. Different resources on the Internet suggest that it would be best to utilize both ethernet cards for optimal results. For example, one can be put to a virtual bridge and serve the networking needs of the virtual machines (in a separate VLAN), while the other can be used for administrative purposes, like accessing the web GUI or transferring the VM disk images (in a separate VLAN).

Everything works with just 1 lan card.

cat ifcfg-br0
-------------------
TYPE=Bridge
BOOTPROTO=static
IPV6INIT=no
NAME=br0
DEVICE=br0
ONBOOT=yes
DEFROUTE=yes
IPADDR=10.24.1.2
PREFIX=16
GATEWAY=10.24.1.1
DNS1=192.168.1.1
NM_CONTROLLED=no
--------------------

cat ifcfg-eno1
--------------------
TYPE=Ethernet
BOOTPROTO=none
IPV6INIT=no
NAME=eno1
DEVICE=eno1
UUID=c51defbe-c838-4cf9-92c4-ebd6ddf78fb4
ONBOOT=yes
BRIDGE=br0
NM_CONTROLLED=no
--------------------

But when I configure the 2-nd lan card, I lock myself out of the machine. More specifically, I can connect to the server on the 2-nd network interface only (enp5s2). I cannot connect on the 1-st interface (eno1) which is added to the bridge (br0).

cat ifcfg-enp5s2
-------------------------
TYPE=Ethernet
BOOTPROTO=static
IPV6INIT=no
NAME=enp5s2
DEVICE=enp5s2
UUID=c0e4b977-870e-4a2a-98cf-827ff6b0a148
ONBOOT=yes
DEFROUTE=yes
IPADDR=192.168.1.153
PREFIX=24
GATEWAY=192.168.1.200
DNS1=192.168.1.1
NM_CONTROLLED=no
-------------------------

If I take eno1 out of the bridge and disable the bridge, both network interfaces are accessible (i.e., the server is accessible on both IP addresses).

Why is that?

2

Hello,
Check network settings. I think problem with 2 default gateways.

GATEWAY=10.24.1.1
GATEWAY=192.168.1.200

1 Like
3

Hello, I think that @UAnton is right

4

I commented out GATEWAY= in the ifcfg-enp5s2 file and the result was the same. Tried the other file ifcfg-br0 - still the same. Any other thoughts?

By the way, please, note that everything works file when the eno1 is taken out of the virtual bridge (EVEN WITH BOTH GATEWAY=), so there must be something else.

5

If you disable gateway on fcfg-enp5s2, you need setup route for 192.168.1.0 network.

6

Well, OK, it seems I know very little about networking, then. :frowning: I guess, this is going to be one of those things that, if someone doesn’t tell me exactly how to do it step-by-step, I won’t figure it out.

So, is anyone willing to tell me just how to “setup route for 192.168.1.0 network”? I would appreciate that.

P.S.: Besides, this seems to be quite a standard setup. Judging by the Installation & Getting Started guide of OpenNebula, having 2 network interfaces is the norm. And one of them must be in a virtual bridge anyway. Then, CentOS is one of the couple supported OSes. It’s quite a popular one, also (I’d say a go-to option for virtual hosts, the other one being, maybe, Ubuntu Server). So I am amazed how I am the only one having this king of issue. I kept everything default and by the book. By the way, I had the exact same issue with OpenNebula 4.12.1 and that was the reason to go with just 1 network card then, because I didn’t have the time to investigate. Now I am setting up a brand new server with the latest OpenNebula and this time I have decided to track the issue down. I wasn’t sure where to ask - here, or on some CentOS related forum, or maybe on Stack Overflow, so I decided to start here first. The reason is that if I ask elsewhere I’d have to explain why one of the 2 interfaces is added in a virtual bridge. In the normal setup both network cards work OK. Asking here, I don’t have to explain that, because that’s what the documentation suggests.

7

Do you have a firewall on the server?

8

No, I don’t. Please, note that the server is accessible on both network interfaces WHEN eno1 IS TAKEN OUT OF THE BRIDGE br0 and the bridge is deleted i.e. the default configuration when you just install CentOS with two network cards.

So, for example, the following scenario WORKS AS EXPECTED:

cat ifcfg-eno1
--------------------
TYPE=Ethernet
BOOTPROTO=none
IPV6INIT=no
NAME=eno1
DEVICE=eno1
ONBOOT=yes
DEFROUTE=yes
IPADDR=10.24.1.2
PREFIX=16
GATEWAY=10.24.1.1
DNS1=192.168.1.1
NM_CONTROLLED=no
UUID=c51defbe-c838-4cf9-92c4-ebd6ddf78fb4
--------------------

cat ifcfg-enp5s2
-------------------------
TYPE=Ethernet
BOOTPROTO=static
IPV6INIT=no
NAME=enp5s2
DEVICE=enp5s2
ONBOOT=yes
DEFROUTE=yes
IPADDR=192.168.1.153
PREFIX=24
GATEWAY=192.168.1.200
DNS1=192.168.1.1
NM_CONTROLLED=no
UUID=c0e4b977-870e-4a2a-98cf-827ff6b0a148
-------------------------
9

The firewall was just a suggestion to check.

Could you post the output of ip route list and ip -d link list when the two interfaces are raw and when the bridge is configured? You said that you can reach the node via the second interface when the bridge is activated…

BR,
Anton

10 
ip route list
default via 192.168.1.200 dev enp5s2 
default via 10.24.1.1 dev eno1 proto static metric 100 
10.24.0.0/16 dev eno1 proto kernel scope link src 10.24.1.2 metric 100 
169.254.0.0/16 dev enp5s2 scope link metric 1002 
192.168.1.0/24 dev enp5s2 proto kernel scope link src 192.168.1.153 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

ip -d link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 
2: enp5s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether f4:6d:04:96:48:67 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether f4:6d:04:96:51:4b brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:a7:5d:f1 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    bridge forward_delay 200 hello_time 200 max_age 2000 ageing_time 30000 stp_state 1 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:54:0:a7:5d:f1 designated_root 8000.52:54:0:a7:5d:f1 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.53 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  283.06 vlan_default_pvid 1 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 4 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 addrgenmode eui64 
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:a7:5d:f1 brd ff:ff:ff:ff:ff:ff promiscuity 1 
    tun 
    bridge_slave state disabled priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:54:0:a7:5d:f1 designated_root 8000.52:54:0:a7:5d:f1 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on addrgenmode eui64
11

is it possible to show the output when the bridge is configured?

12 
ip route list
default via 10.24.1.1 dev br0 
10.24.0.0/16 dev br0 proto kernel scope link src 10.24.1.2 
169.254.0.0/16 dev enp5s2 scope link metric 1002 
169.254.0.0/16 dev br0 scope link metric 1004 
192.168.1.0/24 dev enp5s2 proto kernel scope link src 192.168.1.153 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

ip -d link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 addrgenmode eui64 
2: enp5s2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT qlen 1000
    link/ether f4:6d:04:96:48:67 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode eui64 
3: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT qlen 1000
    link/ether f4:6d:04:96:51:4b brd ff:ff:ff:ff:ff:ff promiscuity 1 
    bridge_slave state forwarding priority 32 cost 4 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.f4:6d:4:96:51:4b designated_root 8000.f4:6d:4:96:51:4b hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on addrgenmode eui64 
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT qlen 1000
    link/ether f4:6d:04:96:51:4b brd ff:ff:ff:ff:ff:ff promiscuity 0 
    bridge forward_delay 1500 hello_time 200 max_age 2000 ageing_time 30000 stp_state 0 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.f4:6d:4:96:51:4b designated_root 8000.f4:6d:4:96:51:4b root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    0.00 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  202.91 vlan_default_pvid 1 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 4 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 addrgenmode eui64 
5: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:a7:5d:f1 brd ff:ff:ff:ff:ff:ff promiscuity 0 
    bridge forward_delay 200 hello_time 200 max_age 2000 ageing_time 30000 stp_state 1 priority 32768 vlan_filtering 0 vlan_protocol 802.1Q bridge_id 8000.52:54:0:a7:5d:f1 designated_root 8000.52:54:0:a7:5d:f1 root_port 0 root_path_cost 0 topology_change 0 topology_change_detected 0 hello_timer    1.40 tcn_timer    0.00 topology_change_timer    0.00 gc_timer  208.03 vlan_default_pvid 1 group_fwd_mask 0 group_address 01:80:c2:00:00:00 mcast_snooping 1 mcast_router 1 mcast_query_use_ifaddr 0 mcast_querier 0 mcast_hash_elasticity 4 mcast_hash_max 512 mcast_last_member_count 2 mcast_startup_query_count 2 mcast_last_member_interval 100 mcast_membership_interval 26000 mcast_querier_interval 25500 mcast_query_interval 12500 mcast_query_response_interval 1000 mcast_startup_query_interval 3125 addrgenmode eui64 
6: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN mode DEFAULT qlen 1000
    link/ether 52:54:00:a7:5d:f1 brd ff:ff:ff:ff:ff:ff promiscuity 1 
    tun 
    bridge_slave state disabled priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x8001 port_no 0x1 designated_port 32769 designated_cost 0 designated_bridge 8000.52:54:0:a7:5d:f1 designated_root 8000.52:54:0:a7:5d:f1 hold_timer    0.00 message_age_timer    0.00 forward_delay_timer    0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on addrgenmode eui64
13

So to clarify:

  • When you bring br0 up, the default route is replaced with 10.24.1.1 (br0) and you can connect only to 192.168.1.153 via enp5s2
  • The working state works via default route 192.168.1.200 (enp5s2).

There is nothing suspicious beside the doubled default route. So the issue could be on the other end - in the router or the workstation. Can you check the router for suspicious packets, is rp_filter enabled(but it should work against the claimed working state though), etc…

14

When you bring br0 up, the default route is replaced with 10.24.1.1 (br0) and you can connect only to 192.168.1.153 via enp5s2

Yes.

The working state works via default route 192.168.1.200 (enp5s2).

I guess so.

Can you check the router for suspicious packets, is rp_filter enabled

I would have to understand what rp_filter is in the 1-st place, but yes, I will check that. By the way, I forgot to mention that in both cases reverse access IS possible, i.e. I can ssh FROM the node TO my workstation in both scenarios.