Contribution: Vrouter enhacements and load ballancing

Development
1

Hello, I like to contribute OpenNebula community with my work on virtual routers and load ballancing.

This contribution consist of several parts:

  1. Updated context scripts for Alpine linux with support for keepalived load ballancing
  2. Alpine linux 3.5 qcow2 image ready to use
  3. Proposal of changes resp. improvements should be done in OpenNebula

1. Context scripts

Updated context scripts can be found on github

New features:

  1. NAT iptables masquerading enabled by new context var ETHx_VROUTER_GATEWAY = "YES"
  2. Keepalived email notifications:
    • VROUTER_KEEPALIVED_NOTIFY_EMAIL - required to enable this feature
    • VROUTER_KEEPALIVED_FROM_EMAIL - optional, if not defined emails are send from NOTIFY_EMAIL address
  3. Keepalived can define sync interface using var ETHx_VROUTER_KEEPALIVED_INF - usefull if you need save some public IPv4 addresses or just want all vrrp traffix on private interface. For saving addresses need integration in OpenNebula
  4. Keepalived Load Ballancing support
    • both NAT and DR (Direct Routing)
    • Single and Multi Port support using firewall marks
    • FTP Passive support using multiport, fwmarks and ip_vs_ftp module
    • Context vars in following format VROUTER_LB0_DEV, VROUTER_LB0_SERVER0_IP, VROUTER_LB0_SERVER0_CHECK0_TYPE
    • Limitation of 9 LoadBallancers on one vrouter instance each with max 9 real servers. This can be enhanced in context scripts in future.
  5. Iptables and Keepalived IPv6 support - for this time just in dual stack IPv4+IPv6

Specifications of new vars:

1. NAT

ETHx_VROUTER_GATEWAY - by defining var with value YES on specified interface enable iptables masquerading. As source address are IPs from all other interfaces.

Example:
We have three interfaces with IP addresses ETH0, ETH1 - 192.168.1.254 and ETH2 - 192.168.2.254.
We enable masquerading on ETH0 by setting context var ETH0_VROUTER_GATEWAY = "YES"

This generate following IP tables:

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 192.168.1.0/24 -o eth0 -j MASQUERADE
-A POSTROUTING -s 192.168.2.0/24 -o eth0 -j MASQUERADE
COMMIT

2. Notifiacation emails

VROUTER_KEEPALIVED_NOTIFY_EMAIL - by defining this var, global_defs section is created in keepalived.conf file. If you need define more emails, just separate it by space.
VROUTER_KEEPALIVED_FROM_EMAIL - this var just set email FROM address.

Generated example:

global_defs {
  notification_email {
    `VROUTER_KEEPALIVED_NOTIFY_EMAIL`
  }
  notification_email_from `VROUTER_KEEPALIVED_FROM_EMAIL`
  smtp_server 127.0.0.1
  smtp_connect_timeout 30
  router_id Vrouter_`VROUTER_ID`_`ETH0_IP`
}

3. Keepalived sync interface

In some case you need to define keepalive vrrp sync interface. In my case was for saving public IPv4 addresses in other you just don’t want vrrp traffic on public network…

ETHx_VROUTER_KEEPALIVED_INF with value of name of interface on which vrrp sync traffic should be.

Example: ETH0_VROUTER_KEEPALIVED_INF = "eth1"

This generate keepalived instance config like this

vrrp_instance VI_eth0 {
  ...
  interface eth1 # sync interface set to eth1
  virtual_router_id 45
  ...
  virtual_ipaddress {
    192.168.1.1/24 dev eth0 # vip interface on eth0
  }
}
vrrp_instance VI_eth1 {
  ...
  interface eth1
  virtual_router_id 46
  ...
  virtual_ipaddress {
    192.168.2.1/24 dev eth1
  }
}

4. Keepalived Load Ballancing

I implemented load ballancing and tested both modes NAT and DR. You define it by context vars in following manner:

VROUTER_LBx_* - first level - define Load Ballancers(LB)
VROUTER_LBx_SERVERy_* - second level - define Real Servers(RS) for each LB
VROUTER_LBx_SERVERy_CHECKz_* - third level - define health checks for each RS

where x,y,z are integers counting from zero

Example 1 NAT mode single port

NAT Mode schema
keepalived2.png634×588 64.3 KB 

VROUTER_LB0_ALGO = "rr",
VROUTER_LB0_DELAY_LOOP = "6",
VROUTER_LB0_DEV = "ETH0", # Interface of Virtual IP public address
VROUTER_LB0_KIND = "NAT",
VROUTER_LB0_PORT = "80",
VROUTER_LB0_PROTOCOL = "TCP",

VROUTER_LB0_SERVER0_IP = "10.0.0.71",
VROUTER_LB0_SERVER0_CHECK0_TYPE = "HTTP_GET",

VROUTER_LB0_SERVER1_IP = "10.0.0.72",
VROUTER_LB0_SERVER1_CHECK0_TYPE = "HTTP_GET"

This generate keepalived virtual service config

virtual_server `VIP of LB0_DEV` 80 {
  delay_loop 6
  lb_algo rr
  lb_kind NAT
  protocol TCP
  real_server 10.0.0.71 80 {
    HTTP_GET {
      connect_port 80
      connect_timeout 10
      url {
        path /
        status_code 200
      }
    }
  }
  real_server 10.0.0.72 80 {
    HTTP_GET {
      connect_port 80
      connect_timeout 10
      url {
        path /
        status_code 200
      }
    }
  }
}
Example 2 NAT mode multi port HTTP+SSL

We add second port to LB0 config. Multiple ports separate by space. We also added second health check for SSL connection

...
VROUTER_LB0_PORT = "80 443", # Multiple ports separate by space
...

VROUTER_LB0_SERVER0_IP = "10.0.0.71",
VROUTER_LB0_SERVER0_CHECK0_TYPE = "HTTP_GET",
VROUTER_LB0_SERVER0_CHECK1_TYPE = "SSL_GET",

VROUTER_LB0_SERVER1_IP = "10.0.0.72",
VROUTER_LB0_SERVER1_CHECK0_TYPE = "HTTP_GET",
VROUTER_LB0_SERVER1_CHECK1_TYPE = "SSL_GET"

Generated config:

virtual_server fwmark 80 { # in multiport config we use firewall marks
  ...
  real_server 10.0.0.71 {
    HTTP_GET {
      connect_port 80
      connect_timeout 10
      url {
        path /
        status_code 200
      }
    }
    SSL_GET {
      connect_port 443
      connect_timeout 10
      url {
        path /
        status_code 200
      }
    }
  }
  ...
}

and iptables:

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -d `VIP_of_LB0_DEV`/32 -p tcp -m multiport --dports 80,443 -j MARK --set-mark 80
Example 3 NAT mode multi port FTP Passive

FTP uses control port 21 and passive port range, which can be configured in FTP daemon. For this example we use passive port range 10000:20000

...
VROUTER_LB0_PORT = "21 10000:20000", # Multiple ports separate by space
...

VROUTER_LB0_SERVER0_IP = "10.0.0.71",
VROUTER_LB0_SERVER0_CHECK0_PORT = "21",

VROUTER_LB0_SERVER1_IP = "10.0.0.72",
VROUTER_LB0_SERVER1_CHECK0_PORT = "21",

We removed health check type, so default TCP_CHECK will be used, also we added check port.

Generated config:

virtual_server fwmark 21 { # in multiport config we use firewall marks
  ...
  real_server 10.0.0.71 {
    TCP_CHECK {
      connect_port 21
      connect_timeout 10
    }
  }
  real_server 10.0.0.72 {
    TCP_CHECK {
      connect_port 21
      connect_timeout 10
    }
  }
  ...
}

and iptables:

*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -d `VIP_of_LB0_DEV`/32 -p tcp --dport 21 -j MARK --set-mark 21
-A PREROUTING -d `VIP_of_LB0_DEV`/32 -p tcp --dport 10000:20000 -j MARK --set-mark 21
Example 4 Direct Routing mode

DR mode
keepalived3.png549×506 55.6 KB

In this mode load ballancer just balance incomming request and outgoing replies are send directly from real server, so performance of network is better by relegating work from LB.

VROUTER_LB0_ALGO = "rr",
VROUTER_LB0_DELAY_LOOP = "6",
VROUTER_LB0_DEV = "ETH0", # Interface of Virtual IP public address
VROUTER_LB0_KIND = "DR", # Chnaged to DR
VROUTER_LB0_PORT = "80",
VROUTER_LB0_PROTOCOL = "TCP",

VROUTER_LB0_SERVER0_IP = "10.0.0.71",
VROUTER_LB0_SERVER0_CHECK0_TYPE = "HTTP_GET",

VROUTER_LB0_SERVER1_IP = "10.0.0.72",
VROUTER_LB0_SERVER1_CHECK0_TYPE = "HTTP_GET"

This mode also requires configuration of real servers. In short you need to add arp filtering and assing VIP:

arptables -A INPUT -d <virtual ip> -j DROP
arptables -A OUTPUT -s <virtual ip> -j mangle --mangle-ip-s <real ip of server>

ip addr add <virtual ip>/24 dev eth0

Further reading about this mode

5. IPv6 Support

I added support for IPv6 addresses in network config, DNS config, Ip6tables and keepalived. If you network has enabled IP4/IP6 addressing, keepalived instance config should looks like this:

vrrp_instance VI_eth0 {
  ...
  virtual_ipaddress {
    192.168.1.1/24 dev eth0
  }
  virtual_ipaddress_excluded {
    fd59:c4c5:6005::/64 dev eth0
  }
}

For this time IPv6 in keepalived works just in dual stack IPv4/IPv6 config. In future this can be improved by extending context scripts.

2. Alpine linux appliance

I created pull request for adding image to market place. You can also download qcow2 image and use it manually.

Appliance is in marketplace

3. Proposal of changes in OpenNebula

  1. Virtual Router - Add support by adding checkbox for set interface as gateway
    • ETHx_VROUTER_GATEWAY
  2. Virtual Router - Add support by adding email notifications fields
    • VROUTER_KEEPALIVED_NOTIFY_EMAIL
    • VROUTER_KEEPALIVED_FROM_EMAIL
  3. Virtual Router - Add support for configure sync interface and also not allocate addresses if sync is on other interface, but just VIP
    • ETHx_VROUTER_KEEPALIVED_INF
  4. Virtual Router - Add support for define Load Ballancers
    • VROUTER_LBx_ALGO
    • VROUTER_LBx_DELAY_LOOP
    • VROUTER_LBx_DEV
    • VROUTER_LBx_KIND
    • VROUTER_LBx_PORT
    • VROUTER_LBx_PROTOCOL
    • VROUTER_LBx_SERVERy_IP
    • VROUTER_LBx_SERVERy_PORT
    • VROUTER_LBx_SERVERy_WEIGHT
    • VROUTER_LBx_SERVERy_CHECKz_TYPE
    • VROUTER_LBx_SERVERy_CHECKz_PORT
    • VROUTER_LBx_SERVERy_CHECKz_TIMEOUT
  5. Services - Add support for virtual routers / load ballancers
    • also add support by some context var to real servers with direct routing LB mode, so can be updated context scripts which add arp filtering and VIP to interface.
  6. Virtual Routers - remove keepalived_id - no need this, we can relay on vrouter_id. VRRP communication is based on VIP and ID have effect only if multiple instances are running on same NIC
  7. Virtual Routers - keepalived_password should be auto generated

Request created in redmine

8 Likes
2

Wow! Thanks for the contribution and the comprehensive documentation. We will review and add the image to the marketplace as soon as possible. Yours is the first contribution to the new marketplace and we still have to iron out the process. :wink:

Would you like to post in our blog? I think we can use the text from this topic almost as is.

Thanks again!

3

Hello, thank you for recognition :slight_smile: I am also think about blog post, but it should contains more informations and more real world examples, so I am open to write blog post about this.

My contrib also need testing by other devs before posting blog post, to prevent some bugs :slight_smile:

What you think about proposed changes? I think that blog post will be better, if some changes will be incorporated in opennebula for easier configuration.

4

We’ve been discussing internally abut the change requests. Right now we are full steam working on the next release so we are going to prioritize the variables that can not be added or are cumbersome in other ways. Mainly the ones related with nics (ETHx_*)

This will be added to the nic parameters.

These variables can be easily added using user inputs or custom variables.

This will also be added to the interface configuration.

This won’t probably make the next release so it should be added manually as custom attributes.

We are working on vm groups for the next release that will be more suited for these kind of things. This will be evaluated again when the feature is working.

I’ve added the parameter in case the admin needs to specify it for some reason. Maybe to connect it to an external machine not managed by OpenNebula. We should set this value to vrouter id by default.

We are going to add the possibility to auto generate passwords. Maybe make it the default option.

What do you think about this?

1 Like
5

Hello, I understant that you have lot of work and I also pleased that you want implement at least something.

If I can I glad to help with integration that configuration to sunstone.

I also created pull request to alpine-context repo, please check it whether you are ok with that.

Thanks.

6

Github issue