When your application outgrows one server, the communication between the app’s components becomes a problem. The database might end up on a different physical server than the web app. If your infrastructure lacks a physical private net there’s a fast alternative: OpenvSwitch.
In this post I’ll explain how to use OpenVswitch to build a software-based private network across multiple physical hosts to connect guests that run on said hosts.
OpenvSwitch can build GRE tunnels to other hosts. Once the tunnel is established the hosts on the network can communicate directly and traffic isn’t routed through a central node like in OpenVPN.
In the diagram below there are two servers whose only connection is through the internet. Vhost A and Vhost B need to talk to each other but their services must not be exposed to the internet. We thus create a private network (the yellow box) with OpenvSwitch to allow the services to talk in private.
OpenvSwitch Setup for Host A
First we install the openvswitch package and enable the switch:
# pacman -S openvswitch
# systemctl enable openvswitch
# systemctl start openvswitch
Then we create the bridge and add a port through which all traffic flows and create the GRE tunnel to Host B:
# ovs-vsctl add-br ovs-br0
# ovs-vsctl add-port ovs-br0 ovs-br0p1 -- set interface ovs-br0p1 type=internal
# ovs-vsctl add-port vbridge0 vbridge0gre0 -- set interface vbridge0gre0 type=gre options:remote_ip=<Host B's public IP>
Finally Host A’s IP address can be set. I use netctl to setup networking in Arch Linux. The configuration for the OVS-Bridge’s IP goes into /etc/netctl/ovs-br0p1:
Description="Virtual Bridge 0"
Interface=ovs-br0p1
Connection=ethernet
IP=static
Address='192.168.11.1/24'
Once that’s done it’s time to fire up the interface:
# netctl enable ovs-br0p1
# netctl start ovs-br0p1
OpenvSwitch Setup for Host B
Same procedure as on Host A:
# pacman -S openvswitch
# systemctl enable openvswitch
# systemctl start openvswitch
Then we create the bridge and add a port through which all traffic flows and create the GRE tunnel to Host A:
# ovs-vsctl add-br ovs-br0
# ovs-vsctl add-port ovs-br0 ovs-br0p1 -- set interface ovs-br0p1 type=internal
# ovs-vsctl add-port vbridge0 vbridge0gre0 -- set interface vbridge0gre0 type=gre options:remote_ip=<Host A's public IP>
At this point (i.e. as soon as both endpoints have configured their side) the GRE tunnel is established between Host A and Host B.
As on Host A, we configure the bridge with an IP:
Description="Virtual Bridge 0"
Interface=ovs-br0p1
Connection=ethernet
IP=static
Address='192.168.11.2/24'
…and start the interface:
# netctl enable ovs-br0p1
# netctl start ovs-br0p1
Testing Connectivity
Now you should be able to ping 192.168.11.2 from Host A and 192.168.11.1 from Host B. When the network is established we can move on to configure networking for the virtual servers.
Setup for Vhost A
Edit Vhost A’s libvirt configuration:
# virsh edit vhost_a
And add the following XML-Code inside the <devices></devices> section:
<interface type='bridge'>
<mac address='52:54:00:6d:a2:50'/> <!-- use a unique MAC address here -->
<source bridge='ovs-br0'/>
<virtualport type='openvswitch' />
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> <!-- use a unique slot number -->
</interface>
Now you can configure the network on Vhost A. Create /etc/netctl/ens6 (or however your new device is named):
Interface=ens6
Connection=ethernet
IP=static
Address='192.168.11.101/24'
And finally enable and start the interface:
# netctl enable ens6
# netctl start ens6
Now you should be able to ping 192.168.11.1 (Host A) and 192.168.11.2 (Host B).
Setup for Vhost B
And finally set up Vhost B’s:
# virsh edit vhost_b
And add the following XML-Code inside the <devices></devices> section:
<interface type='bridge'>
<mac address='52:54:00:ad:b2:3f'/> <!-- use a unique MAC address here -->
<source bridge='ovs-br0'/>
<virtualport type='openvswitch' />
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> <!-- use a unique slot number -->
</interface>
Create the netctl configuration file
Interface=ens6
Connection=ethernet
IP=static
Address='192.168.11.102/24'
And finally enable and start the interface:
# netctl enable ens6
# netctl start ens6
Networking is now fully functional. You should be able to ping 192.168.11.1 (Host A) and 192.168.11.2 (Host B), 192.168.11.101 (Vhost A).
In order to add another physical host to the network, simply establish a GRE tunnel between any host that’s already connected to the network and the new host and configure the Vhosts as described.
Photo Credits
DJI_0531 - 2015-0703 夜間空拍台中市七期黎明路市政路口 by 準建築人手札網站 (cropped, colors mapped to monochrome range, licensed under CC BY-SA 2.0).