I want to create two VM on two separate hosts (one per host) that reside on the same network. I also want the VMs to reside in a different network than the host's network. To do this with openvswitch is really easy.
- Make sure openvswitch and tun/tap are compiled in kernel
- install openvswitch tools
- create bridge on each VMs
- create masquerading rules to share internet connection with VM network
Openvswitch is already part of the kernel. It needs to be enabled when compiling the kernel. Then the tools need to be installed. On slackware, it was just a matter of ./configure, make, make install.
Creating a bridge with GRE tunnel
To have both VMs communicate with each other, we can build a virtual switch that spans accross the two hosts. This is done with a bridge containing a GRE tunnel. Creating a bridge on each VMs is done as follow. Unless noted otherwise, these commands should be re-executed after every reboot
# create tun/tap adapters /usr/sbin/tunctl -t tap7 # launch openvswitch server ovsdb-server --detach --remote=punix:/usr/local/var/run/openvswitch/db.sock --remote=ptcp:6640 ovs-vsctl --no-wait init ovs-vswitchd --pidfile --log-file --detach # Creating the bridge. Adding tap interface and GRE tunnel in the bridge # This is the exception: the following only needs to be done once as it will be persistant /usr/local/bin/ovs-vsctl add-br br1 /usr/local/bin/ovs-vsctl add-port br1 tap7 /usr/local/bin/ovs-vsctl add-port br1 gre0 -- set interface gre0 type=gre options:remote_ip=**IP_ADDRESS_OF_OTHER_HOST** # bring the bridge up /sbin/ifconfig br1 0.0.0.0 promisc up /sbin/ifconfig tap7 0.0.0.0 promisc up
Then repeat the same on the other host
Now, it is possible to start a VM on each host using their respective TAP7 interface. Each vm is assigned an IP address in the same subnet but different than the one of the hosts. In my case I used 192.168.2.2 and 192.168.2.9. Because a GRE interface is present in the bridges, both VM can communicate with each other now.
Access to the internet
Since the VMs reside on subnet 192.168.2.0 but my internet gateway is at 192.168.1.1, there is no way for my VMs to communicate with the outside world. So the first thing that needs to be done, is to assign an ip address to the bridge of one of the host. This will be the gateway. So obviously, only one is needed. I chose to put it on host2. So the "br1" on host1 still has no ip address assigned to it. On host2 I did:
ifconfig br1 192.168.2.1/24 up
Then, masquerading needs to be done. This is not absolutely required. But if nat is not used, then the gateway of the host's network must know about the VM network and have a route to redirect traffic to the 192.168.2.1 gateway. In my case, I chose to use NAT. The following commands did the trick:
# Enable forwarding. So that linux will see that
# traffic on br1 that wants to go out on a network known by another
# interface, eth0 in my case, will be forwarded on that interface.
echo 1 > /proc/sys/net/ipv4/ip_forward
# Create IP masquering rules. Assuming that eth0 is the host's interface that can
# communicate with the internet gateway (or is the gateway). Packets destined
# for external traffic will leave out eth0. So make a postrouting rule
# that will masquerade those outgoing packets. This will allow other networks
# to reach the VM network without knowing anything about it.
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
Of course, the default route, within the VMs, should point to 192.168.2.1. At this point VMs should be able to connect to this website
- port of type "patch" can be used to "connect a patch cable" between 2 vswitches. This would be used to connect two vswitches together as long as they reside on the same host. To connect two vswitches that reside on different hosts, a port of type GRE would be needed.
- To get additional information on what type of interface can be created (patch, internal, gre, etc.): http://openvswitch.org/ovs-vswitchd.conf.db.5.pdf
In the example above, the ovsdb-server was started with --remote=ptcp:6640. This means that it is possible
to control the vswitch from any other host using "ovs-vsctl --db=tcp:192.168.1.2:6640
". This is great when a virtual switch spans across several host because all host bridges can be controlled from one single point.
The best description of OpenFlow and SDN I found was this: OpenFlow is a protocol which lets you offload the control plane of all the switches to a central controller and lets a central software define the behavior of the network (also called Software Defined Networks). To me, that description was the one that opened the doors to SDN in a time where all of those terminologies was confusing to me.
Before going any further, let's first define what a flow is. An open flow switch makes forwarding/routing decisions based on its flow tables. Each flow represent a "rule" that must be matched and the action that must be taken if the rule does match. Flows are added by a separate piece of software to the switch. The switch does not "build" its own flows. These must be explicitely added to the switch by the switch administrator when first configuring the switch or dynamically by an SDN controller. If using an SDN controller, when a frame/packet/segment does not match any flows, it will be sent to the SDN controller in order to find a flow that matches.
The flows contain a "match" part. It defines fields of frame/packet/segment must be matched so that the flow applies. If the flow does apply, then the "action" part is used. Matches can be specify that a flow must match a source MAC address, a destination TCP port, or any other L2/3/4 fields. Combinations of fields can be matched and wildcards can be used.
The flows also contain an "action" part. Actions are very flexible. I won't detail all the possible actions here. An action could be to drop the packet, forward the frame, redirect to another flow table (internal to the switch) etc.
Using openflow with openvswitch
The man page for ovs-ofctl contains a section that details the flow syntax. Here is an example on how to add a flow that would drop any TCP segment with destination port 80
ovs-ofctl add-flow br0 "table=0, tcp,tp_dst=80, actions=drop"
This one would rewrite any segment going towards TCP port 80 to go to tcp port 25. This is basically port forwarding. Note that two flows need to be written. Once the server on port 25 sends a reply, it needs to be modified to use port 80 since the client is expecting to be communicating on port 80.
ovs-ofctl add-flow br0 "table=0, tcp,tp_dst=80,nw_dst=192.168.2.2, actions=mod_tp_dst:25,normal" ovs-ofctl add-flow br0 "table=0, tcp,tp_src=25,nw_src=192.168.2.2, actions=mod_tp_src:80,normal"
This will add flows to the bridge on the local host. This could be done when configuring the switch initially. Obviously, more complex (and more useful) flows could be written, but this is just to get a hang of it.
It could also be useful to have the switch's flow tables be populated dynamically by some other "brain". Such a brain would be an "opendlow controller". The controller can reside on a separate host.
I downloaded pox, an openflow controller, on one of my host. I am running it on a different host than the one that is hosting the vswitch. vswitch is running on 192.168.1.2 and pox is running on 192.168.1.3
git clone https://github.com/noxrepo/pox cd pox ./pox.py forwarding.l2_learning
And then, on the vswitch host, I tell the bridge who the controller is
ovs-vsctl set-controller br0 tcp:192.168.1.3:6633
Using "forwarding.l2_learning" with pox creates a very basic flow that makes the switch act like a normal layer2 switch. You could write other modules for pox, but that is beyond the scope of this example. At least, now you know how to have a remote controller generate the flows dynamically. You could download another software to do it or even write your own.
By the way, using an openflow controller, we could drop the iptables usage from the above section and implement NAT using openflow only.
What this means
I wanted to learn what this was all about and I'm glad I did. I don't fully understand how this can be very usefull though. I do see the power that this brings to networking infrastructures. But for me, running a couple of computers in my basement with a relatively simple setup, it is not that usefull. I can't list a lot of uses-cases where this would change the world. I'm sure that one day, I'm going to stumble across a problem and say "Hey! That would be super-easy to do with openflow" and I'm gonna be glad I learned the basics of it. And now, 2017, I do fully understand the benefits of it because I work for a major SDN solution manufacturer :)