Docker - Open vSwitch setup

docker-ovs.md

Running Docker on two hosts, bridged with Open vSwitch

This Gist shows how to use Open vSwitch to bridge Docker containers on two hosts. It is based on this blog post http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/.

A similar Gist using Tinc instead of Open vSwitch is available: https://gist.github.com/noteed/11031504.

Setup

To prepare a host (e.g. a new Digital Ocean Ubuntu 14.04 droplet), simply run the install.sh script:

# wget https://gist.githubusercontent.com/noteed/8656989/raw/install.sh
# sh install.sh

Once done, the two first variables of the shared-docker-network.sh script must be changed, and the script can be run:

# vim shared-docker-network.sh
# sh shared-docker-network.sh

Repeat the steps on a second host.

Route

See the other Gist using Tinc to solve a problem where Docker allocates the same IP addresses on both hosts.

Useful debugging commands

root@node-1:~# ip a s
root@node-1:~# ip r s
root@node-1:~# brctl show
root@node-1:~# ovs-vsctl show
root@node-1:~# ping <other host>
root@node-1:~# ping <other host bridge>

install.sh

# For lxc-docker.
echo deb http://get.docker.io/ubuntu docker main \
  > /etc/apt/sources.list.d/docker.list
apt-key adv \
  --keyserver keyserver.ubuntu.com \
  --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

apt-get update
apt-get install -q -y vim openvswitch-switch bridge-utils lxc-docker
echo 'DOCKER_OPTS="--bridge=docker0"' >> /etc/default/docker
service docker restart

# This Gist.
wget https://gist.githubusercontent.com/noteed/8656989/raw/shared-docker-network.sh

echo Edit shared-docker-network.sh and run it.

shared-docker-network.sh

# From http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/

# Edit this variable: the 'other' host.
REMOTE_IP=188.226.138.185

# Edit this variable: the bridge address on 'this' host.
BRIDGE_ADDRESS=172.16.42.1/24

# Name of the bridge (should match /etc/default/docker).
BRIDGE_NAME=docker0

# bridges

# Deactivate the docker0 bridge
ip link set $BRIDGE_NAME down
# Remove the docker0 bridge
brctl delbr $BRIDGE_NAME
# Delete the Open vSwitch bridge
ovs-vsctl del-br br0
# Add the docker0 bridge
brctl addbr $BRIDGE_NAME
# Set up the IP for the docker0 bridge
ip a add $BRIDGE_ADDRESS dev $BRIDGE_NAME
# Activate the bridge
ip link set $BRIDGE_NAME up
# Add the br0 Open vSwitch bridge
ovs-vsctl add-br br0
# Create the tunnel to the other host and attach it to the
# br0 bridge
ovs-vsctl add-port br0 gre0 -- set interface gre0 type=gre options:remote_ip=$REMOTE_IP
# Add the br0 bridge to docker0 bridge
brctl addif $BRIDGE_NAME br0

# iptables rules

# Enable NAT
iptables -t nat -A POSTROUTING -s 172.16.42.0/24 ! -d 172.16.42.0/24 -j MASQUERADE
# Accept incoming packets for existing connections
iptables -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Accept all non-intercontainer outgoing packets
iptables -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
# By default allow all outgoing traffic
iptables -A FORWARD -i docker0 -o docker0 -j ACCEPT

# Restart Docker daemon to use the new BRIDGE_NAME
service docker restart

Recently got a mail from Caleb Crane who faced the same issue and apparently has solved the issue. Am quoting it here, so that the rest of us can use it if faced with the issue. The solution and all credits go to Caleb Crane.

"
I got it working in my environment. It turned out to be necessary to set the MTU to 1420 in the containers. GRE doesn’t support fragmentation so when ssh tried to send a frame at 1500 bytes the client wasn’t receiving the entire thing.
"

I can confirm that --mtu=1420 is required.

see this great tutorial for setting up a "MULTI-HOST DOCKER NETWORK" : https://wiredcraft.com/blog/multi-host-docker-network/

It is also being specific about MTU.