Docker - Tinc setup

docker-tinc.md

Running Docker on two hosts, bridged with Tinc

This Gist is similar to https://gist.github.com/noteed/8656989 which uses Open vSwitch instead of Tinc.

Setup

Download the install.sh script and run it:

> https://gist.githubusercontent.com/noteed/11031504/raw/install.sh
> sh install.sh

Then configure Tinc, and edit and run shared-docker-tinc.sh.

Tinc configuration

On each host, the Tinc configuration is similar:

/etc/tinc/
├── horde
│   ├── hosts
│   │   ├── host_1
│   │   └── host_2
│   ├── rsa_key.priv
│   ├── tinc.conf
│   ├── tinc-down
│   └── tinc-up
└── nets.boot

On a given host N:

• The rsa_key.priv must be generated, and the corresponding public key is put in /etc/tinc/horde/hosts/host_n.

• The Address = entry in the host_n file is not necessary but does no harm.

• The Name = entry in tinc.conf must be set to host_n.

• The ConnectTo = entry in tinc.conf must be set to the "other" host.

To generate Tinc public/private key pairs:

> tincd -n horde -K

Route

If the BRIDGE_ADDRESS variable is set on the two hosts as 172.16.41.1 and 172.16.41.2, the subnet in tinc.conf can be the same on both hosts:

Subnet = 172.16.41.0/24

and the last line in shared-docker-tinc.sh to setup the route is not needed.

In such a configuration there is a problem: Docker will start allocating IP addresses to containers almost identically on both hosts (e.g. you will end up with both hosts having a container with IP 172.16.41.3).

To avoid that problem, the BRIDGE_ADDRESS should be in different subnets. For instance on host_1:

Subnet = 172.16.41.0/24        # In tinc.conf.
BRIDGE_ADDRESS=172.16.41.1/24  # In shared-docker-tinc.sh.
OTHER_BRIDGE_ADDRESS=172.16.42.0

And on host_2:

Subnet = 172.16.42.0/24        # In tinc.conf.
BRIDGE_ADDRESS=172.16.42.1/24  # In shared-docker-tinc.sh.
OTHER_BRIDGE_ADDRESS=172.16.41.0

host_1

Address = __HOST_1_IP__
-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEA5AOEcNvVvE4cvTZCiBylavGkKLCq6x3dvYoGWuXW+EfRyOGCZWNG
IpPapoJeReTjLONpF0xOSUgmDMmMMXPXnFr3EtviJkDRQXLnSLYLAUNm0/JYMpyZ
TuEJm0xKz8haae1m4kLH9X3Jo00/WkEikGcx5zyCTpwvdDL+7dB1/oiQ2tpX0Pxx
Y08MeEAFXTPGP43R2XD1ULEFP6GSWm5hVcqhJAkNGymQWhRSjmnPPBp3phfOzWax
MiCrm/1DSH4Ne143WaXmR9qT9b2GwLD5CROJZEEYh2PZo1luLuyo0gWR4wVT+aI0
VFsDwpyNElxzfPh/CyJL+ETtrcwkTSfoMQIDAQAB
-----END RSA PUBLIC KEY-----

host_2

Address = __HOST_2_IP__
-----BEGIN RSA PUBLIC KEY-----
MIIBCgKCAQEA57ldQ3/VricHmXXnfxg1LUIoI+M9LvFUCF8HPDX15coyeMg7bk1a
V5S4cfo9MpJpLDfpLu8vGqDhdHnm+cH3CdawWAy7yzY1qrdvQn3OxRNdHyTbXFc5
8+rEYNXeWpZ5GjDd7AFmNdOSOMhlyNUcM6VJ7nAG5Ya7LJ6OWitZfdxci4UGMbhz
ISLxlBFd+Us8LCyYk85X61nQIT59rcZB+653kLzESPelXUXTWjNd5WvipQrExKe8
yLh29rulVEqiDaHQZo/bnroByPgSCI5EFdk029umJOVGPJrY71X4JuY3oOXSjIVX
aNTUC9CahDVmailyIpSQUIkany9OElYTlwIDAQAB
-----END RSA PUBLIC KEY-----

install.sh

# For lxc-docker.
echo deb http://get.docker.io/ubuntu docker main \
  > /etc/apt/sources.list.d/docker.list
apt-key adv \
  --keyserver keyserver.ubuntu.com \
  --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9

apt-get update
apt-get install -q -y vim tinc bridge-utils lxc-docker
echo 'DOCKER_OPTS="--bridge=docker0"' >> /etc/default/docker
service docker restart

# This Gist.
wget https://gist.githubusercontent.com/noteed/11031504/raw/shared-docker-tinc.sh

echo Edit shared-docker-tinc.sh and run it.

nets.boot

## This file contains all names of the networks to be started on system startup.
horde

rsa_key.priv

Replace this file by the host's private key.

shared-docker-tinc.sh

# From http://goldmann.pl/blog/2014/01/21/connecting-docker-containers-on-multiple-hosts/,
# modified to use Tinc instead of Open vSwitch.

# Edit this variable: the 'other' host.
REMOTE_IP=188.226.138.185

# Edit this variable: the bridge address on 'this' host.
BRIDGE_ADDRESS=172.16.42.1/24

# Edit this variable: the bridge address on the 'other' host.
OTHER_BRIDGE_ADDRESS=172.16.41.0

# Name of the bridge (should match /etc/default/docker and /etc/tinc/).
BRIDGE_NAME=docker0

# bridges

# Deactivate the docker0 bridge
ip link set $BRIDGE_NAME down
# Remove the docker0 bridge
brctl delbr $BRIDGE_NAME
# Add the docker0 bridge
brctl addbr $BRIDGE_NAME
# Set up the IP for the docker0 bridge
ip a add $BRIDGE_ADDRESS dev $BRIDGE_NAME
# Activate the bridge
ip link set $BRIDGE_NAME up

# Start Tinc, this creates `horde`, similar to `br0` in the Open vSwitch script.
service tinc restart

# Restart Docker daemon to use the new BRIDGE_NAME
service docker restart

# Make the 'other' subnet routable.
route add -net $OTHER_BRIDGE_ADDRESS netmask 255.255.255.0 dev $BRIDGE_NAME

tinc-down

#!/bin/sh
ifconfig $INTERFACE down

tinc-up

#!/bin/sh
ifconfig $INTERFACE 0.0.0.0
brctl addif docker0 $INTERFACE
ifconfig $INTERFACE up

tinc.conf

Name = __THIS_HOST_NAME__
Mode = switch
Subnet = __THIS_SUBNET__  # e.g. 172.16.42.0/24
TCPOnly = yes
ConnectTo = __THE_OTHER_HOST_NAME__
DeviceType = tap