How to get consul-agent and systemd.resolvd to co-exist peicefully and still be able to resolve *.consul hostsnames from within docker

readme.md

I see many people struggeling to make consul-agent work with systemd.resolvd and eventually give up and go with dnsmasq or a similar approach.

Here's a reasonably simple way to make everything play nicely together.

If you found this useful, say thanks. And as much as i'd love your support via patreon, go and donate to the EFF.

Here's an exerpt from the install_consul_agent.sh that my base packer builder runs for all my systemd hosts:

# Binary is in place, secured. Deploy the systemd components to make it useful
##
echo "moving systemd components into place..."
# deploy the service file
mv systemd/consul-agent.service /etc/systemd/system/consul-agent.service
mv systemd/dummy0.netdev /etc/systemd/network/dummy0.netdev
mv systemd/dummy0.network /etc/systemd/network/dummy0.network

# Then get the interface created
echo "reconfiguring network for dummy0..."
systemctl restart systemd-networkd

# Ok, now we're ready!
echo "Attempting to bring consul up for POST..."
systemctl enable consul-agent
systemctl start consul-agent

# Confirm that consul actually came up...
if [ `systemctl is-failed consul-agent.service` == 'failed' ];
then
    echo "Consul failed to start"
    # Bail, packer should fail this build...
    exit 1
fi

It really is that simple. Systemd.resolved will happily do split-zone DNS. I think this functionality was intended for VPN users, but we can take advantage of this for our purposes. Rather than tell systemd.resolved that $host.internal.corp.com can be reached via ppp01, we tell resolvd that $host.consul can be resolved via dummy0 and bind consul-agent to dummy0.

Oh the fun to be had with virtual interfaces.... :).

dummy0.netdev

# Creates a "dummy" network interface
# we'll configure this interface with a link-local address
# See: https://www.freedesktop.org/software/systemd/man/systemd.netdev.html
##

[NetDev]
Name=dummy0
Kind=dummy

dummy0.network

# We "find" the dummy0 interface we created and configure it with a link-local address
# See: https://www.freedesktop.org/software/systemd/man/systemd.network.html
##
[Match]
Name=dummy0

[Network]
Address=169.254.1.1/32

# We want these domains
Domains= ~consul.

# To go to the consul DNS server we'll bind to this address
DNS=169.254.1.1

network.hcl

##
# This file is one of a few in `/etc/consul.d/config` and the consul-agent binary is launched with 
#   something that looks like this in the `consul-agent.service` file:
#
#   ExecStart=/usr/local/bin/consul agent -config-dir=/etc/consul.d/ -config-dir=/etc/consul.d/services/ ... etc
##
#   
# IN order to get split-zone DNS working properly with Consul and systemd resolver and docker,
# we have to slightly tweak the consul network configuration.
##
# The address that consul will bind to for INTER-CLUSTER COMMS.
# By default, this is "0.0.0.0", meaning Consul will bind to all addresses on the local machine and will 
#   advertise the first available private IPv4 address to the rest of the cluster.
#
# However, with the dummy0 and (possibly) docker0 interfaces, consul will find multiple private addresses
#   so we must explicitly tell consul to bind to the *actual* IP/Interface that is linked to amazon's network
#   this is almost always going to be the `eth0` equivelent on a ubuntu host.
##
# See: https://www.consul.io/docs/agent/options.html#_bind
bind_addr = "{{ GetPrivateIP }}"


# See https://www.consul.io/docs/agent/options.html#_client
##
# The address that consul will bind CLIENT SERVIDES (http/dns) to is very different from the 
#   address we want to accept clients on! As the agent is running locally, we want to accept 
#   clients *ONLY* on our link-local / dummy0 address!
#
# Luckily, this is pretty easy. Hashi has go-sockaddr which lets us write golang style template
#   statements to filter out various host network things. We'll use this shortly...
#
# See: https://godoc.org/github.com/hashicorp/go-sockaddr/template
##
# Currently, this is broken. 
# See: https://github.com/hashicorp/consul/issues/5371
# See: https://github.com/hashicorp/go-sockaddr/issues/31
#
# The fix is to hard-code the IP address. Normally, a sin punishable by death
#   but in this case, it's just a paddelin' , as it's a link local and very unlikely
#   to every need changing. I still feel like this is going to bite me one day...
# client_addr = "{{ GetInterfaceIP \"dummy0\" }}"
client_addr = "169.254.1.1"

# the ssyd.resolvd conf does not allow you to set ports, unfortunately. Consul must run on 53 for
#   clients expectations about DNS to work.
ports = {
  dns = 53
}

# Docker runtime can not use 127.* addresses for DNS when the container is on it's own network name space
# This is because the docker runtime hosts it's own resolver in the 127.X space
#
# Docker can't forward DNS queries to 127.0.0.53, but it *can* forward to 169.254.1.1.
# However, consul will not know what to do with DNS queries it gets that do not end in *.consul
#
# We fix this by telling consul to forward queries to the system resolver
# See: https://www.consul.io/docs/agent/options.html#recursors
##
# Forward non *consul domains to the systemd resolver so consul will play nice w/ docker
recursors = ["127.0.0.53"]