Skip to content

Instantly share code, notes, and snippets.

@Drallas
Last active January 17, 2025 23:48

Docker Swarm in LXC Containers

Part of collection: Hyper-converged Homelab with Proxmox

After struggling for some days, and since I really needed this to work (ignoring the it can't be done vibe everywhere), I managed to get Docker to work reliable in privileged Debian 12 LXC Containers on Proxmox 8

(Unfortunately, I couldn't get anything to work in unprivileged LXC Containers)

There are NO modifications required on the Proxmox host or the /etc/pve/lxc/xxx.conf file; everything is done on the Docker Swarm host. So the only obvious candidate who could break this setup, are future Docker Engine updates!

Host Setup

My host are Debian 12 LXC containers, installed via tteck's Proxmox VE Helper Scripts

Install the LXC via the Proxmox VE Helper Script

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/ct/debian.sh)"

Backing filesystems

Docker info shows i'm using overlay2, this is the recommended storage driver for Debian. This storage driver requires XFS or EXT4 as backing file system.

docker info | grep -A 7 "Storage Driver:"

 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd

Set userns-remap

As Neuer_User pointed out, running the Docker Containers unprivileged on a privileged

LXC seems the best compromise to run the containers in a relative secure way.

To do so, add a daemon.json on the Docker Servers that are part of the Swarm.

mkdir /etc/docker
nano /etc/docker/daemon.json
{
  "userns-remap": "root"
}

And reboot reboot the Docker Host.

(This moves everything below /var/lib/docker/ to the folder /var/lib/docker/0.0/ existing workload disappear, hence it's a step pre Docker installation!)

Install Docker

The get-docker.sh script is the most convenient way to quickly install the latest Docker-CE release!

curl -fsSL https://get.docker.com -o get-docker.sh
chmod +x get-docker.sh
./get-docker.sh

Join Create Docker / Swarm

Without this step, the next step(s) fail!

# Manager Node
docker swarm init

# Add Node
docker swarm join --token <some-very-long-token>

# Display Join token again
docker swarm join-token worker
docker swarm join-token manager

Add ipv4 for Ingress_sbox

For Docker in LXC to work, the only thing needed is to execute:

nsenter --net=/run/docker/netns/ingress_sbox sysctl -w net.ipv4.ip_forward=1

on the Docker Swarm Servers

Make it permanent

This doesn't survive reboots, so I created an oneshot systemd service for it, to make sure that after each reboot the setting is applied.

Create a Bash Script

First, we need a Bash script to be executed by the service.

nano /usr/local/bin/ipforward.sh

#!/bin/bash
nsenter --net=/run/docker/netns/ingress_sbox sysctl -w net.ipv4.ip_forward=1

Make it executable

chmod +x /usr/local/bin/ipforward.sh

Create a Systemd Service

This service is of the type oneshot, during startup it waits for the docker.service to be started, and then 10 seconds for run-docker-netns-ingress_sbox.mount to be loaded. Only after that net.ipv4.ip_forward=1 can be applied.

nano /etc/systemd/system/ingress-sbox-ipforward.service
[Unit]
Description = Set net.ipv4.ip_forward for ingress_sbox namespace
After = docker.service
Wants = docker.service

[Service]
Type = oneshot
RemainAfterExit = yes
ExecStartPre = /bin/sleep 10
ExecStart = /usr/local/bin/ipforward.sh

[Install]
WantedBy = multi-user.target

Start the service and check if it's healthy

systemctl daemon-reload
systemctl enable ingress-sbox-ipforward.service
systemctl start ingress-sbox-ipforward.service
systemctl status ingress-sbox-ipforward.service

Final Checks

Without ipv4.ip_forward set to 1, the Ingress Networking to the Docker Swarm is not active. So it's important to verify if the value is applied successfully.

Manual check if ipv4.ip_forward is set to 1

systemctl status ingress-sbox-ipforward.service | grep ipforward.sh

# Or in a script via:

current_value=$(nsenter --net=/run/docker/netns/ingress_sbox sysctl -n net.ipv4.ip_forward)
echo $current_value

(Now, Docker in LXC seems to behave as Docker in a VM.)

Issues

  1. Service in docker-compose resolved wrong ip

To fix this, it’s needed to add a hostname entry for each swarm service, to make it more logical I also add a prefix service to the service names.

services:
  service_nginx: # Prefix service_
    image: nginx
    hostname: nginx

Links

Screenshot

Screenshot 2023-09-21 at 14 38 45

@akhdanfadh
Copy link

akhdanfadh commented Jan 8, 2025

Thank you very much for writing this guide! I found this when I try to use docker secrets in portainer but turns out it only supported in docker swarm environment.

I am trying to understand the user mapping. I don't think simply making the daemon.json file is enough. See this docker docs on namespace,

Some distributions do not automatically add the new group to the /etc/subuid and /etc/subgid files. If that's the case, you are may have to manually edit these files and assign non-overlapping ranges. This step is covered in Prerequisites.

And this is the case for me on a fresh privileged debian LXC. I don't see any mapping in both /etc/subuid and /etc/subgid after following all your guide, it's just empty. I think this is crucial if you concern about unprivileged/privileged in the first place.

So my question is:

  1. Are you aware of this? If so, is this intended (the no mapping in those two files)?
  2. Will existing containers break if I add those two files and the mapping inside?
  3. This question is specific. Basically if I want to use GPU transcoding in an unprivileged LXC, I can trick the mapping in the LXC configuration file (discussed in many guides, this for example). Then the question: even if I deploy a privileged LXC and use this namespace mapping for docker inside it, the GPU won't be able to be used straight away, right?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment