Guide: Multi-site / single-server with Linux Containers + HAProxy

haproxy-multisite.md

Multi-site / single-server with Linux Containers + HAProxy

The below was completed on a Vultr 1xCPU / 1GB VC2 running Ubuntu 18.04, based on tutorials from Digital Ocean and SSDNodes, alongside various other helpful resources. Instructions do not work on Ubuntu 18.10.

Using this system I currently pay US$5.00 per month to host 2x wordpress sites (1x ecommerce) and 1x static nginx.

If you like the guide, consider signing up to Vultr with my affiliate code. Thanks!

Improvements to the guide (or better explanations) are most appreciated!

Table of contents
1. Updating the VPS
2. Create a non-root user
3. Preparing Linux Containers (LXD/LXC)
4. Launching 3x containers
5. Forwarding traffic to HAProxy with IPTables
6. Configure HAProxy
7. Prepare multi-sites
8. Install wordpress on site_b, the Apache container
Resources

1. Updating the VPS

# logged in as root
apt update
apt upgrade

2. Create a non-root {user}

You will need a locally created SSH key for this step.

# create a new {user}
adduser {user}
# add to the sudoers group
usermod -aG sudo {user}
# switch to the user
su {user}
# create an .ssh folder
mkdir -p ~/.ssh
# paste the contents of your local id_rsa.pub into authorized_keys
nano ~/.ssh/authorized_keys
# update permissions
chmod -R go= ~/.ssh
chown -R {user}:{user} ~/.ssh

3. Preparing Linux Containers (LXD/LXC)

Note: This step will fail if you are using Ubuntu 18.10! Use 18.04.

sudo apt install zfsutils-linux
sudo lxd init

Question	Answer (default)
1. clustering	(no)
2. storage pool	(yes)
3. pool name	lxc_pool
4. storage BE	(zfs)
5. new ZFS pool	yes
6. block device	no
7. space	(15GB)
8. MAAS ctn	(no)
9. LNB	yes
10. bridge name	(lxdbr0)
11. IPv4	(auto)
12. IPv6	none
13. LXD over net	no
14. auto-upd cache	yes

4. Launching 3x containers

The first container will take a while to download and launch. The rest will be quick.

# proxy container
lxc launch ubuntu:18.04 HAProxy
# site containers
lxc launch ubuntu:18.04 {site_a}
lxc launch ubuntu:18.04 {site_b}
# note the IP's, these are used later
lxc list

5. Forwarding traffic to HAProxy with IPTables

a. List network interfaces

ifconfig

b. Find the correct {interface, e.g. eth0} by looking for the VPS's {public_ip} towards the top of a listing, e.g:

{interface}: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
      inet {public_ip}  netmask 255.255.255.0  broadcast
      [ ... ]

c. Route http / https traffic to the HAProxy container. This step requires {haproxy_ip} from lxc list (see 4. above). A similar step is used to forward SSH traffic to our containers.

# route http(:80) traffic to HAProxy
sudo iptables -t nat -I PREROUTING -i {interface} -p TCP -d {public_ip}/32 --dport 80 -j DNAT --to-destination {haproxy_ip}:80
# route https(:443) traffic to HAProxy
sudo iptables -t nat -I PREROUTING -i {interface} -p TCP -d {public_ip}/32 --dport 443 -j DNAT --to-destination {haproxy_ip}:443

d. Persist IPTables rules on reboot

sudo apt install iptables-persistent

e. Add security with the ufw firewall

sudo ufw allow http
sudo ufw allow https
sudo ufw allow ssh
sudo ufw enable

6. Configure HAProxy

a. Login to the container as @root

lxc exec HAProxy -- bash

b. Update the container (follow step 1 above)

c. Install HAProxy

apt install haproxy

d. update global and defaults configurations (/etc/haproxy/haproxy.cfg) to include {max_users e.g 2048}, and retain user IP address

• global
  • maxconn {max_users}
  • tune.ssl.default-dh-param {max_users}
• defaults
  • option forwardfor
  • option http-server-close

e. update frontend and backend configurations (nano /etc/haproxy/haproxy.cfg) to route to container based on domain, e.g.

frontend http_frontend
    bind *:80
    
    acl web_host1 hdr(host) -i {site_a_domain, e.g: example.com}
    acl web_host2 hdr(host) -i {site_b_domain, e.g: example.com.au}
 
    use_backend {site_a} if web_host1
    use_backend {site_b} if web_host2
    
backend {site_a}
    balance leastconn
    http-request set-header X-Client-IP %[src]
    server {site_a} {site_a}.lxd:80 check

backend {site_b}
    balance leastconn
    http-request set-header X-Client-IP %[src]
    server {site_b} {site_b}.lxd:80 check

f. check valid configuration

/usr/sbin/haproxy -f /etc/haproxy/haproxy.cfg -c

g. restart HAProxy

service haproxy reload

h. exit out of the container

exit

7. Prepare multi-sites

a. Login to the {site_a} container as @root

lxc exec {site_a} -- bash

b. Update the container (follow step 1 above)

c. Update the container SSH port to enable direct remote login (continued in 7.g):

• edit config: nano /etc/ssh/sshd_config
• update port: Port 901{i, e.g 9011, 9012, 9013, etc.}
• restart the sshd service: sudo service sshd restart

d. Install nginx

apt install nginx

e. Create a non-root {user} (follow step 2 above)

• use the same non-root username and SSH key as the main server
• ideally use a new password

f. exit the container

exit

g. Route ssh traffic to the container. This step requires the {container_ssh_port} configured in 7.c above, the {interface} from ifconfig see 5.a, 5.b above, and the {container_ip} from lxc list (see 4. above).

iptables -t nat -A PREROUTING -i {interface} -p tcp --dport {container_ssh_port} -j DNAT --to {container_ip}:{container_ssh_port}

Once complete you should be able to log in to the main server or a container directly (and e.g. its MariaDB instance):

# log in to the main server as usual, i.e. excluding the port:
ssh {user_name}@{public_ip}
# include the container ssh port to log in to a container
ssh -p {container_ssh_port} {user_name}@{public_ip}

h. Repeat above steps for {site_b}, this time with Apache in step 7.e (above)

apt install apache2
# once installed
systemctl stop apache2.service 
systemctl start apache2.service 
systemctl enable apache2.service

i. update your DNS to point at your new public IP address and check {site_a} and {site_b} domains for nginx / apache test pages

j. exit back to the main server and login as root su root so you can persist the changes to the iptables (otherwise they will revert on reboot)

sudo iptables-save > /etc/iptables/rules.v4

k. exit the server and test ssh login from your local machine

ssh -p {container_ssh_port} {username}@{public_ip}

8. Install wordpress on site_b, the Apache container

a. Install MariaDB (mySQL fork)

sudo apt install mariadb-server mariadb-client
# once installed
sudo systemctl stop mariadb.service 
sudo systemctl start mariadb.service 
sudo systemctl enable mariadb.service

b. Secure MariaDB Server by creating a root password and disallowing remote root access

sudo mysql_secure_installation

Question	Answer (default)
1. current pass	(none)
2. set root pass	(Y)
3. new pass	{enter new pass}
4. re-enter pass	{repeat pass}
5. remove anon	(Y)
6. disallow remote	(Y)
7. remove test	(Y)
8. reload privs	(Y)

c. Install PHP and related modules

sudo apt install software-properties-common 
sudo add-apt-repository ppa:ondrej/php
# when ready
sudo apt install php7.3 libapache2-mod-php7.3 php7.3-common  php7.3-sqlite3 php7.3-curl php7.3-intl php7.3-mbstring php7.3-xmlrpc  php7.3-mysql php7.3-gd php7.3-xml php7.3-cli php7.3-zip php7.3-curl

d. Update PHP config nano /etc/php/7.3/apache2/php.ini setting the following:

# below may appear in multiple places - make sure only one is set
short_open_tag = On
# others should appear once only
memory_limit = 256M
date.timezone = Australia/Brisbane

and then restart Apache2

sudo systemctl restart apache2.service

e. Create a DB for Wordpress using MariaDB sudo mysql -u root -p. You will need to decide a {db_name}, new {db_user, e.g. 'wordpress'}, and generate a {strong_password}

CREATE DATABASE {db_name};
CREATE USER '{db_user}'@'localhost' IDENTIFIED BY '{strong_password}';
GRANT ALL ON {db_name}.* TO '{db_user}'@'localhost' IDENTIFIED BY '{strong_password}' WITH GRANT OPTION;
FLUSH PRIVILEGES;
EXIT;

f. Configure Wordpress site in Apache sudo nano /etc/apache2/sites-available/wordpress.conf. You will need your {email_address}, {domain_name} and and {alternative_domains, e.g. www.domain_name}.

<VirtualHost *:80> 
     ServerAdmin {email_address}
     DocumentRoot /var/www/html/      
     ServerName {domain_name}
     ServerAlias {alternative_domains}

     <Directory /var/www/html/>         
            Options +FollowSymlinks         
            AllowOverride All
            Require all granted 
     </Directory> 
     
    ErrorLog ${APACHE_LOG_DIR}/error.log      
    CustomLog ${APACHE_LOG_DIR}/access.log combined
 
</VirtualHost>

and enable it

sudo a2ensite wordpress.conf 
sudo a2enmod rewrite

g. Download Wordpress and set permissions to allow and and group write, then add {user_name} to the group.

cd /tmp && wget https://wordpress.org/latest.tar.gz 
tar -zxvf latest.tar.gz 
sudo mv wordpress/* /var/www/html
sudo chown -R www-data:www-data /var/www/html/ 
sudo chmod -R 775 /var/www/html/
sudo usermod -aG www-data {user_name}

h. Configure Wordpress for MariaDB

sudo mv /var/www/html/wp-config-sample.php /var/www/html/wp-config.php
sudo nano /var/www/html/wordpress/wp-config.php

Then update the config file with the DB credentials created earlier, including {db_name}, {db_user} and the {strong_password} as per example below:

// ** MySQL settings - You can get this info from your web host ** // /** The name of the database for WordPress */ 
define('DB_NAME', '{db_name}'); 

/** MySQL database username */ 
define('DB_USER', '{db_user}'); 

/** MySQL database password */ 
define('DB_PASSWORD', '{strong_password}');
 
/** MySQL hostname */ 
define('DB_HOST', 'localhost'); 

/** Database Charset to use in creating database tables. */ define('DB_CHARSET', 'utf8'); 

/** The Database Collate type. Don't change this if in doubt. */ define('DB_COLLATE', '');

/** Cloudfront SSL (set Cloudflare SSL Option to 'flexible' under 'Crypto') **/
if ( isset( $_SERVER['HTTP_CF_VISITOR'] ) && strpos( $_SERVER['HTTP_CF_VISITOR'], 'https' ) !== false ) {
  $_SERVER['HTTPS'] = 'on';
}

i. Restart Apache

sudo systemctl reload apache2.service

j. Remove the default index file and test your Wordpress site in a browser!

sudo rm /var/www/html/index.html

Done.

Resources:

• LXC & HAProxy Setup
  • Digital Ocean Tutorial
  • SSDnodes Tutorial
• SSH using putty into LXC container on ubuntu server
• Create a sudo user with SSH key
  • How To Create a Sudo User on Ubuntu
  • How to Upload an SSH Public Key to an Existing Droplet
• Installing WP
• Installing PHP 7.3

Hey thanks @achrjulien. The primary intention was to reduce complexity (tho looking at the size of the gist I may have failed there!).

Docker is built on top of lxc so this seemed a more 'bare metal' approach. No extra APIs to learn/maintain. If you've got a problem you're debugging Linux/Apache/nginx. It also 'feels' like a better method to host multiple low-rent sites on a single box, without stepping on any toes. Happy to hear other people's thoughts on this.