commit dd965bec146be020be9c01c494809bf1119816bf
parent 27bbad5d9aa66e29352aea95443a91cd6c6094b3
Author: pyratebeard <root@pyratebeard.net>
Date: Thu, 27 Jan 2022 23:50:43 +0000
multi_lxc_with_haproxy
Diffstat:
1 file changed, 162 insertions(+), 6 deletions(-)
diff --git a/entry/multi_lxc_with_haproxy.md b/entry/multi_lxc_with_haproxy.md
@@ -1,25 +1,181 @@
Near the beginning of last year I hit a few issues with some of my Docker containers and part of my CI/CD pipeline. Around the same time I seemed to be reading more about LXC, and a few people on IRC mentioned that it was worth learning. I decided to take a step back from Docker and give LXC a go.
## what the chroot
-LXC or Linux Containers, is a virtualisation method allowing the kernel to be used between multiple environments or containers. While traditionally with Docker you would run applications inside a container then network them together (web server, database, etc.) LXC gives you a "full" Linux system but unlike a virtual machine it shares the same kernel as the host.
+LXC or Linux Containers, is a virtualisation method allowing the kernel to be used between multiple environments or containers. While traditionally with Docker you would run single applications inside a container then network them together (web server, database, etc.) LXC gives you a "full" Linux system but unlike a virtual machine it shares the same kernel as the host.
There are pros and cons to LXC but I don't want to get into that in this post. If you would like to know more about LXC check out the [official website](https://linuxcontainers.org). I should also point out that I have stuck with LXC and not LXD, which is a next generation container manager.
-Setting up LXC is straightforward.
+Setting up LXC is straightforward by following the [official guide](https://linuxcontainers.org/lxc/getting-started/).
-Then you can create a container, selecting an image from the list shown.
+Creating a container is as easy as
+```
+lxc-create -t download -n <name>
+```
+
+selecting an image from the list shown.
+
+Or if you know the image you want to use you can specify it
+```
+lxc-create -n <name> -t download -- --dist <distro> --release <release_number> --arch <architecture>
+```
-After I created the container I started it and set it up as I would any other system. This then became my "base image". Any new container I wanted could be cloned from this so it is already set up. I renew the base image periodically with updates etc.
+After I created my container I started it and set it up as I would any other system. This then became my "base image". Any new container I wanted could be cloned from this so it is already set up. I renew the base image periodically with updates etc.
-To make a clone of a container incant
+Cloning a container can be done by incanting
```
+lxc-copy -n ${BASE} -N ${NEW}
```
This command is _suppose_ to change the hostname of the cloned container but I found it didn't. To remedy that incant
```
+sudo sed -i "s/${BASE}/${NEW}/" ${HOME}/.local/share/lxc/${NEW}/rootfs/etc/hostname
```
## virtualise all the things
I was using Docker to run a number of things on a single VPS, using an Nginx container as a proxy.
-For no particular reason, with LXC I opted for Haproxy. The VPS I am using runs Debian, but I chose Devuan for the containers so I didn't have to use systemd.
+For no particular reason, with LXC I opted for HAProxy. My HAProxy is running in a container. On the host server I set the following firewall rules to send traffic to the HAProxy container
+```
+iptables -t nat -I PREROUTING \
+ -i ${INTERFACE} \
+ -p TCP \
+ -d ${PUBLIC_IP_ADDRESS}/${CIDR} \
+ --dport 80 \
+ -j DNAT \
+ --to-destination ${HAPROXY_CONTAINER_IP}:80
+
+iptables -t nat -I PREROUTING \
+ -i ${INTERFACE} \
+ -p TCP \
+ -d ${PUBLIC_IP_ADDRESS}/${CIDR} \
+ --dport 443 \
+ -j DNAT \
+ --to-destination ${HAPROXY_CONTAINER_IP}:443
+```
+
+Then I could login to HAProxy container to configure it. The config file may be either /etc/haproxy.cfg or /etc/haproxy/haproxy.cfg, on my container it is the latter.
+
+Of course I want to use SSL and it is advised to set the Diffie-Hellman parameter to 2048 bits instead of the default 1024. I included the following to the `global` section of haproxy.cfg
+```
+tune.ssl.default-dh-param 2048
+```
+
+I am using LetsEncrypt for my SSL certificates, so I installed `certbot`. This will be used later to generate our SSL certificates. One of the best solutions I found for LetsEncrypt with HAProxy is from [janeczku](https://github.com/janeczku/haproxy-acme-validation-plugin) on Github. I put a copy of the `acme-http01-webroot.lua` script into /etc/haproxy/ and added the following to the `global` section of haproxy.cfg
+
+```
+lua-load /etc/haproxy/acme-http01-webroot.lua
+```
+
+To tell HAProxy to use SSL I had to configure a couple of `frontends` after the `default` section
+```
+frontend http_frontend
+ bind *:80
+
+ acl url_acme_http01 path_beg /.well-known/acme-challenge/
+ http-request use-service lua.acme-http01 if METH_GET url_acme_http01
+
+ redirect scheme https
+
+frontend https_frontend
+ bind *:443
+```
+
+This config will redirect HTTP traffic on port 80 to HTTPS on 443.
+
+Now I can declare a `backend` and `acl` to route traffic. For the sake of example my LXC container is called "pyratelog" and the domain I am pointing to is "log.pyratebeard.net".
+
+The `acl` is declared in the `https_frontend` section
+```
+frontend https_frontend
+ bind *:443
+
+ acl pyratelog hdr(host) -i log.pyratebeard.net
+ use_backend pyratelog if pyratelog
+```
+
+Then beneath the `frontend` the `backend` section is configured
+```
+backend pyratelog
+ balance leastconn
+ http-request set-header X-Client-IP %[src]
+ server pyratelog pyratelog:80 check
+```
+
+LXC has built in container name resolution, so you can use the name of the container instead of its IP address.
+
+A reload of HAProxy picks up the changes.
+
+I used `certbot to request a new SSL cert
+```
+certbot certonly --text \
+ --webroot --webroot-path /var/lib/haproxy \
+ -d log.pyratebeard.net \
+ --renew-by-default \
+ --agree-tos \
+ --email me@email.com
+```
+
+This created two PEM files, a private key and a chain file. I combined these into one file to be read by HAProxy
+```
+cat /etc/letsencrypt/live/log.pyratebeard.net/privkey.pem \
+ /etc/letsencrypt/live/log.pyratebeard.net/fullchain.pem \
+ | tee /etc/letsencrypt/live/pem/pyratelog.pem
+```
+
+Now I had to alter the `https_frontend` section to point to the SSL cert directory
+```
+frontend https_frontend
+ bind *:443 ssl crt /etc/letsencrypt/live/pem/
+```
+
+and reloaded HAProxy.
+
+When I added another LXC container behind HAProxy I simply add a new `backend` and include an `acl` in the `https_frontend`, so it would looks something like this
+```
+frontend https_frontend
+ bind *:443
+
+ acl pyratelog hdr(host) -i log.pyratebeard.net
+ use_backend pyratelog if pyratelog
+
+ acl pyrateweb hdr(host) -i pyratebeard.net
+ use_backend pyrateweb if pyrateweb
+
+backend pyratelog
+ balance leastconn
+ http-request set-header X-Client-IP %[src]
+ server pyratelog pyratelog:80 check
+
+backend pyrateweb
+ balance leastconn
+ http-request set-header X-Client-IP %[src]
+ server pyrateweb pyrateweb:80 check
+```
+
+Then I ran the `certbot` command again, and combine the PEM files
+```
+certbot certonly --text \
+ --webroot --webroot-path /var/lib/haproxy \
+ -d pyratebeard.net \
+ --renew-by-default \
+ --agree-tos \
+ --email me@email.com
+
+cat /etc/letsencrypt/live/pyratebeard.net/privkey.pem \
+ /etc/letsencrypt/live/pyratebeard.net/fullchain.pem \
+ | tee /etc/letsencrypt/live/pem/pyrateweb.pem
+
+```
+
+A reload of HAProxy picks up the changes.
+
+From now on renewing an SSL cert is done by incanting
+```
+sudo certbot certonly --text --webroot --webroot-path /var/lib/haproxy -d log.pyratebeard.net
+```
+
+then combine the PEM files again, overwriting the previous file, and reloading HAProxy.
+
+I was happy with how easy it was to get LXC running with HAProxy, and now comfortably run a number of containers on a single host.
+
+Docker hasn't completely been removed from my systems, depending on the use case I do lean towards LXC a bit more these days. I have been running my LXC setup for over a year and have had no issues. The "CI/CD" has had to change though, and I will cover how I publish these blog posts onto my LXC container in a later post.
