20220127-multi_lxc_with_haproxy.md (7116B)
1 Near the beginning of last year I hit a few issues with some of my Docker containers and part of my CI/CD pipeline. Around the same time I seemed to be reading more about LXC, and a few people on IRC mentioned that it was worth learning. I decided to take a step back from Docker and give LXC a go. 2 3 ## what the chroot 4 LXC or Linux Containers, is a virtualisation method allowing the kernel to be used between multiple environments or containers. While traditionally with Docker you would run single applications inside a container then network them together (web server, database, etc.) LXC gives you a "full" Linux system but unlike a virtual machine it shares the same kernel as the host. 5 6 There are pros and cons to LXC but I don't want to get into that in this post. If you would like to know more about LXC check out the [official website](https://linuxcontainers.org). I should also point out that I have stuck with LXC and not LXD, which is a next generation container manager. 7 8 Setting up LXC is straightforward by following the [official guide](https://linuxcontainers.org/lxc/getting-started/). 9 10 Creating a container is as easy as 11 ``` 12 lxc-create -t download -n <name> 13 ``` 14 15 selecting an image from the list shown. 16 17 Or if you know the image you want to use you can specify it 18 ``` 19 lxc-create -n <name> -t download -- --dist <distro> --release <release_number> --arch <architecture> 20 ``` 21 22 After I created my container I started it and set it up as I would any other system. This then became my "base image". Any new container I wanted could be cloned from this so it is already set up. I renew the base image periodically with updates etc. 23 24 Cloning a container can be done by incanting 25 ``` 26 lxc-copy -n ${BASE} -N ${NEW} 27 ``` 28 29 This command is _suppose_ to change the hostname of the cloned container but I found it didn't. To remedy that incant 30 ``` 31 sudo sed -i "s/${BASE}/${NEW}/" ${HOME}/.local/share/lxc/${NEW}/rootfs/etc/hostname 32 ``` 33 34 ## virtualise all the things 35 I was using Docker to run a number of things on a single VPS, using an Nginx container as a proxy. 36 37 For no particular reason, with LXC I opted for HAProxy. My HAProxy is running in a container. On the host server I set the following firewall rules to send traffic to the HAProxy container 38 ``` 39 iptables -t nat -I PREROUTING \ 40 -i ${INTERFACE} \ 41 -p TCP \ 42 -d ${PUBLIC_IP_ADDRESS}/${CIDR} \ 43 --dport 80 \ 44 -j DNAT \ 45 --to-destination ${HAPROXY_CONTAINER_IP}:80 46 47 iptables -t nat -I PREROUTING \ 48 -i ${INTERFACE} \ 49 -p TCP \ 50 -d ${PUBLIC_IP_ADDRESS}/${CIDR} \ 51 --dport 443 \ 52 -j DNAT \ 53 --to-destination ${HAPROXY_CONTAINER_IP}:443 54 ``` 55 56 Then I could login to HAProxy container to configure it. The config file may be either /etc/haproxy.cfg or /etc/haproxy/haproxy.cfg, on my container it is the latter. 57 58 Of course I want to use SSL and it is advised to set the Diffie-Hellman parameter to 2048 bits instead of the default 1024. I included the following to the `global` section of haproxy.cfg 59 ``` 60 tune.ssl.default-dh-param 2048 61 ``` 62 63 I am using LetsEncrypt for my SSL certificates, so I installed `certbot`. This will be used later to generate our SSL certificates. One of the best solutions I found for LetsEncrypt with HAProxy is from [janeczku](https://github.com/janeczku/haproxy-acme-validation-plugin) on Github. I put a copy of the `acme-http01-webroot.lua` script into /etc/haproxy/ and added the following to the `global` section of haproxy.cfg 64 65 ``` 66 lua-load /etc/haproxy/acme-http01-webroot.lua 67 ``` 68 69 To tell HAProxy to use SSL I had to configure a couple of `frontends` after the `default` section 70 ``` 71 frontend http_frontend 72 bind *:80 73 74 acl url_acme_http01 path_beg /.well-known/acme-challenge/ 75 http-request use-service lua.acme-http01 if METH_GET url_acme_http01 76 77 redirect scheme https 78 79 frontend https_frontend 80 bind *:443 81 ``` 82 83 This config will redirect HTTP traffic on port 80 to HTTPS on 443. 84 85 Now I can declare a `backend` and `acl` to route traffic. For the sake of example my LXC container is called "pyratelog" and the domain I am pointing to is "log.pyratebeard.net". 86 87 The `acl` is declared in the `https_frontend` section 88 ``` 89 frontend https_frontend 90 bind *:443 91 92 acl pyratelog hdr(host) -i log.pyratebeard.net 93 use_backend pyratelog if pyratelog 94 ``` 95 96 Then beneath the `frontend` the `backend` section is configured 97 ``` 98 backend pyratelog 99 balance leastconn 100 http-request set-header X-Client-IP %[src] 101 server pyratelog pyratelog:80 check 102 ``` 103 104 LXC has built in container name resolution, so you can use the name of the container instead of its IP address. 105 106 A reload of HAProxy picks up the changes. 107 108 I used `certbot to request a new SSL cert 109 ``` 110 certbot certonly --text \ 111 --webroot --webroot-path /var/lib/haproxy \ 112 -d log.pyratebeard.net \ 113 --renew-by-default \ 114 --agree-tos \ 115 --email me@email.com 116 ``` 117 118 This created two PEM files, a private key and a chain file. I combined these into one file to be read by HAProxy 119 ``` 120 cat /etc/letsencrypt/live/log.pyratebeard.net/privkey.pem \ 121 /etc/letsencrypt/live/log.pyratebeard.net/fullchain.pem \ 122 | tee /etc/letsencrypt/live/pem/pyratelog.pem 123 ``` 124 125 Now I had to alter the `https_frontend` section to point to the SSL cert directory 126 ``` 127 frontend https_frontend 128 bind *:443 ssl crt /etc/letsencrypt/live/pem/ 129 ``` 130 131 and reloaded HAProxy. 132 133 When I added another LXC container behind HAProxy I simply add a new `backend` and include an `acl` in the `https_frontend`, so it would looks something like this 134 ``` 135 frontend https_frontend 136 bind *:443 137 138 acl pyratelog hdr(host) -i log.pyratebeard.net 139 use_backend pyratelog if pyratelog 140 141 acl pyrateweb hdr(host) -i pyratebeard.net 142 use_backend pyrateweb if pyrateweb 143 144 backend pyratelog 145 balance leastconn 146 http-request set-header X-Client-IP %[src] 147 server pyratelog pyratelog:80 check 148 149 backend pyrateweb 150 balance leastconn 151 http-request set-header X-Client-IP %[src] 152 server pyrateweb pyrateweb:80 check 153 ``` 154 155 Then I ran the `certbot` command again, and combine the PEM files 156 ``` 157 certbot certonly --text \ 158 --webroot --webroot-path /var/lib/haproxy \ 159 -d pyratebeard.net \ 160 --renew-by-default \ 161 --agree-tos \ 162 --email me@email.com 163 164 cat /etc/letsencrypt/live/pyratebeard.net/privkey.pem \ 165 /etc/letsencrypt/live/pyratebeard.net/fullchain.pem \ 166 | tee /etc/letsencrypt/live/pem/pyrateweb.pem 167 168 ``` 169 170 A reload of HAProxy picks up the changes. 171 172 From now on renewing an SSL cert is done by incanting 173 ``` 174 sudo certbot certonly --text --webroot --webroot-path /var/lib/haproxy -d log.pyratebeard.net 175 ``` 176 177 then combine the PEM files again, overwriting the previous file, and reloading HAProxy. 178 179 I was happy with how easy it was to get LXC running with HAProxy, and now comfortably run a number of containers on a single host. 180 181 Docker hasn't completely been removed from my systems, depending on the use case I do lean towards LXC a bit more these days. I have been running my LXC setup for over a year and have had no issues. The "CI/CD" has had to change though, and I will cover how I publish these blog posts onto my LXC container in a later post.