lxc.md (10143B)
1 # lxc 2 3 ## set up 4 ### arch 5 [archwiki_setup][] 6 [redhat_guide][] 7 [linux_containers_guide][] 8 9 * install `lxc` 10 ``` 11 pacman -S lxc dnsmasq 12 ``` 13 * or on debian 14 ``` 15 apt-get install lxc dnsmasq-base uidmap acl libpam-cgfs 16 echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf 17 reboot 18 ``` 19 * add the following line to '/etc/pam.d/system-login' 20 * (debian '/etc/pam.d/login') 21 ``` 22 session optional pam_cgfs.so -c freezer,memory,name=systemd,unified 23 ``` 24 * create '/etc/default/lxc-net' config 25 ``` 26 # Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your 27 # containers. Set to "false" if you'll use virbr0 or another existing 28 # bridge, or mavlan to your host's NIC. 29 USE_LXC_BRIDGE="true" 30 31 # If you change the LXC_BRIDGE to something other than lxcbr0, then 32 # you will also need to update your /etc/lxc/default.conf as well as the 33 # configuration (/var/lib/lxc/<container>/config) for any containers 34 # already created using the default config to reflect the new bridge 35 # name. 36 # If you have the dnsmasq daemon installed, you'll also have to update 37 # /etc/dnsmasq.d/lxc and restart the system wide dnsmasq daemon. 38 LXC_BRIDGE="lxcbr0" 39 LXC_ADDR="10.0.3.1" 40 LXC_NETMASK="255.255.255.0" 41 LXC_NETWORK="10.0.3.0/24" 42 LXC_DHCP_RANGE="10.0.3.2,10.0.3.254" 43 LXC_DHCP_MAX="253" 44 # Uncomment the next line if you'd like to use a conf-file for the lxcbr0 45 # dnsmasq. For instance, you can use 'dhcp-host=mail1,10.0.3.100' to have 46 # container 'mail1' always get ip address 10.0.3.100. 47 #LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf 48 49 # Uncomment the next line if you want lxcbr0's dnsmasq to resolve the .lxc 50 # domain. You can then add "server=/lxc/10.0.3.1' (or your actual $LXC_ADDR) 51 # to your system dnsmasq configuration file (normally /etc/dnsmasq.conf, 52 # or /etc/NetworkManager/dnsmasq.d/lxc.conf on systems that use NetworkManager). 53 # Once these changes are made, restart the lxc-net and network-manager services. 54 # 'container1.lxc' will then resolve on your host. 55 #LXC_DOMAIN="lxc" 56 ``` 57 * add the following lines to '/etc/lxc/default.conf' 58 ``` 59 lxc.net.0.type = veth 60 lxc.net.0.link = lxcbr0 61 lxc.net.0.flags = up 62 lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx 63 lxc.idmap = u 0 100000 65536 64 lxc.idmap = g 0 100000 65536 65 ``` 66 * start `lxc-net` 67 ``` 68 systemctl restart lxc-net 69 ``` 70 * check that `lxcbr0` bridge has been created 71 ``` 72 ip a s lxcbr0 73 ``` 74 * create '/etc/subuid' 75 ``` 76 pyratebeard:100000:65536 77 ``` 78 * create '/etc/subgid' 79 ``` 80 pyratebeard:100000:65536 81 ``` 82 * create '/etc/lxc/lxc-usernet' for allowing user to create network devices 83 ``` 84 pyratebeard veth lxcbr0 10 85 ``` 86 - `veth` - virtual ethernet 87 - `lxcbr0` - network bridge 88 - `10` - number of devices allowed 89 * create local dirs 90 ``` 91 mkdir ~/.{config,cache}/lxc 92 mkdir ~/.local/share 93 ``` 94 * create '~/.config/lxc/default.conf' 95 ``` 96 lxc.net.0.type = veth 97 lxc.net.0.link = lxcbr0 98 lxc.net.0.flags = up 99 lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx 100 lxc.idmap = u 0 100000 65536 101 lxc.idmap = g 0 100000 65536 102 ``` 103 * make '~/.local/share' executable and set acls 104 ``` 105 chmod +x ~/.local/share 106 setfacl -m u:100000:x /home/pyratebeard 107 setfacl -m u:100000:x /home/pyratebeard/.local 108 ``` 109 110 ## create container 111 ``` 112 lxc-create -t download -n <name> 113 # or 114 lxc-create -n <name> -t download -- --dist alpine --release 3.13 --arch amd64 115 lxc-start -d -n <name> 116 lxc-attach -n <name> 117 ``` 118 or 119 ``` 120 vi ~/.local/share/lxc/powerzone/rootfs/etc/shadow 121 # remove `!` from root user 122 lxc-start -n powerzone 123 lxc-console -n powerzone 124 ``` 125 126 * python module for script api [5][] 127 128 ## alpine linux config 129 ``` 130 /sbin/apk update 131 /sbin/apk upgrade 132 passwd 133 busybox adduser pyratebeard 134 busybox adduser pyratebeard wheel 135 apk add doas vim openssh zsh 136 echo "permit nopass pyratebeard" | tee -a /etc/doas.d/doas.conf 137 ln -s /bin/zsh /usr/bin/zsh 138 /sbin/rc-update add sshd 139 /sbin/rc-service sshd start 140 /sbin/rc-status 141 ``` 142 logout (`ctrl-a q` to exit console) 143 144 ## debian config 145 ``` 146 passwd 147 apt-get install openssh-server python3 148 vi /etc/ssh/sshd_config 149 PermitRootLogin yes 150 systemctl reload sshd 151 ``` 152 153 ### alpine services 154 add files to /etc/init.d/ 155 ``` 156 #!/sbin/openrc-run 157 name="test" 158 command="echo hello" 159 ``` 160 161 ## known errors 162 * systemd containers fail to start 163 ``` 164 Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted 165 [!!!!!!] Failed to mount API filesystems, freezing. 166 Freezing execution. 167 ``` 168 * '/sys/fs/cgroup/systemd' dir doesn't exist 169 * to fix, create dir, mount cgroup, set permissions [lxc-users group post][] 170 ``` 171 sudo mkdir /sys/fs/cgroup/systemd 172 sudo mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd 173 sudo chown pyratebeard:users /sys/fs/cgroup/systemd 174 ``` 175 * keyserver not found on `lxc-create` 176 * to fix add `DOWNLOAD_KEYSERVER="hkp://keyserver.ubuntu.com:80"` before `lxc-create` cmd 177 * https://github.com/lxc/lxc/issues/3874 178 * https://github.com/lxc/lxc/commit/f2a5d95d00a55bed27ef9920d67617cc75fecad8 179 ``` 180 Setting up the GPG keyring 181 ERROR: Unable to fetch GPG key from keyserver 182 ``` 183 * wait_ondaemonized_startL 833 no such file or directory 184 * `lxc-start` in foreground gives segmentation fault 185 ``` 186 lxc-start -n test /bin/sh 187 ``` 188 * unable to start on debian 11 (error "895 received container state aborting" - in forground mode "1365 numerical result out of range") 189 * use `unpriv` start 190 ``` 191 lxc-unpriv-start -n <name> 192 ``` 193 194 ## moving containers 195 [so answer] 196 ``` 197 lxc-stop -n $NAME 198 cd ~/.local/share/lxc/$NAME 199 sudo tar --numeric-owner -czvf ../$NAME.tgz ./* 200 chown pyratebeard: ../$NAME.tgz 201 rsync -avh $NAME.tgz user@hostname:.local/share/lxc/ 202 ssh user@hostname 203 mkdir ~/.local/share/lxc/$NAME 204 cd ~/.local/share/lxc/$NAME 205 sudo tar --numeric-owner -xzvf ../$NAME.tgz . 206 ``` 207 * tried this between wht-rht-obj and fka 208 * container runs (after adding user gid to /etc/subgid) 209 * no ip address though. veth is created but ip4 not given 210 * check dir/file permissions 211 * .local/share/lxc/$NAME = 755 100000:100000 212 * .local/share/lxc/$NAME/rootfs/* = 100000:100000 213 * .local/share/lxc/$NAME/config = pyratebeard:users 214 215 ## example 216 ### setting up multiple websites behind haproxy 217 * install openzfs 218 * start lx daemon 219 ``` 220 sudo apt install zfsutils-linux 221 sudo lxd init 222 ``` 223 * answer questions 224 * launch containers 225 ``` 226 lxc launch ubuntu:18.04 subdomain1 227 lxc launch ubuntu:18.04 subdomain2 228 lxc launch ubuntu:18.04 haproxy 229 lxc list 230 ``` 231 232 [archwiki_setup]: https://wiki.archlinux.org/title/Linux_Containers#Setup 233 [redhat_guide]: https://www.redhat.com/sysadmin/exploring-containers-lxc 234 [linux_containers_guide]: https://linuxcontainers.org/lxc/getting-started/ 235 [lxc-users group post]: https://groups.google.com/a/lists.linuxcontainers.org/g/lxc-users/c/r_8Eww6i9tA 236 [so answer]: https://stackoverflow.com/questions/23427129/how-do-i-backup-move-lxc-containers#34194341 237 [5]: https://github.com/lxc/python3-lxc 238 239 gollum haproxy log pastebin radicale site stagit znc ftp 240 241 ## debian test 242 * debian 10 (aws instance) 243 * 'admin' user 244 * `apt-get install lxc dnsmasq-base uidmap` 245 * follow setup (see own wiki) 246 * building debian containers works well 247 * ansible playbook runs using proxyjump in ssh config 248 * attempting to run haproxy in container 249 * iptables rules for prerouting 250 * `sudo iptables -t nat -I PREROUTING -i eth0 -p TCP -d <public_ip>/24 --dport 80 -j DNAT --to-destination <haproxy_ip>:80` 251 * `sudo iptables -t nat -I PREROUTING -i eth0 -p TCP -d <public_ip>/24 --dport 443 -j DNAT --to-destination <haproxy_ip>:443` 252 * `sudo iptables -L -n -t nat` 253 * `sudo apt-get install iptables-persistent` 254 * haproxy container 255 * `apt-get install haproxy` 256 * add the following to the 'global' section 257 ``` 258 ... 259 maxconn 2048 260 ... 261 tune.ssl.default-dh-param 2048 262 ``` 263 * add the following to the 'defaults' section 264 ``` 265 ... 266 option forwardfor 267 option http-server-close 268 ... 269 ``` 270 * create frontend 271 ``` 272 frontend http_frontend 273 bind *:80 274 acl infratuxture hdr(host) -i penguin.renre.com 275 #acl anotherlxc hdr(host) -i anotherdomain.renre.com 276 use_backend penguin if infratuxture 277 #use_backend anotherdomain if anotherlxc 278 ``` 279 * create backend 280 ``` 281 backend penguin 282 balance leastconn 283 http-request set-header X-Client-IP %[src] 284 server penguin 10.0.3.162:80 check 285 286 #backend anotherdomain 287 # balance leastconn 288 # http-request set-header X-Client-IP %[src] 289 # server anotherdomain an.oth.er.ip:80 check 290 ``` 291 * infratuxture container 292 * `apt-get install git lighttpd` 293 * pull git repo in html dir 294 ``` 295 cd /var/www/html 296 git clone https://git.renre.com/infrastructure/linux-patching.github.io.git . 297 ``` 298 299 ## bindmount 300 * mount a dir on lxc, add follwoing to container conf 301 ``` 302 mp0: /path/on/host,mp=/mount/path/on/container 303 ``` 304 305 ## uid/gid mapping 306 * in lxc conf 307 ``` 308 lxc.idmap: u 0 100000 1005 309 lxc.idmap: g 0 100000 1005 310 lxc.idmap: u 1005 1005 1 311 lxc.idmap: g 1005 1005 1 312 lxc.idmap: u 1006 101006 64530 313 lxc.idmap: g 1006 101006 64530 314 ``` 315 * explanation taken from [itsembedded][] 316 317 > The format of the lxc.idmap configuration lines are <u/g> <guest_start_id> <host_start_id> <num_of_ids_to_map>, where <u/g> selects whether the mapping is for user id’s or group id’s. 318 > 319 > Below is an explanation of what each mapping combination does: 320 > 321 > * (u/g) 0 10000 1000 - map 1000 ID’s starting from 0, to ID’s starting at 100000. This means that the ROOT UID/GID 0:0 on the guest will be mapped to 100000:100000 on the host, 1:1 will be mapped to 100001:1000001, and so on. 322 > 323 > * (u/g) 1000 1000 1 - map the UID/GID pair 1000:1000 to 1000:1000 on the host. The number 1 is there to specify we’re only mapping a single ID, and not a range. 324 > 325 > * (u/g) 1001 101000 64535 - map 64535 ID’s starting at 1001, to ID’s starting at 101000. This means that UID/GID pair 1001:1001 on the guest will be mapped to 101000:101000, 1002:1002 to 101001:101001, all the way to finally 65535:65535 to 165534:165534. 326 327 328 ## mounting zfs dataset in lxc container 329 * requires [uid/gid mapping](#uid-gid-mapping) 330 * this example is for using the www-data user with nextcloud 331 * on host 332 ``` 333 zfs set acltype=posixacl pool/dataset 334 setfacl -m u:100033:rwx /path/to/dataset 335 ``` 336 * add mount point as [above](#bindmount) 337 * on container check acl 338 ``` 339 getfacl /path/to/mount 340 ``` 341 342 [itsembedded]: https://www.itsembedded.com/sysadmin/proxmox_bind_unprivileged_lxc/