grimoire

personal wiki
git clone git://git.pyratebeard.net/grimoire.git
Log | Files | Refs

lxc.md (8543B)


      1 # lxc
      2 
      3 ## set up
      4 ### arch
      5 [archwiki_setup][]
      6 [redhat_guide][]
      7 [linux_containers_guide][]
      8 
      9 * install `lxc`
     10 ```
     11 pacman -S lxc dnsmasq
     12 ```
     13 * or on debian
     14 ```
     15 apt-get install lxc dnsmasq-base uidmap acl libpam-cgfs
     16 echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.conf
     17 reboot
     18 ```
     19 * add the following line to '/etc/pam.d/system-login'
     20 * (debian '/etc/pam.d/login')
     21 ```
     22 session    optional    pam_cgfs.so    -c freezer,memory,name=systemd,unified
     23 ```
     24 * create '/etc/default/lxc-net' config
     25 ```
     26 # Leave USE_LXC_BRIDGE as "true" if you want to use lxcbr0 for your
     27 # containers.  Set to "false" if you'll use virbr0 or another existing
     28 # bridge, or mavlan to your host's NIC.
     29 USE_LXC_BRIDGE="true"
     30 
     31 # If you change the LXC_BRIDGE to something other than lxcbr0, then
     32 # you will also need to update your /etc/lxc/default.conf as well as the
     33 # configuration (/var/lib/lxc/<container>/config) for any containers
     34 # already created using the default config to reflect the new bridge
     35 # name.
     36 # If you have the dnsmasq daemon installed, you'll also have to update
     37 # /etc/dnsmasq.d/lxc and restart the system wide dnsmasq daemon.
     38 LXC_BRIDGE="lxcbr0"
     39 LXC_ADDR="10.0.3.1"
     40 LXC_NETMASK="255.255.255.0"
     41 LXC_NETWORK="10.0.3.0/24"
     42 LXC_DHCP_RANGE="10.0.3.2,10.0.3.254"
     43 LXC_DHCP_MAX="253"
     44 # Uncomment the next line if you'd like to use a conf-file for the lxcbr0
     45 # dnsmasq.  For instance, you can use 'dhcp-host=mail1,10.0.3.100' to have
     46 # container 'mail1' always get ip address 10.0.3.100.
     47 #LXC_DHCP_CONFILE=/etc/lxc/dnsmasq.conf
     48 
     49 # Uncomment the next line if you want lxcbr0's dnsmasq to resolve the .lxc
     50 # domain.  You can then add "server=/lxc/10.0.3.1' (or your actual $LXC_ADDR)
     51 # to your system dnsmasq configuration file (normally /etc/dnsmasq.conf,
     52 # or /etc/NetworkManager/dnsmasq.d/lxc.conf on systems that use NetworkManager).
     53 # Once these changes are made, restart the lxc-net and network-manager services.
     54 # 'container1.lxc' will then resolve on your host.
     55 #LXC_DOMAIN="lxc"
     56 ```
     57 * add the following lines to '/etc/lxc/default.conf'
     58 ```
     59 lxc.net.0.type = veth
     60 lxc.net.0.link = lxcbr0
     61 lxc.net.0.flags = up
     62 lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
     63 lxc.idmap = u 0 100000 65536
     64 lxc.idmap = g 0 100000 65536
     65 ```
     66 * start `lxc-net`
     67 ```
     68 systemctl restart lxc-net
     69 ```
     70 * check that `lxcbr0` bridge has been created
     71 ```
     72 ip a s lxcbr0
     73 ```
     74 * create '/etc/subuid'
     75 ```
     76 pyratebeard:100000:65536
     77 ```
     78 * create '/etc/subgid'
     79 ```
     80 pyratebeard:100000:65536
     81 ```
     82 * create '/etc/lxc/lxc-usernet' for allowing user to create network devices
     83 ```
     84 pyratebeard veth lxcbr0 10
     85 ```
     86 	- `veth` - virtual ethernet
     87 	- `lxcbr0` - network bridge
     88 	- `10` - number of devices allowed
     89 * create local dirs
     90 ```
     91 mkdir ~/.{config,cache}/lxc
     92 mkdir ~/.local/share
     93 ```
     94 * create '~/.config/lxc/default.conf'
     95 ```
     96 lxc.net.0.type = veth
     97 lxc.net.0.link = lxcbr0
     98 lxc.net.0.flags = up
     99 lxc.net.0.hwaddr = 00:16:3e:xx:xx:xx
    100 lxc.idmap = u 0 100000 65536
    101 lxc.idmap = g 0 100000 65536
    102 ```
    103 * make '~/.local/share' executable and set acls
    104 ```
    105 chmod +x ~/.local/share
    106 setfacl -m u:100000:x /home/pyratebeard
    107 setfacl -m u:100000:x /home/pyratebeard/.local
    108 ```
    109 
    110 ## create container
    111 ```
    112 lxc-create -t download -n <name>
    113 # or
    114 lxc-create -n <name> -t download -- --dist alpine --release 3.13 --arch amd64
    115 lxc-start -d -n <name>
    116 lxc-attach -n <name>
    117 ```
    118 or
    119 ```
    120 vi ~/.local/share/lxc/powerzone/rootfs/etc/shadow
    121 	# remove `!` from root user
    122 lxc-start -n powerzone
    123 lxc-console -n powerzone
    124 ```
    125 
    126 * python module for script api [5][]
    127 
    128 ## alpine linux config
    129 ```
    130 /sbin/apk update
    131 /sbin/apk upgrade
    132 passwd
    133 busybox adduser pyratebeard
    134 busybox adduser pyratebeard wheel
    135 apk add doas vim openssh zsh
    136 echo "permit nopass pyratebeard" | tee -a /etc/doas.d/doas.conf
    137 ln -s /bin/zsh /usr/bin/zsh
    138 /sbin/rc-update add sshd
    139 /sbin/rc-service sshd start
    140 /sbin/rc-status
    141 ```
    142 logout (`ctrl-a q` to exit console)
    143 
    144 ## debian config
    145 ```
    146 passwd
    147 apt-get install openssh-server python3
    148 vi /etc/ssh/sshd_config
    149 	PermitRootLogin yes
    150 systemctl reload sshd
    151 ```
    152 
    153 ### alpine services
    154 add files to /etc/init.d/
    155 ```
    156 #!/sbin/openrc-run
    157 name="test"
    158 command="echo hello"
    159 ```
    160 
    161 ## known errors
    162 * systemd containers fail to start
    163 ```
    164 Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
    165 [!!!!!!] Failed to mount API filesystems, freezing.
    166 Freezing execution.
    167 ```
    168 	* '/sys/fs/cgroup/systemd' dir doesn't exist
    169 	* to fix, create dir, mount cgroup, set permissions [lxc-users group post][]
    170 ```
    171 sudo mkdir /sys/fs/cgroup/systemd
    172 sudo mount -t cgroup -o none,name=systemd systemd /sys/fs/cgroup/systemd
    173 sudo chown pyratebeard:users /sys/fs/cgroup/systemd
    174 ```
    175 * keyserver not found on `lxc-create`
    176 	* to fix add `DOWNLOAD_KEYSERVER="hkp://keyserver.ubuntu.com:80"` before `lxc-create` cmd
    177 	* https://github.com/lxc/lxc/issues/3874
    178 	* https://github.com/lxc/lxc/commit/f2a5d95d00a55bed27ef9920d67617cc75fecad8
    179 ```
    180 Setting up the GPG keyring
    181 ERROR: Unable to fetch GPG key from keyserver
    182 ```
    183 * wait_ondaemonized_startL 833 no such file or directory
    184 	* `lxc-start` in foreground gives segmentation fault
    185 ```
    186 lxc-start -n test /bin/sh
    187 ```
    188 * unable to start on debian 11 (error "895 received container state aborting" - in forground mode "1365 numerical result out of range")
    189 	* use `unpriv` start
    190 ```
    191 lxc-unpriv-start -n <name>
    192 ```
    193 
    194 ## moving containers
    195 [so answer]
    196 ```
    197 lxc-stop -n $NAME
    198 cd ~/.local/share/lxc/$NAME
    199 sudo tar --numeric-owner -czvf ../$NAME.tgz ./*
    200 chown pyratebeard: ../$NAME.tgz
    201 rsync -avh $NAME.tgz user@hostname:.local/share/lxc/
    202 ssh user@hostname
    203 mkdir ~/.local/share/lxc/$NAME
    204 cd ~/.local/share/lxc/$NAME
    205 sudo tar --numeric-owner -xzvf ../$NAME.tgz .
    206 ```
    207 * tried this between wht-rht-obj and fka
    208 	* container runs (after adding user gid to /etc/subgid)
    209 	* no ip address though. veth is created but ip4 not given
    210 * check dir/file permissions
    211 	* .local/share/lxc/$NAME = 755 100000:100000
    212 	* .local/share/lxc/$NAME/rootfs/* = 100000:100000
    213 	* .local/share/lxc/$NAME/config = pyratebeard:users
    214 
    215 ## example
    216 ### setting up multiple websites behind haproxy
    217 * install openzfs
    218 * start lx daemon
    219 ```
    220 sudo apt install zfsutils-linux
    221 sudo lxd init
    222 ```
    223 * answer questions
    224 * launch containers
    225 ```
    226 lxc launch ubuntu:18.04 subdomain1
    227 lxc launch ubuntu:18.04 subdomain2
    228 lxc launch ubuntu:18.04 haproxy
    229 lxc list
    230 ```
    231 
    232 [archwiki_setup]: https://wiki.archlinux.org/title/Linux_Containers#Setup
    233 [redhat_guide]: https://www.redhat.com/sysadmin/exploring-containers-lxc
    234 [linux_containers_guide]: https://linuxcontainers.org/lxc/getting-started/
    235 [lxc-users group post]: https://groups.google.com/a/lists.linuxcontainers.org/g/lxc-users/c/r_8Eww6i9tA
    236 [so answer]: https://stackoverflow.com/questions/23427129/how-do-i-backup-move-lxc-containers#34194341
    237 [5]: https://github.com/lxc/python3-lxc
    238 
    239 gollum haproxy log pastebin radicale site stagit znc ftp
    240 
    241 ## debian test
    242 * debian 10 (aws instance)
    243 	* 'admin' user
    244 * `apt-get install lxc dnsmasq-base uidmap`
    245 * follow setup (see own wiki)
    246 * building debian containers works well
    247 * ansible playbook runs using proxyjump in ssh config
    248 * attempting to run haproxy in container
    249 * iptables rules for prerouting
    250 	* `sudo iptables -t nat -I PREROUTING -i eth0 -p TCP -d <public_ip>/24 --dport 80 -j DNAT --to-destination <haproxy_ip>:80`
    251 	* `sudo iptables -t nat -I PREROUTING -i eth0 -p TCP -d <public_ip>/24 --dport 443 -j DNAT --to-destination <haproxy_ip>:443`
    252 	* `sudo iptables -L -n -t nat`
    253 	* `sudo apt-get install iptables-persistent`
    254 * haproxy container
    255 	* `apt-get install haproxy`
    256 	* add the following to the 'global' section
    257 		```
    258 		...
    259 		maxconn 2048
    260 		...
    261 		tune.ssl.default-dh-param 2048
    262 		```
    263 	* add the following to the 'defaults' section
    264 		```
    265 		...
    266 		option forwardfor
    267 		option http-server-close
    268 		...
    269 		```
    270 	* create frontend
    271 		```
    272 		frontend http_frontend
    273 				bind *:80
    274 				acl infratuxture hdr(host) -i penguin.renre.com
    275 				#acl anotherlxc hdr(host) -i anotherdomain.renre.com
    276 				use_backend penguin if infratuxture
    277 				#use_backend anotherdomain if anotherlxc
    278 		```
    279 	* create backend
    280 		```
    281 		backend penguin
    282 				balance leastconn
    283 				http-request set-header X-Client-IP %[src]
    284 				server penguin 10.0.3.162:80 check
    285 
    286 		#backend anotherdomain
    287 		#		balance leastconn
    288 		#		http-request set-header X-Client-IP %[src]
    289 		#		server anotherdomain an.oth.er.ip:80 check
    290 		```
    291 * infratuxture container
    292 	* `apt-get install git lighttpd`
    293 	* pull git repo in html dir
    294 		```
    295 		cd /var/www/html
    296 		git clone https://git.renre.com/infrastructure/linux-patching.github.io.git .
    297 		```
    298 
    299 ## bindmount
    300 * mount a dir on lxc, add follwoing to container conf
    301 ```
    302 mp0: /path/on/host,mp=/mount/path/on/container
    303 ```