commit f232c3ae4886cbdd434b7db26d85331a87bcdc2a
parent e29308e51963210ad377201b854701826360e8a2
Author: pyratebeard <root@pyratebeard.net>
Date: Tue, 2 Jan 2024 22:43:26 +0000
respect_my_authoritah
Diffstat:
1 file changed, 67 insertions(+), 48 deletions(-)
diff --git a/entry/respect_my_authoritah.md b/entry/respect_my_authoritah.md
@@ -1,13 +1,13 @@
I enjoy trying to improve security in the environments I manage as a sysadmin. One of the hardest parts of improving security is not making it a hindrance to users. (There is a fine line between good security and hindrance, but that's a rant for another time).
-Recently I had been asked to improve the SSH security in an environment. SSH keys weren't really being used, so that was my first suggestion, but I needed to make key management as simple as possible without forgoing any security benefits; so I decided to take a proper look at [SSH Certificate Authorities](TK){target="_blank" rel="noreferrer"} (CA).
+Recently I had been asked to improve the SSH security in an environment. SSH keys weren't really being used, so that was my first suggestion, but I needed to make key management as simple as possible without forgoing any security benefits; so I decided to take a proper look at SSH Certificate Authorities (CA).
I've known about SSH CA for a while, though never played around with it. Before taking it into a production environment I decided to become familiar on my own turf.
There are two types of SSH CA, Host and User.
### gracious host
-Lets look at the Host side first. When you install OpenSSH on a system a number of host keys are generated
+Lets look at the Host side first. When you install OpenSSH (or other SSH servers) on a system a number of host keys are generated
```
/etc/ssh/ssh_host_dsa_key
/etc/ssh/ssh_host_dsa_key.pub
@@ -53,12 +53,12 @@ Host key verification failed.
Phew, now we can contact our sysadmin and make sure we're not being intercepted.
-Or, more presumably, we remove the "offending" line and reconnect
+Or, as is more common, we remove the "offending" line and reconnect, selecting to accept the new key
```
ssh-keygen -f ~/.ssh/known_hosts -R 10.1.2.3
```
-An SSH CA can help improve security by removing this need to confirm the key fingerprints, because as we know nobody does it anyway.
+An SSH CA can help improve security by removing this need to confirm the key fingerprints, because as we know few people do it anyway.
By signing all your server's host keys with a CA key your users only need to trust one entity, the CA.
@@ -71,7 +71,7 @@ Now I have _hostca-key_ and _hostca-key.pub_. I prefer using ED25519 instead of
The public key is the entity we want to trust, so we add that to our _known_hosts_ file, prefixed with `@cert-authority` and the domain your servers are in
```
-cat >> ~/.ssh/known_hosts << EOF
+cat >> EOF << ~/.ssh/known_hosts
> @cert-authority *.pyratebeard.net [hostca public key]
> EOF
```
@@ -81,7 +81,7 @@ In my example I am using the domain `pyratebeard.net` and a wildcard for the sub
Next we want to take a copy of the host public key
```
# method 1
-cat >> server-ssh_host_ed25519_key.pub << EOF
+cat >> EOF << server-ssh_host_ed25519_key.pub
> [public key]
> EOF
@@ -89,7 +89,7 @@ cat >> server-ssh_host_ed25519_key.pub << EOF
rsync server.pyratebeard.net:/etc/ssh/ssh_host_ed25519_key.pub server-ssh_host_ed25519_key.pub
# method 3
-ssh-keyscan server.pyratebeard.net | grep ed25519 | tee server-ssh_host_ed25519_key.pub
+ssh-keyscan server.pyratebeard.net | awk '/ssh-ed25519/ {print $2" "$3}' | tee server-ssh_host_ed25519_key.pub
```
Now we can sign the host public key using our CA private key
@@ -110,7 +110,7 @@ ssh-keygen -L -f server-ssh_host_ed25519_key-cert.pub
Once the certificate has been generated it needs to be put on the server it is for, usually in the /etc/ssh directory
```
# method 1 (on server)
-cat >> /etc/ssh/ssh_host_ed25519_key-cert.pub << EOF
+cat >> EOF << /etc/ssh/ssh_host_ed25519_key-cert.pub
> [server key cert]
> EOF
@@ -127,75 +127,94 @@ If every server in the environment is signed with the same CA then only one line
If you're an admin of a shared Linux system, or are able to manage users' workstations, then drop the `@cert-authority` line into _/etc/ssh/ssh_known_hosts_ for it to work system-wide without bothering the users.
-TK if CA is compromised - does changing host cert cause a warning on the user's end?
-
-## user
-* create user ca key
+### total luser
+Traditionally, to set up SSH key authentication as a user the public key is added to the _~/.ssh/authorized_keys_ file
```
-ssh-keygen -t ed25519 -C userca@pyratebeard.net -f userca-key
-cat userca-key.pub
+ssh-copy-id -f -i ~/.ssh/pyratebeard-ssh_ed25519_key.pub pyratebeard@server
```
-* copy user key
-```
-# method 1
-cat >> pyratebeard-ssh_ed25519_key.pub << EOF
-> [public key]
-> EOF
+Using `ssh-copy-id` usually requires password authentication, or if you have console access, manually adding the public key to the _authorized_keys_ file.
-# method 2
-rsync pyratebeard-ssh_ed25519_key.pub server:pyratebeard-ssh_ed25519_key.pub
+Depending on how you manage your SSH keys you may have a number of authorised keys available on the remote system. If your key was ever compromised, lost, or needed changing for some other reason, you would need to add a new public key to the remote system. Maybe you have multiple remote systems, and maybe you haven't got an automated way to push out new keys. Or maybe your current key isn't working and you can't use password authentication as you have correctly locked down your SSH config.
+
+By using a User CA the remote system(s) only have to trust the CA, then any user key that is signed will be granted access. This is especially useful in a production environment if you have multiple users across a magnitude of systems. When a new user joins the team they get their SSH key signed by the CA and they can instantly gain access to all the remote systems they require.
+
+A User CA, the same Host, is an SSH keypair
+```
+ssh-keygen -t ed25519 -C userca@pyratebeard.net -f userca-key
```
-* sign user public key with ca private key
+Then we can sign a user public SSH key (note we don't use the `-h` option)
```
ssh-keygen -s userca-key -I pyratebeard@pyratebeard.net -n pyratebeard -V +52w pyratebeard-ssh_ed25519_key.pub
```
- * -s sign
- * -I unique identifier
- * -n principal
- * -V validity
+ * -s sign - tells `ssh-keygen` which key to sign with
+ * -I unique identifier - an identifier used for logging
+ * -n principal - username the certificate is valid for, comma-delimit multiple usernames
+ * -V validity - length of time the certificate is valid for, omit for never expiring
-* confirm new user key cert
+The above command will produce a certificate (`[key-name]-cert.pub`). To confirm the new key cert details incant
```
ssh-keygen -L -f pyratebeard-ssh_ed25519_key-cert.pub
```
-* copy new key cert back to users machine
+Let SSH know which certificate to use by adding the following to _~/.ssh/config_
```
-# method 1 (on users machine)
-cat >> pyratebeard-ssh_ed25519_key-cert.pub << EOF
-> [user key cert]
-> EOF
-
-# method 2
-rsync server:pyratebeard-ssh_ed25519_key-cert.pub pyratebeard-ssh_host_ed25519_key-cert.pub
+Host *.pyratebeard.net
+ CertificateFile /path/to/pyratebeard-ssh_host_ed25519_key-cert.pub
```
-* copy user ca public key to all servers
+Just like we included the `@cert-authority` line in _known_hosts_, the remote system(s) need to know the User CA to trust. Copy the User CA public key to the remote system(s)
```
# method 1 (on server)
-cat >> /etc/ssh/userca.pub << EOF
+cat >> EOF << /etc/ssh/userca-key.pub
> [userca public key]
> EOF
# method 2
-rsync userca.pub server:/etc/ssh/userca.pub
+rsync userca-key.pub server:/etc/ssh/userca-key.pub
```
-* on servers add following to _/etc/ssh/sshd_config_
+In the remote system(s) _/etc/ssh/sshd_config_ add the following and reload `sshd`
```
-TrustedUserCAKeys /etc/ssh/userca.pub
+TrustedUserCAKeys /etc/ssh/userca-key.pub
```
-* reload ssh daemon
-* on users machine add following to ~/.ssh/config
+Setting up a User CA means that the _authorized_keys_ file is no longer required for signed keys. If the user key changes, as long as it is signed by the trusted CA it will be granted access.
+
+### a problem shared is still a problem
+All the benefits of SSH CA look great. Less messing around with _known_hosts_ and _authorized_keys_. Quicker access for new user SSH keys and quicker access to new remote systems. But when it comes to security it can't stay like that forever.
+
+If a Host CA key is compromised a malicious system could be disguised as a trusted system, users wouldn't be warned against a fingerprint change.
+
+In order to revoke a Host CA change `@cert-authority` to `@revoke` in the relevant _known_hosts_ file.
+
+When this happens SSH will warn that the system authenticity can't be established, as we saw before when logging in for the very first time.
+
+If this is in a production environment the sysadmin would have to re-sign all the remote system keys and then inform users to update their _known_hosts_ file.
+
+A bit of overhead, but remedied easily enough.
+
+What if the User CA key is compromised? A bad actor could gain access to all the remote systems by simply signing their SSH key (make sure you use passwords when generating CA keys!).
+
+To revoke a User CA add the path to the CA in /etc/ssh/revoked_keys on the remote system(s)
```
-CertificateFile /path/to/pyratebeard-ssh_host_ed25519_key-cert.pub
+cat << EOF >> /etc/ssh/revoked_keys
+> /etc/ssh/userca-key.pub
+> EOF
+```
+
+Let SSH know about the revocation file in _/etc/ssh/sshd_config_ and reload `sshd`
+```
+RevokedKeys /etc/ssh/revoked_keys
```
-* now no longer need authorized_keys file on server
+This will revoke access to all user keys which have been signed by the compromised CA. This could be a lot of users, and could potentially cause great impact. Ideally you would have multiple CA certs configured, maybe group teams to a CA so the impact is reduced to a small number of users.
+
+This is where SSH CA can fall down. I know from my own experience that not many environments use CA. There have been attempts to make things more dynamic and transparent, especially on the user side of things. There is [BLESS](https://github.com/Netflix/bless){target="_blank" rel="noreferrer"}, developed by Netflix, which has received much praise. It is based on using AWS Lambda functions, not always available to environments.
+
+There is also [sshrimp](https://github.com/stoggi/sshrimp){target="_blank" rel="noreferrer"} by Jeremy Stott, who gave an interesting talk ([Zero Trust SSH](https://lca2020.linux.org.au/schedule/presentation/54/){target="_blank" rel="noreferrer"}) at LinuxConfAU in 2020. `sshrimp` also works using Lambda functions.
+
+In my client's environment I can't use Lambda at the moment. Potentially I could get something like BLESS configured but that would be an uphill battle of approvals with various stakeholders, something I'm not ready for right now. I tried looking at getting the CAs set up first, at least as a proof of concept to showcase the client. All the users are running Windows, with [MobaXTerm](https://mobaxterm.mobatek.net/){target="_blank" rel="noreferrer"} being the connection tool of choice. For a few days I fought with MobaXTerm, trying to get it to pick up the trusted Host CA and the signed User cert. I have since drawn a line under it until I can come up with a new plan.
-* must have dns resolv on greyskull (and nublar for lxc)
-* must have short name in ssh/config on nublar for lxc
-* must have cert-auth known host on nublar for lxc
+The upside to all of this is that I have learned a lot about SSH Certificate Authorities, and I (finally) have SSH CA configured for my own systems.
