20240102-respect_my_authoritah.md (12681B)
1 I enjoy trying to improve security in the environments I manage as a sysadmin. One of the hardest parts of improving security is not making it a hindrance to users. (There is a fine line between good security and hindrance, but that's a rant for another time). 2 3 Recently I had been asked to improve the SSH security in an environment. SSH keys weren't really being used, so that was my first suggestion, but I needed to make key management as simple as possible without forgoing any security benefits; so I decided to take a proper look at SSH Certificate Authorities (CA). 4 5 I've known about SSH CA for a while, though never played around with it. Before taking it into a production environment I decided to become familiar on my own turf. 6 7 There are two types of SSH CA, Host and User. 8 9 ### gracious host 10 Lets look at the Host side first. When you install OpenSSH (or other SSH servers) on a system a number of host keys are generated 11 ``` 12 /etc/ssh/ssh_host_dsa_key 13 /etc/ssh/ssh_host_dsa_key.pub 14 /etc/ssh/ssh_host_ecdsa_key 15 /etc/ssh/ssh_host_ecdsa_key.pub 16 /etc/ssh/ssh_host_ed25519_key 17 /etc/ssh/ssh_host_ed25519_key.pub 18 /etc/ssh/ssh_host_rsa_key 19 /etc/ssh/ssh_host_rsa_key.pub 20 ``` 21 22 When you first SSH to a server you are presented with a message that looks something like this 23 ``` 24 The authenticity of host '10.1.2.3 (10.1.2.3)' can't be established. 25 ECDSA key fingerprint is SHA256:CwrcHjdd9349u38rj392fr9j389rj3298rj23. 26 Are you sure you want to continue connecting (yes/no/[fingerprint])? 27 ``` 28 29 Now what should happen, if it isn't your server, is the sysadmin would provide you with the fingerprint of the host key(s) so you can confirm it is in fact the correct server you are connecting to. 30 31 But we all know that doesn't happen. Everyone types `yes` and moves on with their life. 32 33 When you accept the fingerprint SSH adds the host public key to your _known_hosts_ file, usually located under ~/.ssh. 34 35 If the host key ever changed you would want to know. It could be the sysadmin has rotated the keys (unlikely), the server has been rebuilt or another has taken that IP (likely), or someone is attempted a Man in the Middle attack (scary!). 36 37 Thankfully SSH does warn you, have you ever seen this? 38 ``` 39 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 40 @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ 41 @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 42 IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! 43 Someone could be eavesdropping on you right now (man-in-the-middle attack)! 44 It is also possible that a host key has just been changed. 45 The fingerprint for the ECDSA key sent by the remote host is 46 SHA256:CwrcHjdd9347u38fj392fr9f389rj289snjd. 47 Please contact your system administrator. 48 Add correct host key in /home/pyratebeard/.ssh/known_hosts to get rid of this message. 49 Offending ECDSA key in /home/pyratebeard/.ssh/known_hosts:1 50 ECDSA host key for 10.1.2.3 has changed and you have requested strict checking. 51 Host key verification failed. 52 ``` 53 54 Phew, now we can contact our sysadmin and make sure we're not being intercepted. 55 56 Or, as is more common, we remove the "offending" line and reconnect, selecting to accept the new key 57 ``` 58 ssh-keygen -f ~/.ssh/known_hosts -R 10.1.2.3 59 ``` 60 61 An SSH CA can help improve security by removing this need to confirm the key fingerprints, because as we know few people do it anyway. 62 63 By signing all your server's host keys with a CA key your users only need to trust one entity, the CA. 64 65 For SSH, a CA is actually just another SSH key pair. You then sign a hosts public key using the CA private key to produce a certificate. To generate a CA it's as easy as creating a new key pair 66 ``` 67 ssh-keygen -t ed25519 -C hostca@pyratebeard.net -f hostca-key 68 ``` 69 70 Now I have _hostca-key_ and _hostca-key.pub_. I prefer using ED25519 instead of RSA or ECDSA. 71 72 The public key is the entity we want to trust, so we add that to our _known_hosts_ file, prefixed with `@cert-authority` and the domain your servers are in 73 ``` 74 cat >> EOF << ~/.ssh/known_hosts 75 > @cert-authority *.pyratebeard.net [hostca public key] 76 > EOF 77 ``` 78 79 In my example I am using the domain `pyratebeard.net` and a wildcard for the subdomain. This means if I `ssh` to `server.pyratebeard.net` then this CA will be used. You could use a wildcard (`*`) with no domain if you use the same CA for every system you need to access. 80 81 Next we want to take a copy of the host public key 82 ``` 83 # method 1 84 cat >> EOF << server-ssh_host_ed25519_key.pub 85 > [public key] 86 > EOF 87 88 # method 2 89 rsync server.pyratebeard.net:/etc/ssh/ssh_host_ed25519_key.pub server-ssh_host_ed25519_key.pub 90 91 # method 3 92 ssh-keyscan server.pyratebeard.net | awk '/ssh-ed25519/ {print $2" "$3}' | tee server-ssh_host_ed25519_key.pub 93 ``` 94 95 Now we can sign the host public key using our CA private key 96 ``` 97 ssh-keygen -s hostca-key -h -I server@pyratebeard.net -n server.pyratebeard.net,server -V +52w server-ssh_host_ed25519_key.pub 98 ``` 99 * `-s` sign - tells `ssh-keygen` which key to sign with 100 * `-h` host cert - indicates we're creating a host cert (not a user cert) 101 * `-I` unique identifier - an identifier used for logging 102 * `-n` principals - hostname(s) certificate is valid for, good practice to use short hostname and FQDN 103 * `-V` validity - length of time the certificate is valid for, in this example it is 52 weeks 104 105 This command will produce a certificate (`[key name]-cert.pub`). To confirm the new server key cert details incant 106 ``` 107 ssh-keygen -L -f server-ssh_host_ed25519_key-cert.pub 108 ``` 109 110 Once the certificate has been generated it needs to be put on the server it is for, usually in the /etc/ssh directory 111 ``` 112 # method 1 (on server) 113 cat >> EOF << /etc/ssh/ssh_host_ed25519_key-cert.pub 114 > [server key cert] 115 > EOF 116 117 # method 2 118 rsync server-ssh_host_ed25519_key-cert.pub server:/etc/ssh/ssh_host_ed25519_key-cert.pub 119 ``` 120 121 So that SSH knows to use a certificate add the following to _/etc/ssh/sshd_config_ and reload the SSH service 122 ``` 123 HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub 124 ``` 125 126 If every server in the environment is signed with the same CA then only one line is needed in _known_hosts_, the `@cert-authority` entry. New servers can be signed and users won't have to accept any fingerprints because they will already be trusted. Systems can be rebuilt, or keys regenerated, and it won't cause any impact to users. 127 128 If you're an admin of a shared Linux system, or are able to manage users' workstations, then drop the `@cert-authority` line into _/etc/ssh/ssh_known_hosts_ for it to work system-wide without bothering the users. 129 130 ### total luser 131 Traditionally, to set up SSH key authentication as a user the public key is added to the _~/.ssh/authorized_keys_ file 132 ``` 133 ssh-copy-id -f -i ~/.ssh/pyratebeard-ssh_ed25519_key.pub pyratebeard@server 134 ``` 135 136 Using `ssh-copy-id` usually requires password authentication, or if you have console access, manually adding the public key to the _authorized_keys_ file. 137 138 Depending on how you manage your SSH keys you may have a number of authorised keys available on the remote system. If your key was ever compromised, lost, or needed changing for some other reason, you would need to add a new public key to the remote system. Maybe you have multiple remote systems, and maybe you haven't got an automated way to push out new keys. Or maybe your current key isn't working and you can't use password authentication as you have correctly locked down your SSH config. 139 140 By using a User CA the remote system(s) only have to trust the CA, then any user key that is signed will be granted access. This is especially useful in a production environment if you have multiple users across a magnitude of systems. When a new user joins the team they get their SSH key signed by the CA and they can instantly gain access to all the remote systems they require. 141 142 A User CA, the same Host, is an SSH keypair 143 ``` 144 ssh-keygen -t ed25519 -C userca@pyratebeard.net -f userca-key 145 ``` 146 147 Then we can sign a user public SSH key (note we don't use the `-h` option) 148 ``` 149 ssh-keygen -s userca-key -I pyratebeard@pyratebeard.net -n pyratebeard -V +52w pyratebeard-ssh_ed25519_key.pub 150 ``` 151 * -s sign - tells `ssh-keygen` which key to sign with 152 * -I unique identifier - an identifier used for logging 153 * -n principal - username the certificate is valid for, comma-delimit multiple usernames 154 * -V validity - length of time the certificate is valid for, omit for never expiring 155 156 The above command will produce a certificate (`[key-name]-cert.pub`). To confirm the new key cert details incant 157 ``` 158 ssh-keygen -L -f pyratebeard-ssh_ed25519_key-cert.pub 159 ``` 160 161 Let SSH know which certificate to use by adding the following to _~/.ssh/config_ 162 ``` 163 Host *.pyratebeard.net 164 CertificateFile /path/to/pyratebeard-ssh_host_ed25519_key-cert.pub 165 ``` 166 167 Just like we included the `@cert-authority` line in _known_hosts_, the remote system(s) need to know the User CA to trust. Copy the User CA public key to the remote system(s) 168 ``` 169 # method 1 (on server) 170 cat >> EOF << /etc/ssh/userca-key.pub 171 > [userca public key] 172 > EOF 173 174 # method 2 175 rsync userca-key.pub server:/etc/ssh/userca-key.pub 176 ``` 177 178 In the remote system(s) _/etc/ssh/sshd_config_ add the following and reload `sshd` 179 ``` 180 TrustedUserCAKeys /etc/ssh/userca-key.pub 181 ``` 182 183 Setting up a User CA means that the _authorized_keys_ file is no longer required for signed keys. If the user key changes, as long as it is signed by the trusted CA it will be granted access. 184 185 ### a problem shared is still a problem 186 All the benefits of SSH CA look great. Less messing around with _known_hosts_ and _authorized_keys_. Quicker access for new user SSH keys and quicker access to new remote systems. But when it comes to security it can't stay like that forever. 187 188 If a Host CA key is compromised a malicious system could be disguised as a trusted system, users wouldn't be warned against a fingerprint change. 189 190 In order to revoke a Host CA change `@cert-authority` to `@revoke` in the relevant _known_hosts_ file. 191 192 When this happens SSH will warn that the system authenticity can't be established, as we saw before when logging in for the very first time. 193 194 If this is in a production environment the sysadmin would have to re-sign all the remote system keys and then inform users to update their _known_hosts_ file. 195 196 A bit of overhead, but remedied easily enough. 197 198 What if the User CA key is compromised? A bad actor could gain access to all the remote systems by simply signing their SSH key (make sure you use passwords when generating CA keys!). 199 200 To revoke a User CA add the path to the CA in /etc/ssh/revoked_keys on the remote system(s) 201 ``` 202 cat << EOF >> /etc/ssh/revoked_keys 203 > /etc/ssh/userca-key.pub 204 > EOF 205 ``` 206 207 Let SSH know about the revocation file in _/etc/ssh/sshd_config_ and reload `sshd` 208 ``` 209 RevokedKeys /etc/ssh/revoked_keys 210 ``` 211 212 This will revoke access to all user keys which have been signed by the compromised CA. This could be a lot of users, and could potentially cause great impact. Ideally you would have multiple CA certs configured, maybe group teams to a CA so the impact is reduced to a small number of users. 213 214 This is where SSH CA can fall down. I know from my own experience that not many environments use CA. There have been attempts to make things more dynamic and transparent, especially on the user side of things. There is [BLESS](https://github.com/Netflix/bless){target="_blank" rel="noreferrer"}, developed by Netflix, which has received much praise. It is based on using AWS Lambda functions, not always available to environments. 215 216 There is also [sshrimp](https://github.com/stoggi/sshrimp){target="_blank" rel="noreferrer"} by Jeremy Stott, who gave an interesting talk ([Zero Trust SSH](https://lca2020.linux.org.au/schedule/presentation/54/){target="_blank" rel="noreferrer"}) at LinuxConfAU in 2020. `sshrimp` also works using Lambda functions. 217 218 In my client's environment I can't use Lambda at the moment. Potentially I could get something like BLESS configured but that would be an uphill battle of approvals with various stakeholders, something I'm not ready for right now. I tried looking at getting the CAs set up first, at least as a proof of concept to showcase the client. All the users are running Windows, with [MobaXTerm](https://mobaxterm.mobatek.net/){target="_blank" rel="noreferrer"} being the connection tool of choice. For a few days I fought with MobaXTerm, trying to get it to pick up the trusted Host CA and the signed User cert. I have since drawn a line under it until I can come up with a new plan. 219 220 The upside to all of this is that I have learned a lot about SSH Certificate Authorities, and I (finally) have SSH CA configured for my own systems.