[HN Gopher] SSH Bastion Host Best Practices
___________________________________________________________________
SSH Bastion Host Best Practices
Author : old-gregg
Score : 165 points
Date : 2022-01-13 17:52 UTC (5 hours ago)
(HTM) web link (goteleport.com)
(TXT) w3m dump (goteleport.com)
| rawkode wrote:
| Some great tips here, take my upvote!
| cbsmith wrote:
| People still use Bastion hosts?
|
| I'm trying to grok why they're better than SOCKS5 proxies... Is
| it because they provide shell access and a larger attack surface?
| ;-)
| perlgeek wrote:
| Auditing requirements.
|
| Many certifications or legal requirements demand that you log
| all changes to your systems, including administrative changes.
|
| Bastion host are a well-understood (both by operators and
| auditors) way to implement that, so it's still a go-to
| solution.
| tptacek wrote:
| I think this article has come up before? Either way, it's a
| quirky thing for Gravitational to post, since their flagship
| project --- Teleport --- basically eliminates bastion servers
| altogether (you might think of it as an API-controlled self-
| contained bastion server). Teleport is free, and worth checking
| out: it solves a bunch of SSH management problems, not just
| controlling access, but also linking SSH access to SSO, running
| fleet-wide commands selectively, and generating transcripts of
| SSH sessions.
|
| Teleport is kind of big and sprawling. But they've repeatedly
| contracted Doyensec to do assessment work for it, and Doyensec is
| a fantastic firm. I think parked behind Tailscale, so none of
| your SSH infra is exposed to the Internet to begin with, it's a
| pretty great solution, and I'd do that again before I ever hand-
| tooled an SSH bastion host again.
| javajosh wrote:
| An interesting category of HOWTO have been companies teaching
| you how to do it yourself, for real. Before you get started,
| they pitch you at the end offering a paid-for option that has 0
| learning curve. That's a good pitch.
| [deleted]
| nolta wrote:
| It recommends: # Configure idle time logout
| ClientAliveInterval <value in seconds>
|
| but i don't think this is correct. AFAICT, this is a keep-alive
| mechanism, not a timeout. I don't think openssh has an option to
| kill idle sessions.
| ghjnut wrote:
| Could someone do me a solid and explain best security practices
| around bastion hosts and vpn?
|
| e.g. - would you still require users connected to the vpn to go
| through a bastion host? - would you ever run bastion/vpn through
| the same box? - are there preferred access use cases for each?
| chasil wrote:
| I run an "internal" set of bastion hosts that are gateways into
| a system that runs telnet. This internal system is able to run
| SSH, but connections stop around 100 because of OS limits. We
| need to support 400-500 logins, and that has to be telnet.
| Everybody connecting has to go through these bastions,
| including VPN users.
|
| I recently built an nspawn container with tinysshd server, with
| a .profile that execs telnet to the relevant system on login.
|
| We had previously used an old version of Microfocus Reflections
| (terminal emulation) with stunnels deployed on all the clients
| and bastions. That was not containerized, but the server
| stunnels were set to chroot() on startup.
|
| I recently was forced to support the latest version of
| Reflections, and since it doesn't support chacha-poly, I also
| built dropbear SSH server just for them. Reflections is very
| expensive (~$500/seat), and the best that it supports is
| aes256-ctr, using Tatu Ylonen's commercial ssh.com (which
| appears to be abandonware). I really hope we can get rid of
| that.
| tptacek wrote:
| Yes, you would still have people connect to the bastion if
| they're on the VPN; part of the point of a bastion is to have a
| central place to monitor and control SSH access, which a VPN
| doesn't really do for you. Additionally, you will inevitably
| end up with team members who need access to the VPN (to reach
| staging and test versions of your applications, or to access
| customer support consoles) but don't get SSH access; a bastion
| gives you a standard configuration to apply to your fleet to
| ensure that "on the VPN" doesn't ever equate to "can log into a
| server".
|
| You should generally do both things.
|
| Wait, I should word that better. You should generally have both
| sets of controls: network access control with a VPN, and fine-
| grained, auditable SSH-level access control. I don't love the
| "Linux shell server" approach to providing those SSH controls.
| ghjnut wrote:
| Thanks for the response, that clears things up quite a bit.
| Would you create jump-boxes per environment or do you
| generally just have 1 with all the different service/env
| access logic?
| tptacek wrote:
| It depends. It's more important to have some controls in
| place than to make super-complicated controls. Again: shell
| servers you SSH into to SSH out of are kind of an anti-
| pattern. See elsewhere on the thread about Teleport, which,
| combined with Tailscale, is I think a pretty good answer to
| these concerns.
| [deleted]
| oxfordmale wrote:
| SSH Bastion Best Practices --> Don't use one
| samlader wrote:
| Could you share your reasons for not using one in the provided
| scenario?
| oxfordmale wrote:
| You can use a service like Teleport or set up an SD WAN. I am
| using a split tunnel AWS VPN client service to avoid having
| to set up an Bastion server
| LinuxBender wrote:
| Good writup. One thing I would add for bastions if you wanted to
| harden them would be to disable session multiplexing if you are
| using MFA/2FA. MaxSessions 1
|
| The default is 10. The plus side of multiplexing is that
| subsequent connections using the same ssh connection _channels_
| are not validated against the authorization mechanisms such as
| _login_ or _2FA_. This reduces friction and speeds up the login
| process _because login is not actually occurring_. The trade-off
| of multiplexing is that all subsequent logins using that ssh
| connection are not logged nor are they validated with MFA. This
| means a person phishing your team members can easily hijack their
| connections without needing a password or 2FA and there are no
| _lastlog_ entries. SSH Session multiplexing combined with
| passwordless sudo makes taking over a company trivial even if
| they have 2FA and strong passwords.
|
| Another risk with a bastion model is port forwarding. As an
| organization you have to decide what is appropriate for that
| bastion. Unrestricted forwarding? Restricted? Denied?
| AllowAgentForwarding no
| AllowTcpForwarding yes PermitOpen
| 192.168.1.2:22
|
| If this bastion is for a PCI environment then one may want
| tighter restrictions. If it is for a development environment then
| maybe less restrictions and just better auditing on each host to
| enable forensic remediation.
|
| If your bastion is also used for automation to drop files into a
| staging area, you can limit that automation to file transfers and
| even limit what it may do with files. This prevents the
| automation from having a shell or performing port forwarding.
|
| The keys should be outside of the home directories to prevent
| malicious tools from appending additional authorized_keys into
| the account. Make use of automation to manage key trusts and add
| a comment to keys to map them to an internal tracking system like
| Jira. This assumes your MFA/2FA is excluding specific accounts or
| groups via PAM and permitting the use of ssh keys with specific
| groups or accounts. AuthorizedKeysFile
| /etc/ssh/keys/%u Match Group
| sftpusers Banner
| /etc/ssh/banner_sftp.txt PubkeyAuthentication
| yes PasswordAuthentication no
| PermitEmptyPasswords no GatewayPorts
| no ChrootDirectory /data/sftphome/%u
| ForceCommand internal-sftp -l DEBUG1 -f AUTHPRIV -P
| symlink,hardlink,fsync,rmdir,remove,rename,posix-rename
| AllowTcpForwarding no AllowAgentForwarding
| no
|
| -P sets limits on what may not be done in sftp. -p does the
| inverse and limits what may be done. [1] -l DEBUG1 or VERBOSE
| will give you syslog entries of what commands were executed on
| the files. This is useful for audits. Some redundant settings
| above are also useful to set explicitly for audits.
|
| Another thing mentioned in the article is iptables. In a PCI
| environment one may want to also have explicit outbound rules
| using the _owner_ module to limit what users or groups are
| permitted to ssh out. So if your organization have a group of
| people allowed to use this host as a bastions, then one could
| write a rule like iptables -I OUTPUT -m owner
| --gid-owner devops -p tcp --dport 22 -d 192.168.0.0/16 -j ACCEPT
|
| Or specify what CIDR blocks, ports, protocols may be used. You
| can use REJECT rules after this rule to make it obvious a
| connection was not allowed so that people do not spend hours
| debugging. This module is also handy for limiting which daemons
| may speak to your infrastructure. How strict or liberal the rule
| is entirely at the needs of your organization.
|
| Lastly I would add that bastions should have as minimal an OS
| install possible and have SELinux enforcing. Actions denied by
| SELinux should go to a security operations center after you spend
| some time tuning out the noise and false positives.
|
| [1] - https://man7.org/linux/man-pages/man8/sftp-server.8.html
| whynotminot wrote:
| I wish we could save posts. So this reply is my method...
| thanks for the write up.
| tsuraan wrote:
| You can click the timestamp on a post, and then click the
| "favorite" link, and that'll add the comment to your
| favorites list (which I think would be https://news.ycombinat
| or.com/favorites?id=whynotminot&commen... for you).
| whynotminot wrote:
| TIL! Thank you very much
| mindcrime wrote:
| You can always just upvote the comment. Your profile page has
| a link to see comments (and stories) you've upvoted in the
| past. See:
|
| https://news.ycombinator.com/upvoted?id=YOURUSERNAME&comment.
| ..
| unixhero wrote:
| Thanks a lot, great hardening considerations.
|
| It would be interesting to hear what you think of Keykloak.
| LinuxBender wrote:
| Sorry I have never used it so I don't have an opinion. That
| looks like an oauth/openid/saml ssh integration?
| vladdoster wrote:
| A superset of these best practices in the article would be CIS
| benchmarks. Collectively agreed on by industry leaders and
| provide extensive resources that span the gamut of cloud,
| networking, and storage infrastructure.
|
| CIS supported technologies: https://www.cisecurity.org/cis-
| benchmarks
|
| CIS Audit AWS infra: https://github.com/toniblyx/prowler
|
| Better to be _proactive_ than _reactive_ :^)
| belter wrote:
| If you are on AWS you dont need bastion hosts anymore. Use
| Session Manager.
| acdha wrote:
| I agree in general but there are a handful of edge cases
| which Google solved better with IAP: SSM can't forward ports
| to other hosts or any resource other than EC2. It's great for
| using SSH, SFTP, even tools like Ansible work fine, but if
| you need to get a port forward to something like RDS, a
| service in Fargate, etc. you'll need something else.
| alecco wrote:
| Twice I've seen Bastion Hosts compromised. Both times it
| practically gave the attackers the highest access. In one case it
| basically hid where the attack came from (compromised logs and
| all). In another it let them hijack an admin's password by
| reading his sudo.
|
| IMHE, Bastion Hosts suck.
|
| If you are forced to use one, send logs to a safer one-way
| storage encrypted and put tampering triggers everywhere you can
| in the Bastion Host. Also make sure you log outgoing connections.
| And make sure you can easily match incoming to outgoing.
|
| If you absolutely have to use sudo on the Bastion Host force it
| to OTP only. Or if absolutely not possible, use 2FA, but this is
| a risk as something somewhere might not be properly protected and
| the password will leak. But the better way would be to have the
| bastion host run on some read-only image and not letting it
| upgrade or do any admin task at all. Maybe even remove admin
| users, SSH, the whole lot.
|
| And related, do not have a single account with god-like access to
| everything. Isolate permissions. This is probably the hardest to
| get OK'd but it's the classic SPOF where they got you by the
| balls.
| llama052 wrote:
| IMHE, Bastion Hosts suck.
|
| I agree, any security standards you're going to apply to a
| bastion host, just apply them to your entire network if
| possible, add security at every layer. So many times a bastion
| host just serves as a checkbox with added toil of jumping
| through a host. I despise them for the most part.
| loudmax wrote:
| I can see that you can get a lot of things wrong with a bastion
| host, but if implemented sensibly, it should just be one more
| layer of a defense-in-depth strategy. What would you recommend
| _instead_ of a bastion host?
| mberning wrote:
| Having seen how bastion hosts or "jump boxes" work inside the
| enterprise I share your view. In practice they are generally
| not very well protected and are a very attractive target for
| attackers. It's better to use a privileged session manager or
| regular ssh with mfa and ideally some type of identity
| proofing.
| gz5 wrote:
| very nice writeup - one of the better ones i have seen. you can
| go a step further and eliminate open inbound port 22 (make the
| sshd server 'dark' to the network) with open source solutions
| like this:
|
| https://ziti.dev/blog/zitifying-ssh/
|
| disclosure: we build SaaS on top of OpenZiti (the open source) so
| are opinionated in this domain. and, to be clear, the above is
| just one layer...other layers of security still apply.
| dovholuknf wrote:
| i generally end up liking what teleport is doing and what they
| are all about... i keep meaning to try their opensource stuff
| out. does teleport's sshd 'listen' on port 22 and does it need
| an opening in a firewall?
|
| still, having sshd listen on localhost and not a public ip is
| pretty cool imo. Ken and I did exactly that on a stream one day
| https://youtu.be/oSlwZcwZcsU if anyone is interested. The one
| extra step one could do is to convert sshd to only allow
| connections from localhost by editing /etc/ssh/sshd_config and
| set the ListenAddress to only 127.0.0.1
|
| Make those bastions dark!
| kevin_nisbet wrote:
| > does teleport's sshd 'listen' on port 22 and does it need
| an opening in a firewall?
|
| Sorry, one of those crappy it depends answers. The teleport
| node agents, the agent running on the server you want a
| session on, can be configured to listen to inbound
| connections from the proxy (but doesn't use port 22 by
| default), or can be configured in a reverse tunneling mode
| where it does outbound dialing towards the Teleport proxy
| service. When using the reverse tunneling mode, you don't
| need inbound access to the end nodes, but still need the
| nodes to be able to make an outward connection to the
| Teleport infrastructure.
|
| This is how the cloud hosted Teleport works as well, we can't
| be expected to have outbound network access to peoples
| machines, so all the agents will dial the cloud hosted
| proxies, and setup reverse tunnels that are then used for the
| inbound connection requests.
|
| In most setups though, the Teleport Proxies would then still
| have inbound connectivity and are meant to be internet
| facing, so a client can request an SSH or other session, but
| that single way into the environment can be hardened, layered
| with additional security, as the environment may require.
|
| Note: I'm affiliated with Teleport, my comments are my own.
| mrweasel wrote:
| There's a few weird things, but it's mostly okay.
|
| Do not trust the firewall on the bastion host, if an attack can
| get into the bastion host, they can disable the firewall, so it
| cannot be used to limit egress. It's better than nothing, but
| consider using a firewall that's managed on a via a separate
| management network. I do agree that you should only allow SSH
| from a few known IPs.
|
| Limiting the number of users is weird, and not recommended.
| Create all the accounts you need to provide individual accounts
| for the staff that need to access the bastion host, you will need
| that as things like HIPAA require named accounts for auditing.
| None of the accounts need any privileges other than the most
| basic. Users do not need sudo/root privileges on a jump host.
|
| Other than those two complains, it's good recommendations.
|
| A final recommendation: If you use AWS though, consider using
| Session Manager instead of SSH and drop the bastion host. You can
| still connect using the SSH command, using proxy command in
| OpenSSH, but no public IP or bastion host is required.
| staticassertion wrote:
| > if an attack can get into the bastion host, they can disable
| the firewall, so it cannot be used to limit egress.
|
| This assumes that the attacker can get unconstrained root
| access to the system. It's fine to assume that attackers will
| but it's not as if you can't make that difficult.
|
| Agree with the rest of what you said though.
| vngzs wrote:
| From a defense standpoint, one should consider "shell on a
| box" to _usually_ mean attackers can get root on a box. If
| they can get persistence, they can wait for a kernel CVE to
| abuse.
|
| Now, if you're just using a bastion as a jump host, you don't
| need to offer shells on it. Just allow people to proxy a port
| to behind the bastion and be done with it.
| PermitTTY no ForceCommand /usr/sbin/nologin
| AllowTcpForwarding yes AllowAgentForwarding no
| staticassertion wrote:
| I think it's probably reasonable when performing your
| incident response or even threat modeling to assume the
| attacker has or could escalate privileges. The linked
| article doesn't discuss anything that would make that
| harder, although perhaps practices like staying patched and
| minimizing attack surface are somewhat assumed (they do
| bring up choosing your OS based on minimizing attack
| surface for example).
|
| There's also a lot you can do to harden that boundary. You
| can harden your kernel, you can execute user's shells in
| constrained environments like docker containers or
| restricted shells, leverage sandboxing technologies like
| apparmor or selinux, etc.
|
| The user/root boundary can be a lot thinner than people
| expect, so I get why you'd want to point out that reliance
| on the attacker not escalating should be met with an
| evaluation of that boundary, but I think it may be
| understating the boundary to unconditionally not trust a
| host based firewall, or to say that getting onto the
| bastion itself is enough to disable the firewall when it
| does indeed require escalation.
| rorski wrote:
| > A final recommendation: If you use AWS though, consider using
| Session Manager instead of SSH and drop the bastion host. You
| can still connect using the SSH command, using proxy command in
| OpenSSH, but no public IP or bastion host is required.
|
| Yes, this. Also check out https://github.com/rewindio/aws-
| connect for a convenient wrapper around SSM to make it easier
| to use (I'm not the author).
| karimfan wrote:
| Or you could just use strongdm!
| servercobra wrote:
| Can someone explain to me the benefits of limiting the IPs that
| can SSH into the bastion? It seems to me the main thing that's
| protecting against are misconfigurations of SSH (accidentally
| letting root log in with no password or something) or a zero day
| in SSH but I'm not convinced by either.
| xvector wrote:
| > misconfigurations of SSH (accidentally letting root log in
| with no password or something) or a zero day in SSH
|
| these are entirely valid concerns. defense in depth, principal
| of least privilege. humans make errors.
| AtNightWeCode wrote:
| What do you want to be paranoid about? One access point or a
| million access points?
| talideon wrote:
| The company I work for does it so that bastions hosted on some
| public cloud hosting service are only accessible from the
| company network or by machines connected to its VPN. We handle
| _very_ sensitive data, and some engineer screwing up the
| configuration for a bastion would be _very_ bad. Defence in
| depth is important.
___________________________________________________________________
(page generated 2022-01-13 23:00 UTC)