[HN Gopher] Remote Code Execution in OpenSSH's forwarded SSH-agent
___________________________________________________________________
Remote Code Execution in OpenSSH's forwarded SSH-agent
Author : vitplister
Score : 107 points
Date : 2023-07-19 17:36 UTC (5 hours ago)
(HTM) web link (blog.qualys.com)
(TXT) w3m dump (blog.qualys.com)
| 1letterunixname wrote:
| I use a USB security key for gpg-agent.
|
| gpg-agent is used as an ssh-agent replacement (also replaces the
| venerable monkeysphere script). gpg agent forwarding is also
| needed via StreamLocalBindUnlink for signing built RPMs in a
| docker container on a remote (local LAN) Docker host. Sometimes
| rarely, I need to enable "ssh-agent" forwarding where I probably
| could proxyjump instead. My interactive ssh workflow is sometimes
| slowed as I use TOTP (via PAM) for 2FA. Noninteractive system
| accounts get dedicated ssh keys. host keys are constantly scanned
| and compared to a source of truth. Maybe I'll get around to
| deploying LDAP with OpenSSH-LPK to move away from flat files.
|
| _Knee-jerk suggestion to rewrite the world in Rust combined with
| formal verification._
|
| But seriously, being an expert knife juggler is still gambling
| with the obvious compared to using safer tools in a safer manner.
| Rust needs the ubiquity of GCC[0] (partially by adding them to
| LLVM and adding std crates) and more attention paid to bloat
| (cargo-bloat, etc) before attempting to rewrite the world (apart
| from special cases).
|
| 0. https://gcc.gnu.org/backends.html
| apienx wrote:
| Agent forwarding is discouraged by the OpenSSH crew. Yet, it's
| commonly used because of the convenience it affords.
|
| "Agent forwarding should be enabled with caution. Users with the
| ability to bypass file permissions on the remote host (for the
| agent's UNIX-domain socket) can access the local agent through
| the forwarded connection." https://man.openbsd.org/ssh.1
|
| Glad the vulnerability got fixed.
| shandor wrote:
| I can't parse the exact meaning of the "victim system" and
| "attacker controlled system" in the OpenSSH release page.
|
| Does this vulnerability allow the attacker to compromise the
| original system where the user starts the agent-forwarded
| connection? Or "only" compromise machines forward from the jump
| host?
|
| http://www.openssh.com/releasenotes.html#9.3p2
| perlgeek wrote:
| If an attacker can compromise the jump host, they can
| compromise the system where the user starts the agent-forwarded
| connection.
| shandor wrote:
| Ouch, that's horrible.
|
| Time to prompt everyone at $WORK to upgrade...
| Arnavion wrote:
| To be clear, agent-forwarding to a (potentially) malicious
| ssh server has always been a bad idea. Yes TFA's bug makes
| it a _worse_ idea and it 's absolutely worth it to patch
| it, but you should not be agent-forwarding to (potentially)
| malicious ssh servers in the first place.
| gunapologist99 wrote:
| That's 100% correct. This is why no one should use agent
| forwarding with a jumpbox. Only -J. (see
| https://userify.com/docs/jumpbox )
|
| Another point is that no one should have any real access to
| the jumpbox and it should be as minimal and stripped-down as
| possible. It's literally your bastion host, so you've got to
| keep it as strong as possible.
| slt2021 wrote:
| what is CVSS/severity of this? Critical? 9+?
| tptacek wrote:
| CVSS is meaningless, literally a Ouija board that reflects the
| intuition of whoever's computing the score. A more reasonable
| way to look at severity is on a two dimensional scale of lo-hi
| severity and lo-hi situationality. This is a high-severity,
| moderately situational bug (it would be highly situational if
| it required user interaction, and not situational at all if it
| didn't require control over the remote SSH server).
| tedunangst wrote:
| CVSS with rice: 10/10
| slt2021 wrote:
| do you have a source for this by any chance? or screenshot?
|
| thank you
| wiml wrote:
| He's just making a reference to an old reddit joke, I don't
| think it's a CVSS 10
| NoZebra120vClip wrote:
| CVSS are bunk. https://portswigger.net/daily-swig/cvss-system-
| criticized-fo...
| gunapologist99 wrote:
| Note: you are not vulnerable if you're not doing agent forwarding
| (-A).
|
| Honestly, you probably should never do -A anyway; use -J
| (proxyjump) instead.
|
| https://www.man7.org/linux/man-pages/man1/ssh.1.html
|
| https://userify.com/docs/jumpbox
| pcthrowaway wrote:
| There are situations where `-J` won't work.
|
| For example, if I'm on a remote server and I want to clone or
| pull from a private repository, or push to any repository
|
| In fact, VS code expects you to forward an agent when
| developing on a remote instance with the remote-ssh extension.
| I believe it uses it for syncing repo changes primarily
|
| I agree though that you should consider it insecure to forward
| a connection to a shared host if it has other users who are
| equally privileged, or more privileged
| gunapologist99 wrote:
| > There are situations where `-J` won't work.
|
| That is true, but actually you should avoid -A in those
| situations also.
|
| > In fact, VS code expects you to forward an agent when
| developing on a remote instance with the remote-ssh
| extension. I believe it uses it for syncing repo changes
| primarily
|
| That's the case if you're ever trying to initiate an SSH
| connection via Git _from_ a remote host. That 's because the
| actual SSH connection isn't being triggered on your desktop
| or laptop, where -J would do it, but because it's being
| initiated right on the remote server itself.
|
| Let's call the remote server VSCODE (your desktop-in-the-
| cloud), your laptop/desktop LAPTOP, and the remote repo
| GITHUB.
|
| So, the correct answer there is actually counter-intuitive:
|
| Generate a private key on VSCODE (not LAPTOP) for that GITHUB
| repo. This key will be used only on that originating remote
| server (VSCODE), never anywhere else.
|
| In fact, that repo key will ideally be scoped as tightly as
| possible in Github/Gitlab/etc.
|
| As a general rule of thumb, generate at least one SSH key for
| each "location" where you'll be logging into another
| location. Keep your connections point-to-point unless you can
| use proxyjump.
| pcthrowaway wrote:
| > Generate a private key on VSCODE (not LAPTOP) for that
| GITHUB repo. This key will be used only on that originating
| remote server (VSCODE), never anywhere else.
|
| Deploy keys are more secure in most situations, but I'm not
| convinced that generating a bunch of additional keys with
| write access to your repo is _always_ more secure.
|
| Regardless, it doesn't really fulfill the same need of "get
| access to repo from cloud desktop asap". I work with 20-30
| private repos at my company, and I don't have admin access
| to most of them (so I couldn't create deploy keys for
| them). That would need to be kicked up and down a request
| pipeline, for each user accessing remotely, for each host
| they're accessing it from. It's _much_ easier to Get The
| Work Done if I just forward my agent.
|
| And sure, the company could maybe set up better practices
| (though again, I'm not even really clear on how much more
| secure it would be to generate untold numbers of deploy
| keys), but I'm more likely to get the axe for not producing
| fast enough than they are to make sweeping changes to their
| security practices.
| gunapologist99 wrote:
| Then just create a key on that box and use it for _all_
| of your repos (add it to your github keychain); you 're
| still much better off than with one key to rule them all
| (and one RCE to rule your laptop ha)
| pcthrowaway wrote:
| That's a good point, probably the best way to do this,
| but does require manual management steps for every VM you
| want to use (which is perhaps a feature)
| yjftsjthsd-h wrote:
| > Honestly, you probably should never do -A anyway; use -J
| (proxyjump) instead.
|
| I want to copy files between servers, ex. `ssh -A serverA` then
| `rsync ./foo serverB:`. Is there a way to do that without `ssh
| -A`? (And obviously without creating an actual private SSH key
| on either server)
| gunapologist99 wrote:
| Yes. Create a temporary (or permanent) private key on server
| A and then put its public key in server B.
|
| That's the correct way to handle this situation, definitely
| not -A.
| berkle4455 wrote:
| What's a better solution if you want to be able to SSH across
| multiple machines? Do you need to always close the current
| connection to get back to localhost prior to a fresh SSH?
|
| e.g. how would I ssh into foo, and then later into bar, or
| perhaps pull some code from github onto foo that is authenticated
| by my key? localhost -> foo -> bar
| localhost -> foo -> github access
|
| It seems like the answer is either: a) ssh -A, or b) install my
| private key on foo.
| gunapologist99 wrote:
| Better options:
|
| c) use proxyjump (-J) to access bar
| (https://www.man7.org/linux/man-pages/man1/ssh.1.html), and/or
|
| d) generate a new private key on foo and use that to access
| github.
|
| You might think that's a crazy waste of time compared to
| forwarding, but it's actually much, much safer and actually
| only takes a moment: ssh-keygen -t ed25519
| cat ~/.ssh/id_ed25519.pub
|
| and then just paste that in Github.
|
| See also https://userify.com/docs/jumpbox for more jumpbox
| docs.
| cassianoleal wrote:
| As others have pointed out already, JumpHost or -J is the
| preferred way.
| [deleted]
| gunapologist99 wrote:
| I agree!
|
| The Github example complicates things, but the correct
| solution is actually another key:
|
| Generate a separate private key on foo and place that one in
| github, ideally per project as a deploy key.
| totallywrong wrote:
| You can set up an ssh tunnel on localhost to any port on bar or
| github via foo.
| [deleted]
| slt2021 wrote:
| I use hashi corp's stack: Boundary + Vault for ssh credential
| injection
|
| https://developer.hashicorp.com/boundary/tutorials/credentia...
|
| it completely replaces the need for bastion hosts and ssh-agent
| at scale (I'm a hashicorp's paying customer)
|
| compared to Teleport, boundary is lightweight (you dont need to
| install agent on each target server) and is integrated with the
| Vault for certificate based auth or simple cred brokering
| wahern wrote:
| I wrote a macOS app that supports using Vault Transit Engine
| keys with OpenSSH, as well as GnuPG and PKCS#11. (OpenSSH is
| supported via both the agent protocol and PKCS#11.) You can
| even use an Apple T2 (Secure Enclave) key for peer mTLS
| authentication to the Vault server. https://www.keymux.com/
|
| Using this teams can _share_ an SSH key without exposing the
| key; nor do you need to configure certificate PKI, use jump
| hosts, or otherwise change your existing software or
| workflows.
| aberoham wrote:
| Teleport is far and away the leading solution in this area --
| certificate based end-to-end bastion style topology with many
| nice features including support for kubectl.
| throwawaaarrgh wrote:
| Make an ssh connection to foo with a port forward. Then make an
| ssh connection to bar through foo. Keys stay on your machine.
| I'm pretty sure there's built in features to do this.
| gunapologist99 wrote:
| That won't work for a connection to github that's being
| initiated from foo.
|
| The best solution in that situation is a separate key that's
| private to foo.
| binkHN wrote:
| Full details:
| https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh...
| aidenn0 wrote:
| Even without this announcement, friends don't let friends forward
| their ssh agent. It essentially grants that machine access to
| your private keys. A RCE vulnerability is strictly worse than key
| exposure, but you probably shouldn't have been using it anyways.
| starfallg wrote:
| Yup, and this is explained in detail in the ssh(1) man page.
| No-one should be using -A. Using a jump host via -J is the way
| to go.
| Arnavion wrote:
| One use of -A, namely using the ssh server as a jumphost, is
| covered by -J. The general use of -A, namely doing operations
| on the ssh server that require keys from the ssh client, is
| not.
|
| If I'm on machine foo and I want to connect to
| bar.example.org and clone a git repo there from
| baz.example.org, and baz.example.org requires an identity key
| that is in foo's ssh agent, then -A is the only option.
| gunapologist99 wrote:
| No, you might as well copy the keys over there for all the
| security you're getting (actually, that's safer given this
| RCE which compromises both sides, even given that there's
| no real way to shred the leaked key, but of course I'm not
| actually suggesting this! they're both really bad.)
|
| Your best option is too easy: just generate a new keypair
| on foo.
|
| Then you can populate baz.example.org with that public key
| instead of your own.
| Arnavion wrote:
| >No, you might as well copy the keys over there
|
| Only if the keys are insecure enough to be copied, ie not
| backed by an HSM, or backed by an HSM but marked
| exportable.
|
| >for all the security you're getting (actually, that's
| safer given this RCE which compromises both sides, even
| given that there's no real way to shred the leaked key,
| but of course I'm not actually suggesting this! they're
| both really bad.)
|
| Connecting to a malicious SSH server is already
| dangeorous with or without this vulnerability, and with
| or without agent forwarding. Eg a malicious server can
| modify the shell to emit VT codes to mess up your
| terminal.
|
| >Your best option is too easy: just generate a new
| keypair on foo.
|
| Sure, but this creates additional identities that
| baz.example.org must be taught to trust, which is not
| always desirable or even an option.
| gunapologist99 wrote:
| > Only if the keys are insecure enough to be copied, ie
| not backed by an HSM, or backed by an HSM but marked
| exportable.
|
| Agent forwarding forwards _the keys_ insecurely. If you
| can launch an agent-forwarded connection, then the HSM is
| already irrelevant.
|
| > "Connecting to a malicious SSH server" is already
| dangeorous with or without this vulnerability, and with
| or without agent forwarding. Eg a malicious server can
| modify the shell to emit VT codes to mess up your
| terminal.
|
| yes, but messing up your terminal is not an RCE (and you
| should be able to fix it with stty sane). It's just an
| annoyance and just tipped you off that obviously there's
| something _wrong_ there. :) Obviously you should be
| checking your host keys etc and ideally you woud never
| accidentally connect to a malicious server.
|
| However, in the unlucky but perhaps inevitable event that
| you do connect to a malicious server, you shouldn't risk
| exposing your entire client workstation! (through this or
| another RCE) or your SSH private keys! (through normal
| agent forwarding operation) with agent forwarding.
|
| > Sure, but this creates additional identities that
| baz.example.org must be taught to trust, which is not
| always desirable or even an option.
|
| A key is not necessarily an identity. Most platforms
| permit more than one public key to be associated with a
| user, including github, gitlab, userify, etc.
|
| However, ideally you are correct -- this would result in
| a new user account that has limited access to only the
| things it really needs (principle of least privilege).
| Arnavion wrote:
| >Agent forwarding forwards the keys insecurely. If you
| can launch an agent-forwarded connection, then the HSM is
| already irrelevant.
|
| No? ssh-agent works fine with keys that are not
| exportable from the HSM. By the very definition it's not
| possible for anything to export the key, ssh-agent or
| otherwise.
|
| >yes, but messing up your terminal is not an RCE
|
| It has been, and can be again.
|
| https://www.cyberark.com/resources/threat-research-
| blog/dont...
| drdaeman wrote:
| Another relatively popular use case is pam_ssh_agent_auth
| for passwordless-but-authenticated sudo. This, by
| definition, requires agent forwarding.
|
| (I don't use ssh-agent, I use gpg-agent with an SSH socket
| - I suppose I'm fine?)
| GauntletWizard wrote:
| ssh-askpass is a great tool in these situations. It
| intercedes and allows you to manually control all key usage
| attempts.
| Arnavion wrote:
| I'm not sure what relevance ssh-askpass has to what I
| wrote.
| GauntletWizard wrote:
| Right, this recipe isn't well known.
|
| `ssh-add -c` will cause your ssh agent to pop up with
| ssh-askpass every time it does an authentication. This is
| fiddly; You need to have configured your ssh-agent right,
| and there's a slightly different version to use keychains
| on macos that's equivalent.
| Arnavion wrote:
| Once again, I'm not sure what relevance this has to the
| discussion.
| devnullbrain wrote:
| Ah, a skillful execution of Cunningham's Law
| willbicks wrote:
| I prefer using hardware tokens (in most cases a PKCS#11 smart
| card) because it means that even with a forwarded SSH agent,
| every request to use my private key requires a PIN on my client
| which is verified by the isolated cryptographic processor. It's
| impossible for my private key to leave that card and get cached
| anywhere else. While I haven't enabled it on my Yubikey I
| understand they can do similar.
|
| The downside is that compatibility in edge cases, while much
| better than I'd expect, is still not perfect. In particular
| Windows support outside of Putty gets challenging.
| gorkish wrote:
| The RCE is related to ssh-agent's support for PKCS#11, so,
| yeah you are right this is a valid method to prevent key
| access or theft via the agent (I also have to approve every
| use of my PK), but in this case it's not protecting against
| the RCE, and the workaround in the meantime is to disable
| PKCS#11 `ssh-agent -P ''`
| gunapologist99 wrote:
| That won't save you for this RCE!
| toast0 wrote:
| The other downside is it's much harder to do bulk operations
| against a fleet. It's not reasonable to enter a PIN for each
| access when you need to push something to 1000 nodes. 100
| nodes is probably ok, but not great.
| throwawaaarrgh wrote:
| or you could just use the -c option to ssh-add and be
| prompted every time the key is handed over
| salawat wrote:
| Real talk though.
|
| How do you manage ssh chaining then? Unique set of keys per
| user per machine?
|
| Short of scp'ing your private key over from your starting box,
| I can't really think of another way. Then again, if it's not
| your box, (you don't own the hardware have sole monopoly of
| root), you probably shouldn't be ssh'ing from it anyway. I've
| always held there are no true secrets on a computer... Until
| multi-billion dollar companies decide to collude anyway.
|
| Anywho, honest question. I'm a big fan of ssh roaming, never
| realized that ssh-agent was a thing, but after reading this, if
| I were to use it, I'd most certainly be doing it witout pkcs11
| built in. Shotgunning shared libs for side-effects is
| absolutely mad. Need to slot that into my source code reading
| list.
| jcotton42 wrote:
| Depending on your usecase, ProxyJump may suffice
| https://www.redhat.com/sysadmin/ssh-proxy-bastion-proxyjump
| aidenn0 wrote:
| > How do you manage ssh chaining then
|
| ProxyJump in the .ssh/config file. If your SSH is too old to
| support ProxyJump, you should probably upgrade, but this
| works as well: ProxyCommand ssh -q -W %h:%p
| jumpost
| salawat wrote:
| Noted... Guess I know what I'm doing today!
|
| Edit: Nevermind, tried it, not sufficient for my use case.
|
| I tend to do a lot of mesh-y bouncing around between
| servers, and -J seems to be more intended for a
| star/hub&spoke topology. Common ssh priv-key to all
| machines, or alternatively, a unique set of priv_keys per
| user per dest machine is about the way to go. You still
| have privilege escalations to worry about, but thems the
| breaks.
|
| It's a neat trick, I'll give it that.
| aidenn0 wrote:
| It's not limited to star topologies; You can use
| ProxyJump for any topology that includes fixed routes
| from a client to a host. Just add a separate host entry
| for each machine.
|
| 4 server example (this assumes your client can connect to
| only host C) with the following topology:
| C -> B -> A \-> D
|
| ssh configuration snippet: Host A
| ProxyJump B Host B ProxyJump C
| Host D ProxyJump C
|
| This example is a tree-like topolgy, but you can use host
| aliases (i.e. add a HostName that is different from the
| host entry) to define any fixed route to any machine you
| like.
| saltcured wrote:
| For me, the main use of agent-forwarding is that I need
| to use a command that expects to use SSH to get between
| leaf nodes. For example git or rsync CLIs that need to
| manipulate the local filesystem and tunnel their own
| protocol over SSH to talk to another remote server.
|
| At times, I've wished for something like uMatrix but for
| ssh-agent forwarding, so I could have policies for which
| peer-to-peer authentications should be allowed for which
| keys and whether these specific uses should require
| interactive confirmation.
| gunapologist99 wrote:
| Just generate a new keypair there on your bounce box
| then. Don't do -A because this RCE means that not only do
| you lose your keys, but you lose your laptop, too!
| TechBro8615 wrote:
| I think SSH agent forwarding is a fairly common setup with
| VSCode Remote, where the developer wants to forward their keys
| to the remote dev server so that they can perform Git
| operations using their credentials.
| gunapologist99 wrote:
| Easy fix: generate private keys on the remote dev server and
| then use those keys instead of your own with git.
| lxgr wrote:
| This exposes your private keys to a potentially less-
| trusted dev server. It also just doesn't work with hardware
| tokens for the same reason.
| WhackyIdeas wrote:
| Am I reading this right - any system with ssh-agent installed by
| default is vulnerable?
|
| The link is short on specifics.
| kam wrote:
| It's exploitable only by a SSH server that you connect to with
| agent forwarding enabled (i.e. one that you're already trusting
| with access to your SSH keys).
| jonas-w wrote:
| If I read this correctly it is only when you forward your ssh-
| agent with the "-A" or "ForwardAgent" option
| tetha wrote:
| As far as I read it, it's about _forwarded_ ssh agents.
| Basically, if you `ssh -A user@system`, something might be able
| to execute commands locally. For example, this might turn messy
| for infrastructures using jump-hosts extensively, if people are
| used to <ssh -A jumphost> so they can easily <ssh system>
| afterwards. If you pop the jump host, you could pivot to the
| workstations with this.
|
| At the same time, ssh-agent forwarding makes me queasy from a
| security perspective even without this. As far as I know, if
| you <ssh -A> into a system, admins with privileges on the
| system can gain access to your local ssh-agent already. In the
| example of the jump host, if you popped the jump host and stuck
| around for a while, you could probably harvest SSH keys and
| have some fun later.
| spladug wrote:
| FWIW, `ProxyJump` is a safer way to go through a bastion
| without having to expose your agent to either the bastion or
| the target host behind it.
| chaosite wrote:
| And since the person you're replying to was mentioning
| command-line parameters, it's worth mentioning that this
| can be done with `ssh -J jumphost user@system`.
| tetha wrote:
| Useful, I didn't know that so far.
|
| We don't use bastion servers. My only real use case for ssh
| agent forwarding is if I need some scp / rsync between two
| remote systems during emergencies and those systems have no
| trust via SSH keys setup between them. In that very
| specific case, I don't know a better way than <ssh -A> to
| the first system and have some <rsync -e ssh> from there to
| the second system. Still doesn't feel great, even though I
| know only the people who could steal my keys are on my
| team.
| spladug wrote:
| Ah yeah. Not sure on that one. scp does have the `-3`
| option to copy between two remote hosts via the local
| host, but that can be significantly slower if the remote
| hosts are in the same network and local host is not.
| tetha wrote:
| Exactly. If I need to move a few megabytes around, <scp
| -3> and a coffee or a few simple tickets is a good way. A
| year ago or so, I needed 600GB moved between two systems
| ASAP during an outage that'd turn into a money-bleed at
| 6am. If I piped that through the VPN and my workstation,
| I'd probably still be waiting today.
| LinuxBender wrote:
| Some time take a look at lftp [1] _and its mirror
| subsystem_ for this. It can break up a batch of files or
| even one large file into multiple SFTP streams. Another
| upside is that it can replicate most rsync behavior in a
| SFTP-Only Chroot account. Downside is that without a
| corresponding daemon _like rsync_ on the other end
| directory enumeration is slow which isn 't a problem if
| one does not have a complex directory structure.
|
| Play around with the built in rate limit options _total
| and per thread_ to keep the network people happy.
|
| [1] - https://linux.die.net/man/1/lftp
| tedunangst wrote:
| Well, any system that auto maps the stack executable when
| loading a library. There are details on the specific
| requirements in the advisory.
|
| https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh...
| more_corn wrote:
| I'd you forward your agent to a remote host, that host can mess
| with you.
| sullivanmatt wrote:
| This sounds way worse than it is.
|
| To be clear, the "remote" part of the code execution is that an
| attacker controlling your destination server can cause your
| _client_ to run an attacker-controlled payload, _if_ the client
| is forwarding their credentials (`ssh -A`). Most people don 't
| tend to make connections to arbitrary SSH hosts, and certainly
| they don't do it while forwarding their credentials along.
|
| It's a neat attack, and I applaud the Qualys team on their find,
| but this is not any sort of emergency situation for 99.99% of
| systems.
| justsomeadvice0 wrote:
| Lots of people end up with AgentForward on by default as a sort
| of "make it work" fix, and lots of people use `git+ssh` on
| untrusted servers. Here's an example:
|
| https://abyssdomain.expert/@filippo/109659699817863532
|
| TBF this is a vulnerable config either way; but RCE on the
| client shouldn't be possible.
| tinus_hn wrote:
| I'm not so sure git is secure against a malicious server,
| even if you're not simply pulling in a Makefile written by
| the attacker.
| gunapologist99 wrote:
| I beg to differ: this does _not_ sound way worse than it is. If
| anything, it 's understating the issue.
|
| Not only can it be exploited across a wide variety of clients
| across multiple platforms, but all that's required is that
| you're using key forwarding.
|
| This is devastating, because it's not just that you control the
| destination server and steal the keys, but you can take over
| the user's entire workstation.
|
| Once you've got the user's entire workstation, you potentially
| have access to everything else they have, from their email, to
| other SSH hosts, to key loggers, to Git repos. This is about as
| bad as it gets, and all because someone is using Agent
| Forwarding.
|
| Best of all, the victim has _no idea_ that they 've been
| completely compromised. They can live inside your machine for
| years, upgrade their sploits, and generally exfiltrate all of
| your secrets.
|
| Never use agent forwarding. Just don't. "Agent forwarding
| should be enabled with caution" in the man page is another
| massive understatement. Even if you _think_ you need it, check
| the other responses in this thread for examples of how to work
| around it.
| dumpsterdiver wrote:
| > Never use agent forwarding
|
| Agreed. As this exploit proves, it's not even safe to log
| into your own servers using ssh forwarding if any service is
| exposed remotely, because if an attacker compromises that
| exposed service and gains root then they could extend the
| attack to your workstation, and that's a huge deal -
| especially considering that you have the private key to log
| into that server on your computer (so it's not an unsafe bet
| there might be other keys).
| TechBro8615 wrote:
| The attacker controlled destination server could be a
| compromised host, so this enables lateral movement from a
| deployed VM or remote dev machine into a developer laptop.
| gunapologist99 wrote:
| Aren't all SSH hosts potentially attacker-controlled? ;)
| nocsi wrote:
| Yea, but there's a security boundary wherein you don't want
| the SSH host to be executing code in _your_ environment. Of
| course, the attackers can backdoor sshd to log credentials,
| setup init scripts on the host to execute code every client
| login and other shenanigans.
| gunapologist99 wrote:
| Of course. So it's not really that out of the question _if_
| you are using agent forwarding, so, yeah, this is a big
| deal.
| malux85 wrote:
| Yeah if I'm reading the technical analysis right, your
| conditions that you mention have to be correct and also the
| attacker must have "poisoned" library files on the targets
| machine so they can dlopen them, is that right?
|
| Pretty unlikely
| Arnavion wrote:
| The libraries are on the client's machine, not the server's.
| And they're not "poisoned"; the default distro-provided libs
| already provide the remote execution capabiity (eclipse-
| titan, libkf5sonnetui5, libns3-3v5 and systemd-boot packages
| from Ubuntu 22.04).
| malux85 wrote:
| Ahh I see I thought the attacker also had to have custom
| malicious libs deployed on the client machine I wasn't sure
| if standard ones would do, thanks for clarifying that
| sullivanmatt wrote:
| There must be a specific set of libs present on the victim
| (client), correct. Qualys claims that stock Ubuntu Desktop
| systems often have these libs, and that they haven't looked
| into whether other distros tend to.
|
| But yes, your point stands. Huge number of preconditions here
| to fulfill.
| e28eta wrote:
| I don't know how prevalent it is as a network architecture, but
| it seems like a bastion host / jump box would be a juicy target
| for this exploit, since it'd let the attacker jump upstream.
| jmalicki wrote:
| Sure, but first they have to root the bastion box.
|
| If you root the bastion box, you have user credentials for
| anything inside the network. Controlling the user's laptop
| seems unlikely to be your most profitable next step.
| yjftsjthsd-h wrote:
| It really depends on the setup; ex. I imagine it's easier
| to steal company code from a laptop than server, while data
| is the other way around.
| gunapologist99 wrote:
| > If you root the bastion box, you have user credentials
| for anything inside the network.
|
| But that's not how a (properly-configured!) bastion host
| works.
|
| You won't have user credentials for anything UNLESS users
| are using Forward Agent (which they shouldn't! simplest
| explanation here.. https://userify.com/docs/jumpbox ).
|
| That's the point behind using ProxyJump. Your connection
| actually jumps THROUGH the bastion box and doesn't stop for
| interception along the way.
|
| (And, of course, an attacker can't do anything very useful
| with ssh public keys except for maybe traffic analysis or
| learning more target IP's.)
| akerl_ wrote:
| Increasingly, the role of a bastion host is served either by
| something like Teleport, which handles authn/z and proxying
| without needing forwarded agents, or newer options in OpenSSH
| like ProxyJump where you hop via a bastion host but without
| ever forwarding your agent.
| wahern wrote:
| musl libc refuses to implement dlclose[1] for precisely the
| reason that modules too often misbehave when dropped at runtime,
| and requiring this behavior is very rarely needed if ever. The
| number of modules that will be loaded is almost always bounded,
| and keeping a module around in memory is mostly harmless;
| certainly less harmful on average than trying to unload it.
|
| [1] see https://wiki.musl-libc.org/functional-differences-from-
| glibc...
| a-dub wrote:
| why publish this before the patches are out?
| tedunangst wrote:
| The patches are out.
| a-dub wrote:
| where? i couldn't find them on the openssh webpage and my
| distro has not released a new openssh either.
|
| edit: nvm. i see the release now. just no release for openbsd
| itself it seems.
| tedunangst wrote:
| http://www.openssh.com/releasenotes.html#9.3p2
| davidcuddeback wrote:
| > _just no release for openbsd itself it seems._
|
| It was posted to announce@openbsd.org a few hours ago.
| Binary patches are available through syspatch and source
| patches on the errata page.
| kneebonian wrote:
| Can I just say I know it's really dumb, but I loved that they
| published the explanation as a simple txt file, instead of
| setting up some whizbang website for it, or embedding it in their
| company blog.
|
| https://www.qualys.com/2023/07/19/cve-2023-38408/rce-openssh...
| tptacek wrote:
| This is the bug of the year.
|
| It's well established that if Alice forwards an SSH agent to Bob,
| Bob can use the SSH agent protocol to make Alice open DLLs,
| because there's an agent protocol command
| (SSH_AGENTC_ADD_SMARTCARD_KEY) that OpenSSH implements with
| dlopen: when you ask the agent to access a smart card, OpenSSH
| dlopen()'s the library corresponding to the `id` of the device.
| This is a Jann Horn bug from 2016, and OpenSSH fixed it by
| whitelisting DLLs to /usr/lib and directories like it.
|
| The Qualys bug builds on Horn's bug. When OpenSSH dlopen()'s the
| library, it then tries to look up a PKCS#11 entry point function,
| and, when it doesn't find it, it dlclose()'s the library and
| returns an error.
|
| The issue is that most of the libraries in system library paths
| were never intended to be opened maliciously, and so they do all
| sorts of stuff in their constructors and destructors (any
| function marked `__attribute__((constructor))` or `destructor` is
| called by dlopen and dlclose respectively). In particular, they
| register callbacks and signal handlers. Most of these libraries
| are never expected to dlclose at all, so they tend not to be
| great about cleaning up. Better still, if you randomly load
| oddball libraries into random programs, some of them crash,
| generating SIGBUS and SIGSEGV.
|
| So you've got a classic UAF situation here: (1) force Alice to
| load a library that registers a SIGBUS handler; it won't be a
| PKCS#11 handler so it'll get immediately dlclose()'d, but won't
| clean up the handler. (2) Load another library, which will take
| over the program text address the signal handler points to. (3)
| Finally, load a library that SIGBUS's. If you manage to get a
| controlled jump swapped into place in step (2), you win.
|
| If you're thinking "it's pretty unlikely you're going to be able
| to line up a controlled jump at exactly the address previously
| registered as a signal handler", you're right, but there's
| another quirk of dlclose() they take advantage of: there's an ELF
| flag, NODELETE, that instructs the linker not to unmap a library
| when it's unloaded, and a bunch of standard libraries set it, so
| you can use those libraries to groom the address space.
|
| Finally, because some runtimes require executable stacks, there
| are standard libraries with an ELF flag that instructs the
| process to make the stack executable. If you load one of these
| libraries, and you have a controlled jump, you can write
| shellcode into the stack like it's 1998.
|
| To figure out the right sequence of steps, they basically
| recapitulated the original ROP gadget research idea: they swept
| all the standard Ubuntu libraries with a fuzzer to find
| combinations of loads that produced controlled jumps (ie, that
| died trying to execute stack addresses).
|
| A working exploit loads a pattern of "smartcards" that looks like
| this (all in /usr/lib):
| syslinux/modules/efi64/gfxboot.c32 (execstack)
| pulse-15.0+dfsg1/modules/module-remap-sink.so (groom)
| x86_64-linux-gnu/libgnatcoll_postgres.so.1 (SIGBUS handler)
| pulse-15.0+dfsg1/modules/module-http-protocol-unix.so (groom)
| x86_64-linux-gnu/sane/libsane-hp.so.1.0.32 (groom)
| libreoffice/program/libindex_data.so (groom)
| x86_64-linux-gnu/gstreamer-1.0/libgstaudiorate.so (groom)
| libreoffice/program/libscriptframe.so (groom)
| x86_64-linux-gnu/libisccc-9.16.15-Ubuntu.so (groom)
| x86_64-linux-gnu/libxkbregistry.so.0.0.0 (groom)
| debug/.build-id/15/c0bee6bcb06fbf381d0e0e6c52f71e1d1bd694.debug
| (SIGBUS)
|
| The paper goes on to classify like 4 more patterns whereby you
| can get unexpected control transfers by dlopen() and immediately
| dlclosing() libraries. The kicker: we noticed
| that one shared library's constructor function (which can
| be invoked by a remote attacker via an ssh-agent forwarding)
| starts a server thread that listens on a TCP port, and we
| discovered a remotely exploitable vulnerability (a heap-
| based buffer overflow) in this server's implementation.
___________________________________________________________________
(page generated 2023-07-19 23:01 UTC)