[HN Gopher] Making an SSH client the hard way
___________________________________________________________________
Making an SSH client the hard way
Author : darthShadow
Score : 138 points
Date : 2022-10-27 17:13 UTC (5 hours ago)
(HTM) web link (tailscale.com)
(TXT) w3m dump (tailscale.com)
| heliophobicdude wrote:
| Hi Mihai! Great work! I would love to see where this goes!
|
| Forgive my ignorance but is there any sort of native client
| besides the browser running in the background to help with
| websocket to tcp? Or a tunnel to a cloud service to help there?
| mihaip wrote:
| No native client is running. The browser makes a WebSocket
| connection to our relay server, and we run the WireGuard tunnel
| over that.
| Spivak wrote:
| > To make this possible, we ported the following to WebAssembly:
| the Tailscale client, WireGuard(r), a complete userspace network
| stack (from gVisor), and an SSH client.
|
| I love that they were clearly inspired by fly.io. Warms my heart
| that a random blog post with a good idea can spread like this.
| bradfitz wrote:
| Which blog post are you referring to?
|
| But yes, we love Fly and use them (and they use us) and we
| share a slack channel between our two companies for casual
| banter.
| vngzs wrote:
| This is really cool and fun, but is this a safe way to run SSH
| clients?
|
| If, say, the adblock Chrome extension you're using gets bought by
| a malware operator and backdoored[0], now it also has SSH and VPN
| access.
|
| [0]: https://www.wired.co.uk/article/fake-chrome-extensions-
| malwa...
| Scaevolus wrote:
| If you have an actively malwared extension, there's probably an
| easier way to exploit any given target-- the simplest being
| recording passwords.
|
| There's not really a "safe" website when the browser is
| malicious.
| gunapologist99 wrote:
| This is turning a remote server into a previously "safe
| website" and expanding the threat model to include remote web
| browser attacks.
| dgentry wrote:
| The Tailscale VPN client, the same one which runs on other
| devices, is compiled to WASM. It handles all of the key
| exchanges to connect to the tailnet. The SSH session is running
| as a WASM Tailscale client.
|
| The browser, opening connections from within the browser
| engine, doesn't have the keys for SSH or VPN access.
| shp0ngle wrote:
| It doesn't have the keys, but it can inject any javascript
| and do whatever the user can do.
| Spivak wrote:
| To me it seems they've taken all precautions they can
| reasonably take -- "what if the user installs a keylogger"
| isn't fixable by anyone.
| gunapologist99 wrote:
| "Installing a keylogger" is a _vast_ oversimplification,
| even if it is outside of their threat model.
|
| Installing almost anything in your browser is usually a
| matter of a couple of clicks.
| Asmod4n wrote:
| And then the addon intercepts the loading of the Wasm code,
| injects it's own payload into it and has access to the keys.
| lxgr wrote:
| What keys? I think the implementation does not use regular
| SSH keys for SSH authentication, but rather something
| custom (I believe traffic to port 22 on each SSH enabled
| client is intercepted and the daemon handles authentication
| itself).
| vngzs wrote:
| Sounds like about "as good as this gets" if you happen to
| want to do this in browsers. Good job.
| ikiris wrote:
| citation needed
| dekhn wrote:
| Most of the products from tailscale just seem to be "look at the
| inner platforms we can build that replace the outer platforms".
|
| Having an SSH client in your browser join your VPN violates all
| the principles of modern computing.
| [deleted]
| xaduha wrote:
| From the article
|
| > Web-based SSH clients aren't new. Nearly every VPS and cloud
| provider already lets you connect to your VMs from the web --
| so how is this different?
|
| This is clearly isn't for everyone, but if you need or already
| use something like that, then I think this has a chance to be
| more secure than some other options.
| dekhn wrote:
| I really liked Chrome SSH extension but now I've returned to
| using the ssh command line client on all three platforms.
|
| The issue is wiring up the browser-hosted application with a
| custom network inside the browser. It's a truly interesting
| but highly disruptive concept and I'm curious how it will
| play out. Perhaps in the future every program will statically
| compile its own TCP stack and talk over RAW sockets, but...
| that's sort of throwing away everything BSD and Linux and
| Windows achieved over the last few decades in terms of OS
| abstractions.
| bradfitz wrote:
| > Having an SSH client in your browser join your VPN violates
| all the principles of modern computing.
|
| I think that's a compliment? You're welcome? :)
| convolvatron wrote:
| absolutely. the lines weren't bad, but they were off in
| places and now they are set in stone for no good reason.
|
| edit: _especially_ when it comes to security
| dekhn wrote:
| It's not a compliment. Or rather, within my understanding of
| how things should be architected, it's not. I certainly
| wouldn't claim that my own beliefs about network architecture
| should trump others, and I work in a different domain from
| most of the people with your use case. Whether the disruptive
| work you're doing is good for the world in the long run is
| still a very open question in my mind.
|
| I used to think that everything in the world should move to
| the browser (in my case, that would be high performance
| molecular graphics and microscope control) and ChromeOS was
| the logical extension. Web Assembly to handle the existing
| C++ codebases, a collection of standard web tech to handle
| the user interface. Big fan of SSH extension because it meant
| that machines that only ran a browser could be useful command
| line programming terminals. But didn't like that SSH
| extension was written in a dead-end container technology
| (NaCl) and the CHrome folks sort of messed up multiple ways
| for the extension to integrate better.
|
| But after working with web tech for enough time I came to
| conclude that putting things in the browser like this is an
| antipattern, in particular increasing the surface complexity
| of the software space while not actually replacing existing
| systems (openssh continues to exist, OS-level VPNs continue
| to exist, even after you port the VPN and client to web
| assembly and run it in a browser).
|
| In my mental model, it makes more sense to put the browser in
| a VM and then wire the VM's networking to a VPN handled by
| the OS, rather than putting what is more or less a
| significant fraction of a virtual machine manager's
| capabilities into the browser. If you're going to do that,
| why not go whole-hog and add a VM to chrome so that it can
| run linux with a full networking stack and then host an SSH
| client inside that? So you can run linux in your chrome in
| your linux.
|
| My compliment is: I am impressed at how well you parlayed
| several technical projects into a thought leadership
| position, but we have fundamentally different architectural
| principles and work in different domains. Your work disrupts
| mine, but mine doesn't disrupt yours. My enterprise actually
| disallows me from visiting your company's website on my work
| computer because users installing their own VPNs is
| considered a security risk (fwiw, I bought into BeyondCorp,
| which eschews VPNs, a long time ago, and would prefer my
| enterprise eliminate VPNs, as they don't really protect our
| users).
| qbasic_forever wrote:
| How does a VM or sandboxing help anything? The whole point
| of wireguard is to get the (shitty) OS VPN out of the
| picture entirely.
|
| Give me a stream of bytes and let the whole world see it
| for all I care--wireguard will build a secure private
| network entirely on that stream of bytes. It could be
| totally public coffee shop wifi with zero encryption
| (basically yelling your passwords and secrets out in the
| open) and yet wireguard will make it secure and private for
| me.
|
| So in this case who cares if its a websocket to the browser
| vs a 'proper' (bloated, shitty) OS VPN. Give me a stream of
| bytes and I'll build my own secure and trusted network on
| it thank you very much.
| dekhn wrote:
| The VM encapsulates the OS component for networking and
| networking virtualization. In my experience, the
| virtualized software stack in the VM host is very
| reliable and predictable, and all the existing OS tools I
| need as a sysadmin (like tshark, ip, and other commands)
| all work just fine for debugging.
|
| I understand the desire for moving more TCP logic to
| applications but, given my experience with network
| technology, I would predict that ten years from now,
| people will hate the experience of having to update 30
| apps to get 1 fix to TCP performance that would have just
| been a kernel upgrade. IE, like everything that happened
| with the web and inner platforms, it's more work, doesn't
| replace the existing system, and just makes the admin's
| life harder for the ostensible purpose of being more
| convenient for the developer on their own machine.
| 0xbadcafebee wrote:
| SSH in the web browser is actually the best practice today.
| Here are some examples of why SSH in your browser actually
| _compliments_ modern computing:
|
| - An SSO-authenticated web interface, integrated with a
| host agent on your instances, means you don't have to
| manage SSH keys.
|
| - If you just need a disposable CLI that inherits
| permissions from your SSO-authenticated user role, you can
| do that from a disposable box in a web interface after
| authenticating via the web interface. Google Cloud Shell is
| a good example.
|
| - Cloud-native development is easier if developers can just
| start working on a unified environment, without having to
| set up & maintain a local environment. Utilizing the web
| browser avoids the need to consider separate tools and
| separate methods of network connection.
|
| - SSH'ing to "private" instances is impossible without
| going through a bastion or VPN. The bastion then becomes a
| single point of attack, and is hard to maintain and secure.
| Similarly the VPN is an additional attack vector,
| maintenance headache, and requires client-side software,
| configuration, troubleshooting. Instead of deploying a
| bunch of bastions or setting up a VPN, if you can use the
| backend control plane through an SSO-authenticated API
| gateway, along with a backend proxy to internal networks,
| you can avoid bastions altogether. This is the best
| practice for Zero-Trust. Google Cloud IAP Proxy is a good
| example.
|
| The implementation of it, with
| WebAssembly/WebSockets/WireGuard/DERP, may be lamentable
| for several reasons. But it probably (I assume?) solves
| problems that other SSH Web Interfaces didn't. I hate that
| the web browser has monopolized computing interfaces :) But
| in this case it seems to solve many problems.
| dekhn wrote:
| I'm fine with ssh in a browser. I used Chrome SSH
| Extension for many years to connect to a VM running tmux.
| And I use RDP if I truly need a remote desktop. However,
| it (browser SSH) not a replacement for, it's an
| augmentation of, the OS-level ssh client.
|
| Turning this around. Let's take the idea of using WASM to
| put a full environment in the user's browser. This is a
| logical idea, after all- WASM exists to make it possible
| to write applications in Not-Javascript and deploy them
| in a browser. IE, don't stop with SSH: you should have a
| web server, a shell, multiprocessing, scripting
| languages, everything necessary to host VSCode server and
| a self-hosted compilation toolchain in a browser. Full
| linux user space in a browser, enough to compile ChromeOS
| and boot into a browser running linux
|
| What have you achieved? A very expensive (in terms of
| porting cost, CPU usage, and deployment size) inner
| platform that does what an OS does already. But it's
| inside the browser, with a patched version of code
| (because WASM always trails native apps), with each sub-
| application maybe linking in its own TCP stack. So it
| will always trail innovations in desktops, since it's not
| a full replacement for the existing system. So it makes
| the world more complicated and exposes more surface areas
| for security management.
| asdfasdfasdfass wrote:
| EvanAnderson wrote:
| Talking about inner platforms (and in case there's one
| person left who hasn't seen it):
| https://www.destroyallsoftware.com/talks/the-birth-and-
| death...
| dekhn wrote:
| https://en.wikipedia.org/wiki/Inner-platform_effect
|
| """The inner-platform effect is the tendency of software
| architects to create a system so customizable as to
| become a replica, and often a poor replica, of the
| software development platform they are using. This is
| generally inefficient and such systems are often
| considered to be examples of an anti-pattern."""
|
| Like I said elsewhere, I'm not completely opposed to the
| idea of the browser as a complete and fully functional
| application container for an inner platform. And I want
| to encourage creative people to try new technologies,
| especially WASM to explore the idea of "how much can we
| move to the browser". However, I see the container as the
| mediator of the network, not the application.
| apenwarr wrote:
| (I'm a tailscale cofounder) I think of Tailscale more like
| a set of tools that lets you do any architecture you want.
| Nobody has to use Tailscale ssh console, but if you believe
| in the future of wasm -> apps -> web console -> ssh, now
| you can have it.
|
| On the other hand, if you believe in the future of OS
| private network connectivity -> console -> ssh, then you
| had that already with native Tailscale and Tailscale ssh.
|
| If you believe in OS private network connectivity ->
| browser -> javascript console -> ssh, then you can do that
| too, by installing tailscale in the native OS and then the
| browser can use it.
|
| I actually agree with you, I'm very suspicious about a
| world where we just move everything into the web browser.
| But on the other hand, sometimes it's really handy to have
| that option.
| dekhn wrote:
| Hi Avery. I think we may have chatted when I worked at
| Google (you can figure out my username pretty easily).
|
| To be honest I can't evaluate your product at work- to
| determine whether it helps our users and whether the idea
| of moving more of the network stack into the application
| makes sense- because my corporation (a large
| multinational pharma) disallows us from visiting the
| entire tailscale website because you sell a VPN
| product(!) which isn't our standard one. I'd love to
| change that policy but I'd still want to move to a
| BeyondCorp world (https w/ auth), not put a VPN in my
| browser. Or make Tailscale our standard VPN.
|
| I see the point of "it's really handy". That's how we got
| Javascript which is a cost we now all have to pay.
| apenwarr wrote:
| "https with auth" is fine and good, and obviously the
| world has been heading in this direction. But I secretly
| suspect this is because 90%+ of developers nowadays don't
| know how to hack on any layer below http.
|
| Tailscale is not a typical VPN; it's just a system that
| attempts to provide beyondcorp-like behaviour at a lower
| level of the stack, so that you don't have to rewrite all
| your apps (ssh in this case!) to use https, and don't
| have to have open ports in your firewall, and don't have
| to run everything through the cloud if you don't want.
|
| As in my post above, there's more than one way to do it.
| You can also build traditional-beyondcorp-over-https on
| top of a Tailscale network, so you get all the improved
| network connectivity and also all the benefits of a
| "pure" beyondcorp architecture.
| easton wrote:
| Could the Tailscale client be packaged as an extension so I can
| visit sites on my Tailnet without having to install a client?
| Sometimes I want to visit a "internal" site without having to
| install the client, if I'm using a temporary box for something.
| I'm not sure how much more work would have to be done, might have
| to dig into the open source pieces of this.
| capdeck wrote:
| Putting my vote in for this feature as well... Also, why not
| compile VNC into WASM and get full remote desktop experience
| for graphical apps. It seems that hard work has already been
| done!
| bradfitz wrote:
| > Also, why not compile VNC into WASM and get full remote
| desktop experience for graphical apps. It seems that hard
| work has already been done!
|
| Yup. :)
|
| In fact, that's mentioned in the original public bug:
| https://github.com/tailscale/tailscale/issues/3157
| Scarbutt wrote:
| But can it run emacs?
| xena wrote:
| Yes
| gunapologist99 wrote:
| This significantly increases the threat model for your remote
| servers to include all sorts of remote attacks through the web,
| including: * garden-variety web attacks (i.e.,
| XSS, CRSF, etc) * attacks that might become viable
| against the browser (for example, Mobile Safari has a history of
| vulnerabilities) * various attacks against the backend
| web server (API attacks) * attacks against the WASM
| layer * CDN injections * Tailscale's
| backend (various types of injections, timing attacks, or deeper
| attacks on Tailscale's infrastructure like the nightmares of
| HeartBleed, Shellshock, Meltdown, etc)
|
| That's probably a very incomplete list.
|
| Realistically, this essentially (actually, literally) _opens a
| remote root shell into your entire infrastructure through a web
| page_ , with apparently nothing more than matching an IP address
| pair (https://news.ycombinator.com/item?id=33361837) to
| authenticate.
|
| What could go wrong?
| amluto wrote:
| I can't shake the feeling that Tailscale's SSH authentication
| mechanism is at the wrong layer of the stack. It appears to work
| by looking at the (source, dest) IP address pair and mapping that
| to a Tailscale identity. But this may mean that any user or
| anyone who can initiate TCP connections from an authenticated
| user's IP can authenticate to the destination over Tailscale SSH.
|
| If Tailscale's client was a userspace construct bound to a
| specific user SSH program, maybe fine. But Tailscale's client is
| a regular VPN client. What happens if you connect to the
| Tailscale VPN, open a malicious but sandboxed app of some sort,
| and that app connects to the target on TCP port 22.
|
| For all that it's a seriously unfinished product, Cloudflare's
| SSH offering seems better thought out. Perhaps Tailscale should
| find a way to issue a short-lived certificate and use that in
| addition?
|
| (It looks like regular sshd could _almost_ be convinced to handle
| this. If the SSH_CONNECTION environment variable were passed to
| the AuthorizedPrincipalsCommand helper or if the source and
| destination were available as '%' tokens, then
| AuthorizedPrincipalsCommand could do the Tailscale tuple lookup
| and use it as a second factor in addition to a short-lived
| certificate (or regular SSH key or whatever). I bet openssh would
| accept a patch for this.)
| [deleted]
| matthewaveryusa wrote:
| A step-up prompt or notification wouldn't be a bad idea to
| approve ssh connections before they are established -- This is
| the same issue with ssh agents. I think there's an ACL setting
| in tailscale where you check ssh connections with a re-auth.
| it's time-based with a minimum of 1 minute though.
| fisian wrote:
| I agree.
|
| I feel like there should be a lightweight way to use SSH
| certificates, so you could use it independently or on top of
| Tailscale. Like a server (CA), client and daemon on your
| machines that should be reachable via SSH that handles short
| lived certs and authentication of clients. But I'm not aware of
| anything like that.
|
| I have used Teleport before but it seemed not that great for
| machines not publicly reachable, because then all the traffic
| goes through a proxy.
| pquerna wrote:
| This is how Okta's Advanced Server Access works:
| https://www.okta.com/products/advanced-server-access/
| amluto wrote:
| Smallstep and cloudflared do this. Sadly, both of them seem
| to use essentially identical client-side hacks. Smallstep is
| a small company that I wouldn't trust with the keys to the
| kingdom, and Cloudflare seems to treat their SSH product as
| something thrown over the fence with nothing resembling
| support.
|
| Gravitational's Teleport seems pretty good, but it's
| heavyweight and doesn't have any pricing appropriate for
| small businesses.
| aleph- wrote:
| I'm mildly curious what client side hacks you're talking
| about?
| mihaip wrote:
| Tailscale SSH's check mode
| (https://tailscale.com/kb/1193/tailscale-ssh/#configure-
| tails...) is meant to address the issue of "rogue process
| starts an SSH connection". For truly sensitive applications,
| you can set the check period to be "1s" to always require it.
| amluto wrote:
| Hmm. If the problem is that Tailscale SSH doesn't strongly
| associate the person authenticating with the connection being
| authenticated, asking the person to reauthenticate seems like
| a pretty weak solution.
| skybrian wrote:
| In the old days, people said you shouldn't write crypto in
| JavaScript because it was somehow insecure. Have those concerns
| gone away with WebAssembly and https everywhere?
| rany_ wrote:
| I don't think the argument was ever that "JavaScript was
| insecure." It was that the websites hosting it may be
| compromised or may change the script at any time without any
| indication (or the FBI forcing a site to backdoor the JS for
| some investigation)
| kevin_thibedeau wrote:
| It will never go away because you can't guarantee constant time
| algorithms will be implemented as such when transformed by a
| JIT.
| lxgr wrote:
| If machine code can issue the necessary hints to the hardware
| to skip all time-variant optimizations, why couldn't the same
| work for a WASM runtime?
| easrng wrote:
| Now that HTTPS is everywhere and crypto.getRandomValues
| provides a secure RNG I think most of the concerns are
| mitigated.
| lxgr wrote:
| Maybe for Javascript, the language.
|
| Definitely not for browsers as the execution environment, at
| least browsers running extensions.
| e12e wrote:
| I think the consensus is still that you can't write side-
| channel/timing proof crypto in (most) Javascript (runtimes) -
| but that with webcrypto(?) most runtimes will provide the
| secure crypto primitives you need in order to do (secure)
| crypto with Javascript?
| chatmasta wrote:
| Cool feature! I was just looking at boringtun last night and
| wondering if it could compile to WASM, to get a virtualized
| network interface in the browser.
|
| Did you experiment with the new WebTransport API [0] at all? It's
| only supported in Chromium browsers, but seems promising for this
| kind of use case.
|
| [0] https://chromestatus.com/feature/4854144902889472
| PaulWaldman wrote:
| > To make this possible, we ported the following to WebAssembly:
| the Tailscale client, WireGuard(r), a complete userspace network
| stack (from gVisor), and an SSH client.
|
| Would it be possible to bundle the same into a portable
| application allowing you to use Tailscale without installing it?
| My understanding is that currently if you can't install Tailscale
| on a client you need to use Subnet Router.
| https://tailscale.com/kb/1109/devices-without-tailscale/
| lxgr wrote:
| You'd have to redirect all network usage (i.e. the sockets API
| or your platform's equivalent) through the custom stack, which
| is possible if you can rebuild the source or by using something
| like LD_PRELOAD for binaries, but can get very tricky in the
| general case.
|
| There's an utility called SSHuttle that does something similar
| for SSH instead of Tailscale/Wireguard: It redirects all
| sockets usage to go through an SSH connection, to allow usage
| of SSH port forwarding without explicit SOCKS support on the
| app's side.
___________________________________________________________________
(page generated 2022-10-27 23:00 UTC)