[HN Gopher] Tailscale SSH
___________________________________________________________________
Tailscale SSH
Author : ignoramous
Score : 470 points
Date : 2022-06-22 15:14 UTC (7 hours ago)
(HTM) web link (tailscale.com)
(TXT) w3m dump (tailscale.com)
| jgeralnik wrote:
| So @bradfitz when are you releasing
| https://tailscale.com/connect/ for real? :)
|
| Context for the uninitiated - as a crazy idea on the podcast
| Security Cryptography Whatever (hosted by tptacek and others less
| well known on HN) Avery and Brad of tailscale imagined an ssh
| client in the browser with QR code authentication to SSO to allow
| you to connect to your tailscale network (over tailscale SSH)
| from untrusted computers such as internet cafes. (Or mostly
| untrusted - safe from keyloggers but maybe not from a dedicated
| active malware that injects into your browser and tries to inject
| secret commands into your ssh session).
|
| I created a silly PoC here (video instead of link because don't
| try it for real)
| https://twitter.com/jgeralnik/status/1487913797155233798 back
| when tailscale ssh was a secret binary in the tailscale github
| repo
| bradfitz wrote:
| Actually pretty soon, probably.
|
| It's working. It needs some UI love first and some docs so
| people know how it works and don't immediately freak out. :)
| l30n4da5 wrote:
| my first thought is that this seems less secure than using a
| private ssh key and locking your machine down to only that ssh
| key.
|
| you're essentially using google as your machine login, which
| seems like weaker security, imo.
|
| edit: I'll caveat this and say, I think Tailscale is fantastic!
| I've been using it personally on my machines for a few months
| now, and it is awesome.
| jasonlotito wrote:
| If I have to use a browser to make use of this (which the demo
| shows), I never want to use it. It's like the abomination of Okta
| and Luminate. Absolutely horrible UX.
|
| Nope. Will fight very hard to avoid ever having to use this.
|
| Antagonistic toward developers at best.
| sandstrom wrote:
| Good! Boundary (https://www.boundaryproject.io/) by Hashicorp
| needs some healthy competition.
|
| Teleport is also a tool in this space, for those looking for
| alternatives.
| sandstrom wrote:
| And for anyone looking at Tailscale, I should also mention
| ZeroTier (https://www.zerotier.com/).
|
| In my opinion they have better tech, but they are pretty bad at
| packaging it, and bad at making it work for actual use-cases.
|
| Tailscale seems to be much more clever around building out
| stuff (like this one, SSH) that actually goes all the way for a
| particular use-case. ZeroTier feels more like a building block,
| where you need to bring more stuff yourself.
|
| Either way, both are awesome pieces of technology, and really
| useful!
| philosopher1234 wrote:
| Why is the tech better?
| bradfitz wrote:
| I'm one of the authors of this. Happy to answer any questions.
|
| One of the fun technical details is that, when enabled on a
| machine (tailscale up --ssh), the userspace tailscaled process
| takes over all TCP port 22 packets after the WireGuard decryption
| and doesn't even feed them into the kernel over TUN. We use
| gVisor's netstack to handle the TCP connections in-process.
|
| So it doesn't matter whether you have other processes (or
| iptables rules, etc) that would prevent the Tailscale SSH server
| from binding to port 22. This lets people gradually use Tailscale
| SSH over time without messing with their system one.
|
| The Tailscale SSH server currently only runs on Linux but there's
| support in git main for macOS too but it's not super well tested
| yet and not included in the sandboxed GUI builds currently.
| lolsoftware wrote:
| This looks great, and I'd love to replace AWS SSM (at least for
| the purposes of instance access) with this! One question I have
| is have is around device limits.
|
| With SSM, I can easily run an agent on every instance.
| Tailscale has pretty tight device limits on the Team and
| Business plans. I have no idea what the custom pricing looks
| like, but I'm guessing it would exceed my budget. What's the
| intended way to use this with a large number of servers? A
| small team can easily have more devices than 5x or 10x the
| number of users. Should we just set up some "gateway"/"bastion"
| instances to access via Tailscale SSH and then use regular ssh
| from there? Some sort of more limited device mode that doesn't
| count against the device limit (for ssh only, perhaps?) would
| be great.
| bradfitz wrote:
| You could do a Tailscale SSH bastion thing, yeah. But before
| you build a funky setup to avoid pricing concerns, at least
| reach out to the sales folk to see what it is. We're usually
| pretty flexible on exact quotas and realize that different
| orgs have different user/device shapes.
| atonse wrote:
| What would we need to have open in our security groups for this
| to work?
|
| I think ingress wouldn't be necessary since tailscaled creates
| a tunnel right?
|
| But how about egress traffic? UDP for WireGuard or something
| else?
| bradfitz wrote:
| Security groups where? On the Tailscale ACL side, you need to
| allow tcp/22 in.
|
| On your host where you're running Tailscale, usually nothing.
| You can keep everything locked down for ingress. Outbound UDP
| only, but usually cloud VMs allow outbound traffic already.
| (This is covered more in
| https://tailscale.com/kb/1082/firewall-ports/)
| atonse wrote:
| I meant AWS, that page explains it all, thanks!
| e12e wrote:
| > (...) the userspace tailscaled process takes over all TCP
| port 22 packets after the WireGuard decryption and doesn't even
| feed them into the kernel over TUN. We use gVisor's netstack to
| handle the TCP connections in-process.
|
| > So it doesn't matter whether you have other processes (or
| iptables rules, etc) that would prevent the Tailscale SSH
| server from binding to port 22.
|
| This sounds like a great feature when exploiting buggy
| WordPress/php apps! /s
|
| I realize this is a feature - but it's a bit sad that the
| standard package handling isn't up to the task; leaving (I
| expect) the tailscale daemon as a "magic" netcitizen - not
| featuring in neither "ss" or "iptables" output (why can't I
| login to opensshd?).
| tptacek wrote:
| How do you figure? The idea is that Tailscale is bypassing
| the kernel, which it can only do for requests coming in over
| the tailnet --- it gets those packets raw, directly from
| WireGuard, unlike the normal IP packets your kernel routes
| to/from localhost or an egress interface.
| e12e wrote:
| > [only for..] requests coming in over the tailnet
|
| Well, that's certainly different from "all TCP port 22
| packets" - I suppose some emphasis should be on "after the
| WireGuard decryption" (ie: over the wireguard interface).
| It's not entirely clear from the comment (but probably
| clear to engineers working on the tailscale code).
|
| I read it as if tailscale snapped up packets before the
| kernel from (all) network interfaces...
| tptacek wrote:
| How could it possibly work otherwise? Tailscale owns the
| WireGuard connection, so it gets raw packets from
| WireGuard _before the kernel_.
| password4321 wrote:
| It's not the same, but "Docker Network bypasses Firewall,
| no option to disable"
| https://github.com/moby/moby/issues/22054 (2016)
| tptacek wrote:
| You're right, it's not at all the same. The Tailscale
| bypass exists (1) only for traffic traversing Tailscale
| interfaces (by design, that's the _only_ traffic it can
| impact, because Tailscale can 't run a userland TCP/IP
| stack for non-Tailscale traffic), and (2) only for this
| one feature, and (3) only if you've explicitly allowed it
| for particular users in your Tailscale ACLs. It's not
| clear to me how you could screw it up to, e.g., amplify
| an SSRF attack.
| password4321 wrote:
| All I'm trying to point out is that advertising "this
| bypasses the firewall, by design" has been abused in the
| past.
|
| [edit] It boils down to principle of least surprise,
| managing expectations, etc. - proper documentation is
| indeed key.
| bbarnett wrote:
| This sounds like more of a marketing thing, where the
| wording needs to employ care?
| tptacek wrote:
| You have to explicitly enable Tailscale SSH, both on the
| host and in the ACLs that allow users to use the feature.
| Tailscale's ACLs are much, much better than iptable rules
| (for instance: they have built-in unit testing).
|
| (I'm not impartial about Tailscale.)
| zachallaun wrote:
| Just want to say thanks: This is insanely cool/easy. Combined
| with the VSCode Remote SSH extension and MagicDNS, it's now
| insanely easy to work on a project in a remote environment. I
| was recently reading through a relatively long post on setting
| up SSH through Tailscale to access a WSL2 environment, and now
| it's literally as easy as popping open VSCode in any
| environment that I have Tailscale installed on and accessing
| `user@my-magic-dns-machine`. Great work!
| structural wrote:
| I've been using Tailscale for years but will likely not use
| this feature, even though I would like to.
|
| The fundamental problem with the approach really is that
| connections are different over the tailnet and over the local
| network. Here is a specific use case that is painful:
|
| 1. There exists a cluster of machines, each with large amounts
| of locally attached storage. They are all on the same local
| network and connected with 10Gb (and likely soon 40Gb ethernet
| interfaces).
|
| 2. Each machine is individually on the same tailnet so they can
| be accessed remotely.
|
| 3. Remote users frequently need to move large amounts of data
| between machines. A user copying a few hundred gigabytes of
| data with "scp" is normal.
|
| 4. For performance reasons, it's preferred to avoid the
| Tailscale/wireguard overhead when copying data between adjacent
| machines in a rack.
|
| At this point, if I enable tailscale ssh for remote login, it
| appears that the problem of key management for connections
| between local machines (using ssh over the normal interface,
| not the tailnet) still remains, and in fact, the overall
| authentication configuration is more complex than it was
| before.
|
| What I would _love_ to exist, and would make me instantly use
| this feature, is if the tailnet issued SSH certificates
| (probably injected into its own ssh-agent?), the existing
| tailscale SSH implemention worked just like it currently does
| (it 's great!), AND I could manually configure servers to
| accept certificates issued by the tailnet. Then SSH paths like
| "laptop --> (over tailnet) --> server 1 --> (over local
| network) --> server 2" could be made to work transparently, for
| those machines that need it, and for regular users, it still
| "just works".
| bhawks wrote:
| Any interest in adding mosh like features (https://mosh.org/)?
|
| Low latency typing, session resumption etc
| infocollector wrote:
| Hi Brad: Thanks for helping out with this feature! I've been
| one of the early users of tailscale. My network is around 50
| machines. I've recently started having issues with ssh on some
| of my machines, especially from mac m1 -> some ubuntu boxes.
| Could this be related to this new feature? Any
| suggestions/pointers on how to debug these issues?
| bradfitz wrote:
| Tailscale SSH doesn't mess with your port 22 packets if it's
| off so almost certainly unrelated. Have you reached out to
| support or filed a bug?
| aidos wrote:
| Just a quick thank you to the team working on Tailscale. It's
| hands down the most seamless dev experience I've ever seen.
| Every time I think "such and such would be nice", I search the
| docs, and it's already implemented better than I could have
| expected (eg the stateless mode for ephemeral servers).
|
| I tinkered with cloudflare before that but just couldn't get on
| with the interface of the admin tooling.
|
| With Tailscale I have a lot more confidence that I've set up
| the access rules as I need them. It's all just a lot more
| obvious.
| dstaley wrote:
| Could you share some details about the embedded SSH server? I'm
| curious if this would work to add SSH capabilities to devices
| that run Tailscale but don't include a built-in SSH server.
| Previously I've used dropbear, so it'd be really nice to be
| able to drop that requirement!
| bradfitz wrote:
| If you're already running recent-ish Tailscale on them,
| they're already running an SSH server that's just disabled.
| Run "tailscale up --ssh" to turn it on.
|
| The code's at
| https://github.com/tailscale/tailscale/tree/main/ssh/tailssh
| for all the details. Which details in particular are you
| curious about?
| dstaley wrote:
| Ah it's using crypto/ssh to provide the SSH server, so I
| think that's all the details I needed! Can't wait to give
| this a shot.
| ludsan wrote:
| quick question. Does this do user (de)provisioning like
| Jumpcloud? I.e. if the target machine doesn't have a
| /home/someuser but someuser is in my tailnet ACL, will it
| create the account?
| mfsch wrote:
| From [1]: "Like other SSH clients, Tailscale will only use
| user accounts that already exist on the host, not create new
| accounts."
|
| [1]: https://tailscale.com/kb/1193/tailscale-ssh/#ensure-
| tailscal...
| mike_d wrote:
| How was the decision made to roll this functionality out before
| announcing it to customers (we found it during a previous
| security audit)?
|
| While it might seem logical in your mind to bolt on extra
| features and add value, your customers evaluate risk based on
| functionality of the software they are approving. Customer buys
| a VPN solution, magically gets remote access that bypasses
| firewalls. Can we trust Tailscale to not roll out a remote file
| backup feature and start silently exfiltrating data (as an
| extreme example)?
| bradfitz wrote:
| There are two things have have to be enabled to turn it on:
|
| (1) a target server needs to run "tailscale up --ssh" to
| enable the SSH server
|
| (2) your Tailscale ACLs have to permit it. Our default, if
| you've never set your ACLs (as is usually the case for
| personal users), is that you're allowed to SSH to your own
| untagged devices only.
|
| For an org that's already using ACLs, you won't have any SSH
| rules defined and thus nobody in your org can enable the SSH
| server. (Or rather, they can enable it but nobody can connect
| to it.)
|
| If your concern an org that's using the default "all packets
| are allowed" ACLs?
| delano wrote:
| That's one part of it.
|
| I can't speak for mike_d specifically, but there is a
| concern with having (potentially significant) modifications
| made to the codebase that aren't surfaced in the release
| notes. I imagine closed-source projects do this on a
| regular basis whether customers know (or care) or not.
|
| The expectations for opensource projects are different
| though, particularly when it comes to system-level or near
| system-level components. So not being able to access the
| functionality is a great default but it doesn't address
| side effects of the changes or the desire to know about
| changes being made in our environments.
| kortilla wrote:
| Concern more around what looks like an ssh backdoor showing
| up unannounced. How would they know the subtleties of what
| it takes to enable it when it wasn't announced yet?
| dimitar wrote:
| Can you do ssh tunnelling?
| maisem wrote:
| You can do local port forwarding today, remote port forward
| is still a WIP. What do you want to use it for?
|
| Disclaimer: I am one of the engineers who built Tailscale
| SSH.
| dimitar wrote:
| Accessing Clojure nREPL servers, local port forwarding
| works for that. This allows me to inspect and modify
| running applications, a Lisp superpower.
| YarickR2 wrote:
| So we have no way to secure this besides disabling wireguard ?
| cassianoleal wrote:
| There's no "disabling wireguard" in Tailscale unless you
| don't run it at all.
|
| You can secure this by:
|
| - Not enabling the SSH feature on hosts where it's not needed
| - Creating ACLs so only certain clients are allowed access.
|
| So essentially, just use the same mechanisms as for
| everything else in Tailscale.
| jdadj wrote:
| This is neat. I've used Cloudflare's Zero-Trust SSH, but I've
| been frustrated that it interacts poorly with sftp and scp
| because of the client-side changes that they make to
| ~/.ssh/config
|
| Does tailscale have the same issue?
| bradfitz wrote:
| We don't modify or require changes to your SSH client. You
| can use any SSH client you want.
| xena wrote:
| Tailscale employee here. Tailscale SSH works at the target
| side by listening on the SSH port on that machine. Client
| changes aren't needed for this to work, all that is required
| is to use your SSH client as normal. This should allow you to
| use sftp and scp without issues.
| jgeralnik wrote:
| For what it's worth I encountered the same issue and came up
| with a solution:
|
| https://github.com/cloudflare/cloudflared/issues/574
|
| Cloudflare have ignored the github issue (which includes a
| solution) but at least 3 other people seem to have found my
| solution helpful.
| CaptainJustin wrote:
| > This lets people gradually use Tailscale SSH over time
| without messing with their system one.
|
| That is something I have really appreciated about Tailscale. It
| seems to consistently not mess with the existing environment.
| Considering it does networking witchcraft and it works on a
| variety of architectures and OSs this is quite an
| accomplishment.
|
| I suspect Tailscale's customers have found the same.
| tucif wrote:
| A nice next step would be tailscale managing an ssh key that's
| allowed to do interact with a git(hub) repository. So that I
| wouldn't have to create multiple keys or setup the same key on
| different machines and still be able to interact with a repo
| from all of them.
|
| It'd be really nice just using git transparently and having
| tailscale take over the git ssh connection and authenticate
| using taliscale access controls.
|
| At least for personal projects or small teams that'd be quite
| convenient.
| stormbrew wrote:
| Depending on which part of those things you find painful, you
| might want to look into ssh certificates? They're pretty easy
| to work with, much easier than most kinds of certificate
| systems.
| JayCuthrell wrote:
| Thank you to the Tailscale team for altering my belief that VPN
| could only stand for Vexing Productivity Neutralizer.
|
| Features like Tailscale SSH represent the ruthless removal of
| annoyances.
|
| edit: grammar
| Syzygies wrote:
| I have Tailscale on all my Macs. I use MacOS default SSH
| between my machines, but only via the Tailscale interface.
|
| Nevertheless, I had to open SSH on each machine, and it's a
| nightmare to close up the firewall so only Tailscale gets
| through. You'd think this was the whole point of Tailscale;
| there should be a one click lock to restrict to Tailscale. But
| the Tailscale documentation is wanting. I actually paid for a
| candidate for the best firewall front end, it came with "Let us
| know if we can help!" and radio silence once I explained the
| problem. Likely, restricting to Tailscale requires a
| granularity one can only hand-code.
|
| I can write a firewall, I've written plenty in the past, I just
| couldn't find the several hours to do this as a one-off for me
| when it should be easy, but I was missing needed information.
|
| Tailscale is justly proud of how it connects machines through
| uncooperative routers and such. Tailscale SSH should do the
| same. The idiot's guide to securing a machine so only Tailscale
| SSH gets through should be to find SSH in the preferences and
| turn the fucker off.
| justinsaccount wrote:
| You don't really need a firewall to do that. putting
| ListenAddress 100.x.x.x
|
| where 100.x.x.x is the address on the tailnet, into your
| sshd_config would do what you want. Unfortunately you can't
| specify an interface, but if you have any sort of automation
| in place this is easy enough to template in.
| 867-5309 wrote:
| hello, is sftp supported? thanks
| bradfitz wrote:
| Yes: https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/
| tail...
| newfonewhodis wrote:
| What happens if I use Tailscale SSH and Google (or whatever IDP)
| decides to ban my account? Is there a break-glass or something
| that would let me either change IDPs or re-enable openssh-based
| access without losing my servers?
| leohonexus wrote:
| Totally, I'm a happy customer, but completely relying on
| Tailscale for out-of-band SSH doesn't sit right with me,
| especially for personal machines. One ban and poof, you can't
| access your own servers.
|
| And to be honest, I understand the appeal of not having to muck
| around with the system, but SSH isn't that cumbersome once
| you've set up your /etc/hosts, and encrypt your id_rsa files.
| Couldn't get any easier than `ssh <machine>`.
| atonse wrote:
| Another question, can this be used to create SSO-enabled SFTP?
| Isn't SFTP just ftp over SSH?
| dave_universetf wrote:
| SFTP is a sub-protocol of SSH (technically a "subsystem" in the
| RFC-speak), which implements features similar to "legacy" FTP.
|
| Anyway, our ssh server knows about sftp, so `sftp <host>`
| should just work.
| mfincham wrote:
| SFTP isn't really FTP over SSH, it's just a collision of
| naming.
| rollcat wrote:
| > Isn't SFTP just ftp over SSH?
|
| FTP is a wholly different protocol, that has aged rather
| poorly. You might be thinking of FTPS, which is FTP with TLS.
| SFTP is its own thing and is actually decent/sane.
|
| > can this be used to create SSO-enabled SFTP?
|
| From another reply somewhere else in the comments, apparently
| yes.
| RL_Quine wrote:
| I'm not entirely convinced I want a feature that adds even more
| exposure to the sort of goofy login flow Tailscale has.
| api wrote:
| Totally meta to this discussion: I am disturbed by the SSO/IAM
| trend because it gives root on the entire universe to a small
| collection of companies.
|
| We are looking at a future where a security breach or
| misbehavior by one of a handful of companies could mass-
| compromise millions of businesses and critical infrastructure
| and possibly hundreds of millions to billions of devices. Even
| worse this permission is clandestine. It could be exercised
| against individual targets with no obvious audit trail, since
| auditing tends to also be delegated to the IAM provider (and
| nobody looks at local logs in most cases).
|
| I guess we've been handing vendors a lot of power for a while
| with OS vendors that have "push" software update capability,
| but this is adding not only even more carte blanche permission
| to a small number of companies but extending it across systems
| running different OSes and even open source platforms like
| Linux and BSD.
|
| Software update capability is also fairly coarse grained. Apple
| or Microsoft could push a compromised or malicious update, but
| it would be harder for them to specifically target a single
| user reliably and without being noticed. (Wait... why did I get
| a macOS update and none of my friends did?) SSO/IAM providers
| could easily do this. Imagine a subpoena that targets your
| SSO/IAM provider that gives the government (and maybe not even
| your government!) silent unlimited remote access to everything
| you have.
|
| I can also imagine "cancellation" scenarios where a company
| doesn't like what you say so they dump your account and lock
| you out of all your infrastructure, requiring you to manually
| go around and "root" all your stuff. (If you think this would
| only ever be deployed against Nazis, study history a bit.
| Political winds shift.)
|
| It's just a monstrous amount of power to give out, and I feel
| like people aren't thinking this through or maybe are not even
| aware of the power they are delegating.
|
| I feel like people should at least understand what they are
| doing when they choose to delegate all their authentication to
| Google.
| jsmith45 wrote:
| There is a reason why many companies (especially those with
| few developers, artists etc, so most people run Windows), go
| with Azure AD as their SSO solution. Then they are only
| depending on Microsoft not being compromised. But they are
| already relying on other parts of Microsoft not being
| compromised, so it feels like less of a risk to those
| companies.
| stavrianos wrote:
| I've seen this conversation before, but I've never been clear
| on what exactly the consequences of the SSO are. I imagined,
| it might be that the provider gets an IP address when you
| connect or something. You're saying they potentially get
| _access as you_? Am I understanding that correctly?
| api wrote:
| Anything authenticated with SSO can be accessed by the SSO
| provider since they're able to approve any authorization,
| which means they can just log into all your stuff.
|
| So e.g. if you use "log in with Google" on a web site,
| Google now has access to your account too (if they behaved
| badly or were compromised).
|
| Spreading SSO auth everywhere gives the SSO provider login
| access to absolutely everything you have.
| risho wrote:
| wait so if i authenticate tailscale using google and
| enable tailscale ssh's google can just log into any of my
| tailscale ssh servers?
| api wrote:
| I have not tried Tailscale SSH or looked at it deeply,
| but as a general rule the answer is yes if the system is
| using delegated SSO _alone_ to authenticate. (What I don
| 't know is whether TS SSH supports any secondary methods
| like a password or SSH auth forwarding.)
|
| You are delegating authentication, so your delegated
| authenticator can authenticate anything they want.
|
| I feel like a large number of people adopting SSO/IAM
| systems don't fully understand this. If they do
| understand and are making a cost/benefit based choice to
| do this that's one thing, but... I think people should
| understand.
| dsr_ wrote:
| This is correct.
|
| One answer is to have many, many SSO/IDP systems -- and for
| anyone technical enough to set up a homelab to be able to be
| their own IDP.
| api wrote:
| A lot of people don't let you bring your own IDP. They
| offer a few choices: Google, Microsoft, Okta, etc.
|
| I can foresee this eventually being a revenue stream for a
| lot of companies where they charge payola to be listed as
| an IAM provider, sort of like the browser CA inclusion or
| browser default search engine list rackets.
| [deleted]
| sirtaj wrote:
| I'd love to know of well supported, secure IDP software to
| use for this. I'm afraid of OpenLDAP due to its long
| history of security issues. What are the open source
| alternatives that are both minimal in configuration and
| solid enough to be exposed to the internet if necessary?
| ArchOversight wrote:
| Keycloak is Java based, but its simple to setup and
| configure and requires no OpenLDAP.
|
| It supports both SAML and OpenID Connect/OAuth2. With an
| LDAP backend you can also use that LDAP backend for other
| services that don't support those two protocols for SSO,
| but it is not required.
| vorpalhex wrote:
| Authelia is the fast minimal solution.
|
| Keycloak offers a much more "roll your own" design.
|
| https://www.authelia.com/
| dTal wrote:
| >Wait... why did I get a macOS update and none of my friends
| did?
|
| That's an easy one to circumvent if you don't need to install
| the malware Right Now: just wait until the next update cycle
| and slipstream your targeted malware in with it.
| [deleted]
| RL_Quine wrote:
| > I feel like people should at least understand what they are
| doing when they choose to delegate all their authentication
| to Google.
|
| I think this is my biggest concern, it's really scary to have
| Google Auth as literally the only barrier between no access
| and complete production access. I understand that a lot of
| the time Google accounts hold the literal keys to the kingdom
| anyway (customer data, internal company data, maybe source
| trees), but SSH was one of the last frontiers remaining.
| a-dub wrote:
| it may also be a relic of my old age, but the idea that a
| browser cookie could be all you need for production root
| has terrified me in sso environments of past.
|
| maybe i have an antiquated view of browser security but it
| seemed... unnerving.
| tlrobinson wrote:
| > goofy login flow
|
| Can you be more specific about your complaints?
| RL_Quine wrote:
| I edited the original to contain more detail as I posted it
| but it seems to have been lost somehow. The login flow for
| Tailscale is weird due to the need to accommodate things like
| a headless server being added, when combined with their use
| of SSO as the only method of authentication things get
| confused very easily.
|
| When I add a new server I get given a URL that looks like
| https://login.tailscale.com/a/c44a243b to visit in a browser
| and authenticate the new device, the meaning of which is
| quickly lost as soon as you go through a Google Authenticator
| sign in flow, fill out some recaptchas and find your phone
| for a SMS token, and then the device is added to your account
| with no further clicks (unless you enable device
| authorization). It feels very weak, the connection between
| logging in and performing an action is fuzzy.
|
| Due to the use of Google SSO it just has the usual problems
| that you get. It's not quite clear when you're logged in or
| not or with which of the 12 google accounts you own, it's not
| clear what will pop 2FA requests or login prompts. As a
| service tailscale has made it clear that they don't want to
| be an "identity provider", which means you're sort of stuck
| with something that doesn't feel like you can make
| authoritative decisions about how it acts.
| mdeeks wrote:
| SSO is not the only method of authenticating things. They
| have auth keys specifically for the purpose of authing
| headless servers. e.g. sudo tailscale up --authkey tskey-
| abcdef1432341818
|
| You can also apply an ACL tag to it so that it is no longer
| authorized as the user and instead takes on the permissions
| of the tag.
|
| In our deployments we have the headless servers pull the
| tagged auth key from secrets manager on boot and then just
| `tailscale up --authkey <value>`.
|
| I agree the default login flow is usually not what you want
| for headless servers. It sort of leads you down the wrong
| path.
| jvanderbot wrote:
| This is great -- I wish it was more plain in the admin UI
| that this is the better headless workflow. That seems
| like an easy fix!
| mike_d wrote:
| > which of the 12 google accounts you own
|
| Stop logging in to your personal accounts on your work
| machine.
| tptacek wrote:
| It's helpful for people to know, from context later in the
| thread, that one of the core concerns behind this comment is
| the idea of using SSO at all, and thus giving "the keys to the
| kingdom" to Google.
|
| Of course, it's also worth knowing that SSO is basically a
| universal best-practice for security teams, and while it's not
| _de jure_ required by SOC2, it 's almost _de facto_ required.
| For once, I think the best-practices and compliance people have
| this one right: you are extraordinarily unlikely to get burnt
| for trusting Google in this instance, and the security track
| record of ad-hoc authentication is worse than abysmal (ad-hoc
| authentication is probably implicated in a plurality of all
| major incidents).
| mdeeks wrote:
| Probably the main concern around Google is for personal
| accounts where it sounds like people often find no way of
| recovering their accounts when Google thinks you did
| something bad.
|
| For business users that isn't as much the case since there
| are good support options and it isn't likely you'll get
| locked out.
|
| Great for businesses, potentially risky for individuals.
| tptacek wrote:
| This seems entirely fair to me. I mean, if you're scared of
| Google, you can use something like Okta. If it's me
| choosing, it's not a hard decision: I want centralized
| authentication, so I can quickly do blanket interventions
| like enabling/disabling apps, requiring phishing-proof MFA
| without having to do any implementation work, and linking
| everything to onboarding/offboarding/access review
| policies. And if I'm going to trust anyone's security team
| with this, it's going to be Google's.
| mike_d wrote:
| > it's also worth knowing that SSO is basically a universal
| best-practice for security teams
|
| It is also best practice that some things explicitly don't
| sit behind SSO for defense in depth. If for example you leave
| CrowdStrike and JAMF behind Okta, and Okta gets popped an
| attacker can disable your endpoint protection and push
| ransomware to every machine.
| infogulch wrote:
| I'm very interested in Tailscale for both personal and business
| use-cases, but I'm rather put off by the stark centralization of
| offered identity providers: Microsoft, Github (Microsoft),
| Google, okta (?). What are the chances that Tailscale would offer
| authentication using decentralized/self-hosted identity providers
| like Ory ( https://www.ory.sh/ )?
| doublepg23 wrote:
| There is also the self-hostable Headscale implementation.
| sandstrom wrote:
| In case anyone is looking for the URL:
| https://github.com/juanfont/headscale
|
| 5k stars on Github, and lots of activity. Seems very
| interesting!
| jsmith45 wrote:
| While not official, Tailscale themselves are not opposed to
| this project.
|
| They don't much contribute to it directly, but they have
| gone out of their way on a few occasions to avoid breaking
| it or to make it easier for headscale to implement some
| things (like the new encryption scheme for communicating
| with the control server).
|
| I imagine they don't see it as much of a business model
| threat, since it has no commercial support, requires having
| somebody your organization run and administer it, it is
| single tenant, not multi-tenant, so not really suitable for
| AWS to take and use to outcompete Tailscale proper, etc.
|
| The people most likely to use this are the ones too
| concerned about security to use the official (Tailscale-
| hosted hosted) control plane, or people/orgs that simply
| cannot reasonably afford Tailscale's pricing model. In
| either case, they were not really customers in the first
| place.
| stormbrew wrote:
| Note before anyone gets too excited: doesn't work on iOS, and
| you have to sideload an app on Android, if you want to use it
| from a phone.
|
| These aren't problems for everyone but I think they should be
| front and center when suggesting it.
| mdeeks wrote:
| They offer custom SSO providers using SAML or OIDC
| https://tailscale.com/kb/1119/sso-saml-oidc/
|
| Unfortunately they are locked behind Enterprise pricing because
| of the extra help and debugging needed to get them working.
| Maybe at some point this will be offered standard though.
| [deleted]
| mfsch wrote:
| Looks interesting, but it seems that this doesn't work well for
| servers where every user has a personal account. It appears that
| this use case would require a separate ACL entry for every user,
| which a) can get slightly annoying to manage and b) requires a
| business plan. It would be nice if something like `"users":
| ["autogroup:emailuser"]` was supported to allow
| `alice@example.com` to connect as the user `alice`, but that
| would probably cause issues with e.g. Github organizations, where
| email addresses can have different TLDs.
| midislack wrote:
| What's Tailscale? Some kind of VPN?
| mountainriver wrote:
| Tailscale is my absolute dream networking solution, I would go as
| far as to say it will ultimately change how we develop
| applications in the future
| birdyrooster wrote:
| If the auth flow was as good as Touch ID and no window
| switching, yeah it would be acceptable but this flow would give
| me a headache with all the flashing.
| dave_universetf wrote:
| TouchID and related are on the list. Hooking into the
| existing auth flow was the easiest to get this out the door
| (and more desirable for some companies who want the audit
| event in their SSO stack, arguably), vs. figuring out when to
| nudge people to enroll physical factors and so forth. But I
| definitely also want "tap your security key please" as an
| option :)
| quartzic wrote:
| But you still can't have multiple tailnets. The strategy of "have
| hobbyists try out the software themselves, like it, then
| implement it at their work" seems incompatible with this fact.
| titanomachy wrote:
| Do you use the same Google/Github/Microsoft/whatever account
| for both work and personal stuff?
| enneff wrote:
| A lot of people do just use one account for everything. Many
| smaller companies don't bother giving people corporate
| accounts.
| structural wrote:
| It's more than just a work/personal split. Even at work,
| having "development" and "production" tailnets so that things
| like testing complex ACLs, inhouse apps that use tailscale
| via its API, etc. are possible without having everyone on the
| devops team create an unmanaged/non-company email so they can
| create their own development tailnet, and then deploy a bunch
| of company IP using this rogue account.
|
| It's a pain point.
| mdeeks wrote:
| Agreed this is a big limitation.
|
| The only way to do it is if you have secondary email address
| domains. Say mdeeks@company.com and mdeeks@company.team. You
| can create a separate tailnet for company.team but you also
| have to roll out additional subnet routers (if you use them)
| that are authed on that second tailnet. Also you wont be able
| to easily write rules that interact with things that are not
| authed onto the second tailnet.
|
| They need a first class concept of "canary" or "beta" that
| applies to ACLs, DNS configs, client versions, and all sorts of
| other toggles in the UI. It's a hard product problem and I'm
| not even sure how some of it should work.
|
| I just know I need a way to test changes before I roll it out
| to everyone at the company. Right now there aren't good options
| for that.
| aspyct wrote:
| I started using tailscale a few days ago, and I absolutely love
| it.
|
| However, one thing is still nagging me: technically, they can add
| devices to my network without telling me, right? Or is there
| something I'm missing?
| dimitar wrote:
| How Tailscale different from other VPN solutions?
| matthewmacleod wrote:
| They have a relatively detailed series of comparisons with
| other solutions in their documentation:
| https://tailscale.com/kb/comparisons/vpns/
| edf13 wrote:
| Never login as root... even over secured links!
| alerighi wrote:
| Why? To me it doesn't make sense not to do so. What kind of
| security gives you logging it as non root user and then using
| sudo (maybe even without a password) to become root?
|
| Root makes everything more easier, for example to copy a file
| on a server with scp if you don't have root login you first has
| to copy it to /tmp and the copy it in the right directory by
| logging in as normal user and elevating with su/sudo.
|
| Also you have to remember which username was chosen when the
| server was installed, was it user, admin, or pi, or some
| default?
|
| I get not using root for everyday usage on a desktop, but for a
| server having a non root user is not that useful.
|
| Of course you have to know what you are doing, but sudo doesn't
| protect you from stupid mistakes anyway (especially if
| configured with NOPASSWD as I always see doing because having
| to continuously type the password is annoying and password
| tends to be forgotten)
| kevin_nisbet wrote:
| It probably depends a bit on your security posture and how
| you manage the server. The recommendation for avoiding root
| tends to be central to the idea of least privilege. By
| default, grant/allow the least privileges necessary, helps
| eliminate alot of security surface area, while also being a
| layer of protection against mistakes. It's easy to be
| careless with a superuser account, and have a bad day.
|
| And this is naturally in conflict with being productive. As
| you mentioned, it's easier to be productive if you can just
| do all of the things all of the time. And operating in
| environments where a mistake may not be that devastating, or
| compromise or vulnerability this might be a reasonable
| tradeoff.
|
| But I've worked in environments where this is too risky. For
| example: 1. Engineer accidentally pastes the wrong buffer
| into a terminal. They had accidentally copied some other
| piece of text. 2. The text happens to contain \nhostname
| set\n. 3. The terminal as it's spitting out errors, does see
| a valid command, to change the hostname. 4. That particular
| system was an HA system, and the process monitor in use,
| grepping the running processes for the command + args, of
| which hostname was an argument. And decided the process was
| no longer running. 5. The cluster seeing the failure, decides
| to boot another process. But at this time in history, that
| process was could only handle a single instance. Both
| instances now decided to conflict with each other. 6. Some
| part I don't remember about the failover site. 7. A million
| cell phones can no longer get an IP address.
|
| So it's a question of tradeoffs, but is a generally
| recommended practice to not login directly to root, and
| operate with less privileges when not required. And then
| escalate if granted / required.
| structural wrote:
| This is spot on. One interesting approach I've seen before
| is that all commands executed as the superuser must be
| written in file, and the only command accessible via sudo
| is "please_save_to_audit_log_then_run_it_in_sandboxed_env
| <file>". For particularly high-risk situations, there might
| be a second person reading your script before running an
| approval command that actually lets the script run. Things
| don't move quickly, but the number of mistakes via typo is
| certainly reduced.
| [deleted]
| 8organicbits wrote:
| I think in this context, I'd worry about attributing any
| actions taken on the root account back to a named user. I
| suppose tailscale keeps an auth log (anyone know details?) so
| you likely could determine "root"="alice" by looking across
| different log files. Attributing privileged commands to an
| employee is critical for security.
|
| In other contexts you want to avoid shared root accounts, as
| you'd want to block access for former employees, but you don't
| want to rotate credentials every time. SSO for tailscale makes
| that easier.
| e12e wrote:
| Indeed, "root" is not a person - only persons should have
| authorization (to log in, to elevate via su/sudo).
|
| Ed: although in this case the binding between a system user and
| a person happens at the tailscale level.
| megous wrote:
| "bob" UNIX account is not a person either.
|
| If you ssh in, no matter to what account, your key ID is
| logged and that's what matters.
|
| Anything can happen afterwards, unless you have a really
| tight grip on your system, since local privilege escallations
| are not _that_ hard or uncommon.
| lfmunoz4 wrote:
| alex_dev wrote:
| I've been having trouble adopting Tailscale. As so many others
| say, relying on another identity provider is unfortunate - I,
| too, worry what happens when Google decides to lock me out
| because some algorithm decided my account is fishy.
|
| The biggest blocker has been the issues with the Android client.
| I'm either hitting
| https://github.com/tailscale/tailscale/issues/915 or
| https://github.com/tailscale/tailscale/issues/4611, but neither
| issue appears to have a fix coming soon. Whenever I am on my
| carrier's network, my phone's internet stops until I disable
| Tailscale - that's just a show stopper from using TailScale.
|
| So instead of developing this SSH feature, I would have preferred
| to seen them work on their bug backlog.
|
| In the meantime, I'm experimenting with ZeroTier. While it
| doesn't have the ease and cool magicDNS+LetsEncrypt feature, I
| think I'll survive with something more reliable.
| ratg13 wrote:
| Is anyone using tailscale on an organizational level?
|
| I'm curious to hear about some of the use cases, and whether some
| companies and organizations are attempting to adopt this instead
| of traditional VPN.
| mdeeks wrote:
| We just adopted it to consolidate multiple different OpenVPN
| installations.
|
| Why?
|
| * The Tailscale clients are dead simple and good quality (but
| not perfect). OpenVPN clients for mac and iOS are pretty bad.
| Onboarding OpenVPN users was a large document that generated a
| lot of questions and support issues. Tailscale onboarding is
| about two minutes for most users and we had nearly no support
| requests rolling it out widely to our company.
|
| * Tying OpenVPN to Okta is a truly terrible experience. Users
| would login with their Okta creds and a push would silently go
| to their devices. If they didn't know to check their phone it
| would just fail to login. Alternatively you can paste your TOTP
| code after your password. Yes, really.
|
| * We don't have to manage or debug anything related to LDAP.
|
| * Maintenance on our side is extremely minimal. Just install
| subnet routers (<10 lines of bash) and put our ACLs in source
| control.
|
| * We no longer have to tell users to logout and login to
| another VPN to get to certain resources. We just grant them
| access and suddenly they can reach what they need. ACLs are
| amazing and super easy to script, audit, and test.
|
| * Split DNS that actually works on all operating systems. For
| private domain A, query this resolver (over the wireguard
| link), for private domain B, query this other resolver.
|
| * I rolled it out as a PoC to all of our major VPCs in _a day_.
|
| The bad? It's still a young product and is missing features and
| has some warts.
|
| * Notifications on macOS that you need to relogin are just
| plain broken (they know and are working on it).
|
| * We're currently battling issues with network resets due to
| what looks like a client bug when you have lots of users.
|
| * No access to audit logs yet
|
| * You can't restrict people from using exit nodes
|
| * No good way to canary changes to your user population. Any
| mistake in the UI instantly breaks everyone.
| yebyen wrote:
| > Users would login with their Okta creds and a push would
| silently go to their devices. If they didn't know to check
| their phone it would just fail to login. Alternatively you
| can paste your TOTP code after your password. Yes, really.
|
| This sounds exactly like my Cisco (anyconnect) VPN experience
| from a previous job/life, both before and after Okta was
| introduced... we think it don't be like it is, but it do.
| sconi wrote:
| curious what 'dead simple' means re: clients. Do your users
| still need to login like openvpn, or is it always on?
| mdeeks wrote:
| It's a small icon in the top bar on macOS. You click login,
| it opens your browser, you Google/Okta auth in your browser
| using any factor you want (push, totp, yubikey), and you're
| done. Login literally takes seconds and there is little
| chance for confusion.
| jaywalk wrote:
| > Users would login with their Okta creds and a push would
| silently go to their devices. If they didn't know to check
| their phone it would just fail to login.
|
| How would users not know to check their phone? They had to
| specifically set up this MFA method.
| mdeeks wrote:
| Because they didn't specifically set it up. That is just
| how Okta MFA over LDAP works: https://help.okta.com/en-
| us/Content/Topics/Directory/LDAP-in...
|
| Also people just forget. Some people may only need the VPN
| once per month and in that time they forget about this
| weird login flow. They just assume they typed their
| password wrong or that they lost permissions to the VPN or
| something.
| RL_Quine wrote:
| I would have loved this at a company a couple of years ago
| which was massively all in on Google Auth for literally
| everything. If you're fine with that being your concrete level
| of authentication for everything; internal tools, external
| tools, etc, then tailscale sort of just slots right in, and SSH
| makes it even more so.
|
| I would be very hesitant to build around this personally,
| hijacking Google accounts is already something pretty high
| value to a companies adversary, and using Tailscale and SSH
| like this turns it from a compromise of accounts indirectly
| through password resets into access to unlimited production
| machines, internal services, etc. It feels almost layer
| violating to have a soft social login through Google, that gets
| persisted in every Chrome browser and logged into on every
| employees phones also directly control SSH, but maybe that's
| just me.
| zikduruqe wrote:
| Then host your own Auth server -
| https://github.com/juanfont/headscale
| soSadm4n wrote:
| One of my clients, an industrial/commercial property realtor
| (to contextualize the environment; we're not talking military
| secrets here), uses it.
|
| Day to day I interact with it like any other VPN client except
| I auth via the Google workspace account they gave me.
|
| It's Tailscale, or hosted OpenVPN and cross your fingers
| they're not snooping, or DIY Wireguard or OpenVPN and all the
| usual ups and downs of DIY.
|
| Software based infra is out of the unknown unknowns era these
| days and years of rising usability expectations means Oracle
| level nightmares to deal with do not gain enough momentum to
| survive anymore. Tailscale is plenty easy to deal with. The
| only consideration is do you believe your traffic is really
| secure? Otherwise "it just works" like anything else these
| days.
|
| That said, my project for them is deprecating the infra
| accessed via Tailscale (24/7 EC2 running web dashboards). The
| already Dockerized dashboards will run locally now and use an
| API to retrieve the data. Real people directly in your infra is
| probably best avoided.
| freeplay wrote:
| > It's Tailscale, or hosted OpenVPN and cross your fingers
| they're not snooping...
|
| The "cross your fingers they're not snooping" applies to
| Tailscale as well.
| soSadm4n wrote:
| That's what I meant.
| Sytten wrote:
| This seems like the perfect complement to replace the SSM Agent /
| bastion instance currently used to access AWS VPC (it is super
| clunky to use). This should allow an easier time to do reverse
| tunnelling to databases without having to manage SSH keys.
| dimitar wrote:
| Hmm AFAIK you don't need a bastion to use SSM agent - it even
| allows you access through the browser. I think you meant EC2
| Instance Connect which manages temporary SSH keys.
| andymac4182 wrote:
| That feature was recently added to SSM
| https://aws.amazon.com/about-aws/whats-new/2022/05/aws-syste...
|
| Using something like gossm which I just put a PR in for this
| feature also makes this easier
| https://github.com/gjbae1212/gossm/pull/54
| ryanisnan wrote:
| I'm curious, what's really clunky about SSM?
|
| Other than ensuring the pre-requisites are met, and knowing the
| instance-id, SSM works pretty flawlessly. You can easily write
| a wrapper that looks up the instance-id from the hostname, if
| you prefer to use it that way.
| fladrif wrote:
| Haha I'm not sure if you were being serious, but the workflow
| you just outlined is the clunky part of SSM. The pre-
| requisites are getting all the IAM roles and permissions
| setup (no mean feat), installing the agent, configuring it
| with keys generated by another user, and getting the
| connection information back from the aws console. This
| promises to be a lot easier to setup and authenticate,
| install tailscale, login.
| NoraCodes wrote:
| I know this opinion comes up every time Tailscale is mentioned,
| but requiring SSO _and_ only supporting companies like Google and
| Microsoft on the free tier means a lot of people can't use it
| without being exposed to a ton of risk in the form of automated
| moderation/deletion decisions. I want to be excited about this
| stuff, but it just won't fit into my risk profile until that
| changes.
|
| Hell, I'd be happy to pay $5/mo or whatever if that meant I could
| roll my own SSO, or even just use a cheap-per-user, low-volume
| provider.
| skybrian wrote:
| This is a workaround, but could you minimize that risk by
| signing up for both Microsoft and Google? It should help for
| any policy / moderation / "computer says no" decisions that are
| vendor-specific.
|
| (On the other hand, for security risks, it means that a
| security hole in either one would be a problem.)
| cstejerean wrote:
| I can see some risk with say Google as the surface area is much
| larger (maybe they don't like some play store activity or some
| ads activity or YouTube activity, etc). But I use my GitHub
| account and I'm really not worried about automated moderation
| locking me out of my account.
| pphysch wrote:
| > (SSH certificates are better, but have you tried running your
| own enterprise CA?)
|
| For a small business, what is so hard about keeping a file (CA
| private key) secure and changing it when required?
| lloeki wrote:
| > For a small business, what is so hard about keeping a file
| (CA private key) secure and changing it when required?
|
| For a small business? Well, keeping a file secure and changing
| it when required ^^'
|
| I mean, it's not out of this world hard to generate your
| private CA but there are a thousand footguns, the experience
| isn't exactly friendly, and it's Yet Another Thing To Do And
| Keep Track Off, i.e even if there's someone who has the
| technical chops, they may not have the bandwidth, and also,
| lottery factor. Let alone keeping it properly secure. There's a
| whole framework/procedure to create to set that up properly.
|
| (been there done that, I was exactly in the situation above)
| pphysch wrote:
| Can you be more specific on some of those main footguns?
|
| I need to rotate the CA for some rare reason. Boom, I do it.
| All the old SSH certs are invalidated, but users can get a
| new one through the usual automated flow.
| loriverkutya wrote:
| Changing it when it is required.
| danousna wrote:
| What would be the advantages of this compared to say Teleport ?
|
| Teleport is working fine for us, but I wonder if the network
| based approach (+ wireguard) of Tailscale would be better in
| terms of network redundancy ?
| swozey wrote:
| Well, how long did it take you to set up Teleport?
| tptacek wrote:
| It took us about an hour.
| swozey wrote:
| That's shocking considering my experience of weeks, nice.
| tptacek wrote:
| We thought it was going to be a whole huge project and
| budgeted a week for it. It was not a whole huge project.
| tptacek wrote:
| The big thing you get with Teleport that you don't yet get with
| Tailscale --- apart from entirely owning the source of truth
| for SSH authentication on your own infra, which is a very minor
| issue for almost everyone but is a major issue for some people
| --- is that Teleport gives you transcript-level audit logs of
| your SSH sessions.
|
| Teleport also has that web-based SSH console (it's one of the
| better web-based consoles) and the ability to do joint SSH
| connections. But the audit log is the big one.
|
| Obviously, the flip side of this is that Tailscale's SSH is
| built in; if you're already using Tailscale, and you're not
| already using Teleport, you should enable Tailscale's SSH right
| away; it is hugely better than managing your own SSH service
| ad-hoc.
| dave_universetf wrote:
| Session recording's actually already in the network engine
| for SSH, we just haven't plumbed the whole "push recordings
| somewhere and surface them" yet. Soon :)
| tptacek wrote:
| It's an extremely valuable feature, in that it can knock
| out a bunch of different SOC2 DRL line items with a single
| screenshot.
| outworlder wrote:
| > is that Teleport gives you transcript-level audit logs of
| your SSH sessions
|
| That is extremely valuable. Just in case 'transcript-level
| audit' didn't sink in, it's a session recording - not only
| you can see the all keystrokes typed but you can see all the
| outputs, the whole state. Someone doing a TOP command for an
| hour? You can watch the same thing later.
|
| Think asciinema (https://asciinema.org/).
| bradfitz wrote:
| FWIW, Tailscale SSH can also record sessions in asciinema
| cast format:
|
| https://github.com/tailscale/tailscale/blob/v1.26.1/ssh/tai
| l...
|
| We haven't yet fully "productized" it yet because it only
| records on-device for now. We want to make it stream
| recordings to another device (that you run) first before
| considering it done.
| alexk wrote:
| Sasha, CTO@ Teleport here. Thank you for the kind words!
| And congrats to the Tailscale team on launching SSH
| product.
|
| Let me share a bit more about our auditing capabilities:
|
| Teleport captures session PTY output and stores it in S3 or
| any S3 compatible storage for your records by default.
|
| If you would like to get additional, more in-depth insight
| into the session, Teleport captures syscalls, file access
| calls and network calls done during SSH session by
| correlating it with sessions' cgroup using our BPF module:
|
| https://goteleport.com/docs/server-access/guides/bpf-
| session...
|
| Teleport provides a lot of other in-depth SSH integration
| for auditing and compliance, for example we support
| moderated sessions access control with a required session
| moderator, or per session-MFA.
| riobard wrote:
| I'll have to ask this since it's bothering me for quite a
| while...
|
| If I connect to a server via WireGuard, would it make more sense
| to run simpler & unencrypted `rsh` instead of `ssh`? It's kinda
| pointless to double encrypt.
| bradfitz wrote:
| Yeah, but e.g. no rsh (or telnet!) on macOS.
|
| It's likewise a bit silly that we had to add TLS support to
| Tailscale: https://tailscale.com/blog/tls-certs/
|
| But we want to interoperate well with the clients people
| already have (browsers, their system ssh client, etc...)
| throwaway894345 wrote:
| Is there an option to avoid double encryption on systems that
| do have e.g. rsh?
| dangerlibrary wrote:
| I might be misunderstanding the question but ... just use
| rsh?
| throwaway894345 wrote:
| I think for the same reasons I wouldn't "just use ssh"
| over tailscale--I don't want to have to manage an sshd
| that doesn't require key or password auth but listens
| over tailscale (and nothing else!). Basically, what I
| want is for tailscaled to _be_ my rshd (appropriately
| configured for connections over tailscale network only,
| etc) or in other words to avoid double-encryption (it 's
| not the end of the world, but ideally we don't need to
| doubly-encrypt).
| structural wrote:
| Double (or more) encryption ends up happening a lot in
| larger networks not for technical reasons but for policy
| ones.
|
| This is unsurprising, because it is used for different
| purposes in different layers of the stack. It is not at
| all a black and white state of "encrypted" vs. "not
| encrypted".
|
| For example, in one organiztion I've worked with,
| Wireguard (generally, including Tailscale) is approved
| for restricting connections only to authorized network
| devices/users and that data maintains integrity in
| transit, but is not approved for protecting the
| confidentiality of sensitive information. Connections
| which access specific resources are required to be
| encrypted at the application level using a mechanism
| which has been approved for that information type (given
| a specific threat model).
|
| So you could transmit very small amounts of data over
| TCP/IP, over a Tailscale network, using a set of pre-
| shared, one-time pads. And you might actually want to do
| this! It's really not ridiculous, but you do need to
| assess whether you really do have a threat model that
| needs it.
| raggi wrote:
| Around the time I joined Tailscale, actually just before,
| I had a look at rsh with an eye in this direction.
|
| The problem is that rsh is very stale and unmaintained -
| even those versions that have had releases in recent
| years (e.g. GNU Inetutils) are very old inside - even if
| they've kept up with patches, they have not kept up with
| features e.g. modern user session construction.
|
| It also turns out that ssh the client, much more so than
| ssh the protocol, is really a key integration point and
| API that users end up needing. It has a broad feature set
| that turns up in use cases all over, many of which rsh
| does not handle.
| pehtis wrote:
| You can still use telnet v1.9.4 on macOS. Just copy it from
| an old version of OSX (pre High Sierra). It still works fine
| on Monterey.
| oefrha wrote:
| From the blog post on TLS support:
|
| > However, if your service doesn't have a valid TLS
| certificate, despite the fact that your connection is
| encrypted using Tailscale, your browser will warn you that
| the connection is not secure (it's doing the right thing--it
| doesn't know about Tailscale!). So, to avoid confusing your
| users, you might want to provision a TLS certificate to
| validate your internal services.
|
| Browser warnings and user confusion aren't the only
| consequence of not using HTTPS. The more concrete impact is
| that you lose access to a large and growing number of web
| APIs that are restricted to secure contexts.
|
| https://developer.mozilla.org/en-
| US/docs/Web/Security/Secure...
| bradfitz wrote:
| Yup! In fact, that was the very first sentence of the
| original GitHub bug about TLS certs:
| https://github.com/tailscale/tailscale/issues/1235 ...
| "Many new web APIs (eg: geolocation, sensors, http/2, etc)
| require a TLS certificate"
| runjake wrote:
| FWIW for those reading (I figure Brad already knows):
| echo "alias telnet=nc -v" >> ~/.zshrc && source ~/.zshrc
| apenwarr wrote:
| Alas, the real "telnet" protocol has considerably more
| fanciness than nc. It's just that the telnet cli command
| degrades into a simple line-oriented mode if it doesn't see
| the telnetd init sequence.
| runjake wrote:
| True, but it handles 99% of the use cases of the people
| who lament the demise of telnet in macOS. :-)
|
| For the rest: brew install telnet
| Brian_K_White wrote:
| macports please
| runjake wrote:
| Here you go: sudo port install
| inetutils
| Brian_K_White wrote:
| Better. Thank you :)
| jackthetab wrote:
| I read that in Chris Rock's voice. :-)
| ithkuil wrote:
| For example window size negotiation
| Spooky23 wrote:
| Not really. Who knows what you're connecting to once you
| connect to the tailscale endpoint.
|
| It's more likely that you're gonna screw up and end up doing
| something you don't intend to do for very little gain. SSH
| overhead in 2022 is really low.
| Helitico wrote:
| Stay secure by default.
|
| CPU overhead for encryption is basically non existend.
| dsr_ wrote:
| Yes and no. You shouldn't have rsh on your system at all --
| there's a case for telnet to test connections (though netcat is
| better), but there's no case for rsh.
|
| ssh used to allow setting cipher=none, but that's not available
| anymore.
|
| Think of it this way: you're paying the small overhead of
| double encryption, but you're gaining not fatfingering your way
| to a password compromise.
| throwaway894345 wrote:
| I'm not following. How does double encryption help to avoid a
| password compromise if everything is authed with tailscale in
| the first place?
| xgbi wrote:
| Somebody listening on local connections can sniff your
| password.
| throwaway894345 wrote:
| How does that work? First of all, I would expect that
| rsh-over-tailscale doesn't use password authentication,
| and secondly can a local user sniff traffic before it
| hits tailscaled?
| 8organicbits wrote:
| I suspect they mean that if you have rsh installed for
| tailscale you need to be very careful with how you run it.
| If you accidentally let rsh listen on 0.0.0.0 and don't
| firewall it then you've given attackers a way to guess
| passwords.
|
| Forgetting to firewall services or accidentally exposing
| services to the internet is pretty common. ssh is more
| hardened than rsh, especially with key based auth, so the
| risk is lower.
| throwaway894345 wrote:
| Isn't this true of SSH as well? What's stopping someone
| from letting sshd listen on 0.0.0.0 with password auth?
| Anyway, I would expect that whether you're using ssh or
| rsh/telnet/etc that you're not configuring it for
| password auth, but rather using tailscale authentication.
| Specifically, I would expect (perhaps naively) that
| tailscaled handles the ssh/telnet/rsh/whatever
| connections rather than passing them off to another
| process (sshd, telnetd, etc) and thus allowing it to
| handle the authentication and daemon configuration (e.g.,
| what address it listens on).
| ben174 wrote:
| Double encryption is twice as effective. I use double ROT-13
| for double the security.
| [deleted]
| efunneko wrote:
| I find it more efficient to just use ROT-26
| Shared404 wrote:
| ROT-26? Hasn't that been broken for a few years now?
|
| We all need to be shifting to ROT-52 ASAP.
| runjake wrote:
| > would it make more sense to run simpler & unencrypted `rsh`
| instead of `ssh`?
|
| No, because ssh has evolved to be so much more than "rsh with
| encryption".
| soraminazuki wrote:
| In this case, double encryption is a good idea though.
| Tailscale is a great way to reduce exposure of your
| infrastructure from the public internet, but it's not without
| flaws. In theory, it should be possible for Tailscale and your
| SSO provider to add new nodes to your Tailnet. Though I don't
| believe this is something that they're actually willing to do,
| it's definitely something to keep in mind if you're planning on
| delegating SSH/sudo authentication to Tailscale.
| jgeralnik wrote:
| Double encryption doesn't actually help in that case though -
| if tailscale (maliciously) added nodes to your network the
| ssh session being encrypted wouldn't change the fact that
| they can run commands on your machines. And if they wanted to
| actively MITM you they could do so (by redirecting your
| wireguard connection to a server owned by them) even with
| encryption (presuming they can fake the host key, which they
| could do at worst by running code on your server to steal
| it).
|
| To be clear I implicitly and explicitly trust tailscale not
| to tamper with my networks and if your threat model includes
| tailscale becoming a bad actor you should remember that in
| that case running their binary in the first place could
| already be game over.
| fragmede wrote:
| Is it pointless? SSH doesn't send the password out over the
| wire but instead uses a challenge-response cryptographic system
| so even if one of the interim machines is compromised, they
| don't have access to the clear text password. You shouldn't be
| raising passwords (or even passwords in the first place with
| ssh) but practice defense in depth.
|
| Unless you're transferring large files the overhead of double
| encryption on ssh is totally blown away by waiting for human
| input.
|
| IIRC There's a fork of SSH that supports not encrypting things
| if you _are_ trying to transfer large files.
___________________________________________________________________
(page generated 2022-06-22 23:00 UTC)