[HN Gopher] CVE-2022-41924 - tailscaled can be used to remotely ...
___________________________________________________________________
CVE-2022-41924 - tailscaled can be used to remotely execute code on
Windows
Author : ghuntley
Score : 392 points
Date : 2022-11-21 18:17 UTC (4 hours ago)
(HTM) web link (emily.id.au)
(TXT) w3m dump (emily.id.au)
| ev1 wrote:
| The biggest shock to me here is "aarch64 Windows doesn't have
| calc.exe"
| ferbivore wrote:
| Releasing a patch and a detailed write-up on the same day seems
| like a bit of an unfortunate choice, especially for a WTF!!
| vulnerability like this. In software that doesn't auto-update, no
| less...
| ThePhysicist wrote:
| Malicious actors will monitor patches and reverse-engineer them
| anyway, so probably better to make some noise in this case and
| make sure people update as fast as possible.
| meibo wrote:
| Especially as the fixes seemingly have been going into their
| public GitHub branch for days, since the report. I wonder if
| that was a conscious choice or negligience, maybe I'm missing
| something? I would expect these to be released as
| patches/merged in when the vulnerability is published, like a
| lot of other security-critical open source software does it.
| tptacek wrote:
| The researcher released the writeup. The researcher doesn't
| work for Tailscale. The researcher doesn't release the patch.
| jenscow wrote:
| > Coordinated Disclosure time proposed by Tailscale,
| accepted by us, Tailscale shares planned Security Bulletins
| and blog post
| [deleted]
| jeroenhd wrote:
| Looking at the timeline, it looks like Tailscale opted to allow
| for public release on the day of the patch:
|
| > Sat 19 November: Coordinated Disclosure time proposed by
| Tailscale, accepted by us, Tailscale shares planned Security
| Bulletins and blog post > Tue 22 November, 5:06AM: Blog draft
| shared with Tailscale (a bit last minute, sorry!!!) > Tue 22
| November, 7:00AM: Coordinated disclosure time
|
| Because the code is open source anyway, I'm guessing they
| assume attackers would see the announcement of a vulnerability,
| browse the recent pull requests and figure it all out
| themselves anyway. Delaying publishing of the details saves
| maybe a few days of exposure to risk for motivated attackers,
| especially as the author seems to have done her work together
| with one other person in just over a week.
|
| They've also sent out emails it seems, so people know they
| should update ASAP and why. With the extremely limited amount
| of people running Tailscale (and the even smaller subgroup
| running it on Windows specifically) I don't think it's an
| attack hackers will rush to roll out. Mitigations also exist
| (i.e. block access from the browser to 100.100.100.100) so even
| in situations where you cannot update you can protect yourself.
| ruuda wrote:
| > If you visit my website, I am granted the honour and the
| privilege of executing arbitrary Javascript on your computer. > >
| This is a pretty bad idea
|
| This is why I disable javascript by default, but I suspect that
| on this page it's needed to fix the theme or something, because
| the text is light grey on a white background, and all monospace
| sections are completely illegible.
|
| Edit: I don't mean to hate on the author, the content of the
| article is really interesting!
| bheadmaster wrote:
| > I suspect that on this page it's needed to fix the theme or
| something, because the text is light grey on a white
| background, and all monospace sections are completely
| illegible.
|
| You seem to be correct. I found a single <script> tag in the
| source, with the following code: (() => {
| let v = localStorage.getItem("color-scheme"), a
| = window.matchMedia("(prefers-color-scheme: dark)").matches,
| cl = document.documentElement.classList,
| setColorScheme = v => (!v || v === "auto" ? a : v === "dark") ?
| cl.add("dark") : cl.remove("dark");
| setColorScheme(v); window.setColorScheme = v => {
| setColorScheme(v); localStorage.setItem("color-
| scheme", v) }; })();
|
| Though I don't see what's the point of this since the "light"
| theme, as you've pointed out, is completely illegible.
| edf13 wrote:
| That's a selective quote...
|
| > This is a pretty bad idea, but luckily even the web browser
| has its limits.
| [deleted]
| say_it_as_it_is wrote:
| I've read the description several times and find it hard to
| follow:
|
| ..an attacker-controlled website visited by the node..rebinds DNS
| for the peer API to an attacker-controlled DNS server making peer
| API requests in the client, including accessing the node's
| Tailscale environment variables
| jakedata wrote:
| The client app is not indicating that 1.32.3 for Windows is
| available yet but the download link on the site has been updated.
|
| Tailscale client downloads are extremely slow at the moment, so I
| suggest you distribute one copy manually around your tailnet
| rather than bogging down their servers even more.
| bradfitz wrote:
| If you restart the windows GUI it should refresh the cache and
| show the update is available. Otherwise it can take some hours.
| mayakacz wrote:
| Tailscalar here.
|
| The Windows client caches the current version for a while, so
| may not yet have v1.32.3 available on your device. In that
| case, you can still pull the latest release from
| http://pkgs.tailscale.com/stable.
| [deleted]
| jakedata wrote:
| Tailscale admin here, politely requesting client update push
| capability. Being able to see endpoint version is helpful, I
| will be suspending unpatched endpoints in the near future.
| more_corn wrote:
| Seconded
| adam_arthur wrote:
| Does this mean we won't get spammed with tailscale articles every
| day now?
| afhammad wrote:
| This means you will see more of Tailscale.
|
| Vulnerabilities are inevitable, the actions taken in the hours
| (ideally) and days following the discovery is what matters
| most.
| beanjuiceII wrote:
| i would still rather see less considering where tailscale
| software sits in my privacy/security. at some point i'd ask
| why do i pay to use this swiss cheese ? (not saying thats the
| case, but if I were to continue to see more issues)
| 0xbadcafebee wrote:
| I really appreciate the Superfluous GraphViz.
| fulafel wrote:
| This kind of graph is also known as an attack tree. Agreed,
| it's good visualization of this.
|
| https://en.wikipedia.org/wiki/Attack_tree
| Komodai wrote:
| nix23 wrote:
| Was just a matter of time...and much more will come.
| jalino23 wrote:
| what do you mean?
| snake_plissken wrote:
| The concept of DNS rebinding and DNS records pointing to a
| private/localhost IP address is particularly interesting and I
| remember when I first came across it in the wild. It's not
| exactly re-binding in the classic attack sense described in the
| article: some US sportsbooks make you download a geolocation
| service that verifies your location in order to place bets. The
| sportsbook's front end communicates with it through a DNS record
| pointing back to 127.0.0.1, and opens up a WebSocket to talk to
| the service. I imagine the WebSocket is used to bypass the same-
| origin policy but perhaps someone more knowledgeable can speak to
| that.
| amluto wrote:
| I don't see a writeup of how this was fixed. Merely checking the
| Host header is insufficient -- the vulnerability would still be
| wide open to anyone who can open TCP sockets to localhost.
|
| Windows has APIs (named pipes, DCOM (eww) and such) that allow
| authenticated local access to services. Unixes have unix sockets.
| dang wrote:
| (This comment was in response to the original submission
| https://tailscale.com/security-bulletins/#ts-2022-004, which
| we've since changed)
| Arnavion wrote:
| Windows from W10 onwards has Unix sockets too.
| monocasa wrote:
| The windows implementation lacks facilities like SCM_RIGHTS
| though to ask the kernel who's on the other side.
| JJJollyjim wrote:
| [co-author of the research here]
|
| They actually approximate this functionality in the Windows
| implementation: It checks netstat to enforce that incoming
| TCP connections are from the expected Windows user! https:/
| /github.com/tailscale/tailscale/blob/2a991a3541ae5d56...
|
| That's why we were happy with the solution they implemented
| as a stopgap, until they could switch to named pipes (which
| there is now an open PR for).
| monocasa wrote:
| Huh, ok, that's not so bad then.
|
| It feels like there could still be a TOCTOU issue there,
| but it'd be difficult to use.
| amluto wrote:
| Generally speaking, allowing privileged operations
| because a specific user asked over a TCP socket is asking
| for trouble: there are quite a few ways that unwitting
| processes could open a socket on behalf of an attacker
| without realizing that it is asserting its identity and
| thus granting privilege.
|
| All the major cloud get this IMO entirely wrong with
| their services that issue secrets to instances (e.g. AWS
| IDMS).
| Arnavion wrote:
| Yes, but their existing TCP implementation wouldn't have
| been doing any auth either. So presumably they don't need
| it.
|
| (I don't know anything about Tailscale so I'm just going on
| first principles.)
| kccqzy wrote:
| Didn't the article say they use netstat to do some
| checks?
| Arnavion wrote:
| Ah yes, the new link says that. The old link didn't that
| detail.
| woodrow wrote:
| Looks like it might be this?
| https://github.com/tailscale/tailscale/commit/976e88d430e0c5...
| bradfitz wrote:
| There were a number of changes:
| https://github.com/tailscale/tailscale/commits/v1.32.3
| meibo wrote:
| Do they have enough logs to reach out to people that were
| affected? As far as vulnerabilities go, this set is one is one of
| the worst ones I've seen this decade, and they seem rather
| straightforward.
|
| Would be nice to get a blog post from them that goes a bit into
| impact, not just a report that tells you to update. It's nice
| that they responded quickly, but I feel like this shouldn't have
| happened in the first place for a network security company and it
| makes the Windows client feel like a bit of an afterthought.
| Looks like they have a PR open to switch it to named pipes, I
| hope that is properly reviewed by someone that knows Windows APIs
| before it's merged.
| ripley12 wrote:
| Yes. I got a (concise, well-written) email this morning with
| the following:
|
| > Am I affected?
|
| > Yes. Your tailnet has at least one Windows node running a
| version of Tailscale prior to v1.32.3.
| meibo wrote:
| I received this email as well, I probably should have
| clarified to say that it would be interesting to know if any
| of this was ever actively exploited. I assume this hasn't
| happened, considering the sentence in their report, but this
| is a client vulnerability, so logs may not have reached their
| servers(I know nothing about their telemetry setup or what is
| actually logged, which is why I mentioned that a blog post
| about their part of the procedure might have been nice).
| imachine1980_ wrote:
| 1) they probably don't get notify 2) they also have
| interest to say no if isn't being exploited publicly even
| if they are searching cases internally. In any case the
| response feels solid most companies will try force update
| whit vulnerability a and not disclose what the problem was
| and then blame the public for not have the software update,
| this giveme security about the compaby, it's also
| problematic because it drains the the time of the host to
| update their systems.
| [deleted]
| arkadiyt wrote:
| edit: I stand corrected as pointed out by the replies below.
| Curious what logs they had to prove this!
|
| Original comment:
|
| > Do they have enough logs to reach out to people that were
| affected?
|
| It happens on the client, there are no server logs that
| Tailscale could check
| mlyle wrote:
| "Reviewing all logs confirms this vulnerability was not
| triggered or exploited."
| [deleted]
| bradfitz wrote:
| Reconfiguring the local client daemon (tailscaled) is
| reported to Tailscale log servers so there's server-side
| evidence if it's exploited.
| cmeacham98 wrote:
| > Curious what logs they had to prove this!
|
| My guess is the client sends some kind of "goodbye" message
| when it gets reconfigured to another coordination server, and
| that message has enough information to determine if it
| originated from this attack.
| mcoliver wrote:
| "Reviewing all logs confirms this vulnerability was not
| triggered or exploited."
| Thaxll wrote:
| "worst ones I've seen this decade"
|
| It's not that bad actually.
| semi-extrinsic wrote:
| Super interesting article, and TIL Firefox does not implement PNA
| (Private Network Access).
|
| Does anyone know why? It seems like an obviously good thing to
| have.
|
| https://wicg.github.io/private-network-access/
| klabb3 wrote:
| Related: note also that tailscale's tailnet 100.* subnet is som
| form of CGNAT public ip block. I think Tailscale thought long
| and hard about this, and landed on it because it was a path of
| lesser resistance to break fewer things. And if you squint they
| fit the stated purpose.
|
| But even if browsers now implement PNA the tailnet itself is
| public address space, so that vector still exists. I wonder if
| browsers (and eventually standards) will be pressured to treat
| those blocks as private.
| api wrote:
| The real takeaway here is that you should never treat any
| network boundary as critical for security. This is true
| whether it's a physical boundary or a virtual one (with TS
| being one example of the latter).
|
| If your private net is full of trivial to access things with
| no access control or horribly insecure services, that's a
| huge problem. There are many many many ways to hop over
| firewalls. Hostile JS on web sites is just one.
|
| Network boundaries are only first lines of defense in what
| should be a defense in depth strategy. Never depend on any
| one single boundary completely.
|
| My personal criteria is: if it's not secure enough to be
| connected directly to the Internet with no firewall, it's
| broken. Make it that secure _and then_ put it on a secure
| network.
| angry_octet wrote:
| The usual incompetence. It would break some existing use etc.
| ale42 wrote:
| Any reference about this? (i.e. somebody saying that it would
| break something?) Curious to see how they'd justify it.
| cesarb wrote:
| > If you run non-HTTPS web services on your Tailnet, and those
| services are unauthenticated or rely on Tailscale for
| authentication, implement an allowlist of expected HTTP Host
| headers to prevent malicious Javascript from accessing these
| services.
|
| In my opinion, this should be done not only for non-HTTPS
| services, but for all services: the "default" virtual host (used
| where there is no Host header, or when it has an unexpected
| value) should have nothing except a static 4xx error page. This
| not only avoids DNS rebinding attacks, but also avoids automated
| attacks in which the attacker doesn't know the correct hostname
| for the service (mostly automated scans for vulnerable PHP
| scripts and similar).
| judge2020 wrote:
| In addition, it's basically required if you're using a reverse
| proxy service, eg. Cloudflare or Akamai. CF websites have been
| found via Shodan or Censys because site info is left open via
| https ://[ip]:443.
| loo wrote:
| Heads up that I had to update through the UI twice - first
| brought me from 1.30.x 1.32.2, then second to 1.32.3.
| tailspin2019 wrote:
| > The speed and quality of Tailscale's response to our report is
| unlike any vendor interaction I have experienced, and suggests a
| deep commitment to keeping their customers safe.
|
| I have mixed feelings here as a Tailscale customer.
|
| Yes a quick response is great, but this actual security issue is
| _pretty terrible_ IMHO.
|
| Anything other than an immediate response would have been akin to
| lighting their company on fire and walking away.
| bombcar wrote:
| Your username would have been a good option for a cutesy name
| for the vulnerability, however.
| [deleted]
| hellcow wrote:
| > Anything other than an immediate response would have been
| akin to lighting their company on fire and walking away.
|
| Have we forgotten Zoom, who reinstalled itself secretly on user
| machines with an RCE-vulnerable server, which they described as
| "working as intended?" They're still wildly popular today with
| organizations despite the insane lack of regard for security
| and their users' safety.
|
| Mistakes happen. I applaud Tailscale for moving so quickly and
| doing the right thing.
| xmodem wrote:
| I would caution against grading on a curve
| loa_in_ wrote:
| > Anything other than an immediate response would have been
| akin to lighting their company on fire and walking away.
|
| Companies get away with sweeping customer security issues under
| the rug less and less, but still far too often. I honestly wish
| we as a people would put other players of this game in as high
| standards as you do here for this company here.
| acuteaura wrote:
| As the reporting party of the much less severe (and much less
| interesting) TS-2022-003[1] I can confirm that vendor
| interaction is also swift when not responding would not light
| the company on fire. In fact there were several other vendors
| that were affected that have yet to mitigate it.
|
| [1]: https://notes.acuteaura.net/posts/github-enterprise-
| security...
| ghuntley wrote:
| Technical write up by the security researcher at
| https://emily.id.au/tailscale
|
| ps. she's looking an employer rn // hire her!
| dang wrote:
| That article has so much more information in it that I've
| changed the URL to that from https://tailscale.com/security-
| bulletins/#ts-2022-004. Thanks!
| [deleted]
| erdaniels wrote:
| can you DM me her email? I can't find it anywhere and I'd love
| to start a chat with on a security position at my work.
| maxmcd wrote:
| See here: https://angus.ws/
| ghuntley wrote:
| Have passed your comment over to her. The website has been
| updated with an email address.
| IshKebab wrote:
| Very well written. Also quite worrying given they're supposed
| to be a security company and these kinds of issues are well
| known.
|
| Then again it does seem like the entire universe applies "eh
| probably nobody will try and hack it" to services listening on
| local TCP interfaces.
|
| They certainly don't care about multi-user machines, though I
| suppose there are so many local root exploits these days you're
| basically trusting your users anyway in that situation.
| beanjuiceII wrote:
| sounds like tailscale should
| ghuntley wrote:
| that would be a really cool outcome (tm)
| er4hn wrote:
| where does she say she is looking for an employer? Would be
| worthwhile to start a conversation with her.
| ghuntley wrote:
| Em interned with me earlier this year. Have passed the
| threads over to her.
| freedinosaur wrote:
| > In theory, there is no path for a malicious Tailscale control
| plane to remotely execute code on your machine, unless you happen
| to run network services that are designed to allow it, like an
| SSH server with Tailscale-backed authentication.
|
| Now I feel less crazy for not using Tailscale SSH for similar
| reasons.
|
| I'd like to see a security evaluation of Tailscale, on a per
| feature basis.
|
| I'd like to see tailscaled run with far fewer privileges.
|
| Is there a Tailscale alternative that just does Wireguard + NAT
| traversal and doesn't try to do key management?
| infotogivenm wrote:
| Yep. Same boat. Absolutely zero interest in granting them ssh
| authZ; transport wrapping is all I want to outsource. Just
| deliver my bits and I pay you, tyvm. My suspicions have been
| proven correct here.
|
| Unfortunately reading about this remote RCE vector has me
| wondering whether I can use the product at all without all this
| bloat (taildrop, ssh, etc) affecting me. Going to have my team
| look at zerotier this week, I've heard a few ok things.
| klabb3 wrote:
| > Is there a Tailscale alternative that just does Wireguard +
| NAT traversal and doesn't try to do key management?
|
| I really wish there was a NAT traversal protocol or library
| that wasn't overly complex and focused on the 90% cases. It
| would help not just tailscale's but anyone building p2p tech.
| anderspitman wrote:
| I believe libp2p has some NAT traversal stuff:
| https://docs.libp2p.io/concepts/nat/
| freedinosaur wrote:
| https://github.com/hyprspace/hyprspace is built on top of
| that. It's remarkably simple: libp2p's DHT + libp2p's NAT
| punching + TUN device.
| saghm wrote:
| I'm not sure I've ever seen a detailed technical writeup of a
| vulnerability before that started with such clear and concise
| instructions on the exact steps needed to defend against it at
| the start of the article before. In particular, making clear the
| priority of what to patch is excellent. If I'm a user of a
| product where a bug was found, I'm definitely interested in
| learning about what the bug was, how it was discovered, and
| whether I should be worried about other bugs in the future, but
| the absolute first thing I want is to do whatever I can to make
| sure I'm not affected by it. Listing what to patch and/or change
| in code might be more "boring" than the narrative of how the bug
| was found, and it might spoil the surprise, but I think sometimes
| we focus so much on the fun of the process of finding the bugs or
| revel in the cleverness of an attack (and those things are fun!)
| that we forget that the real point to it is to make our stuff
| safer. There's plenty of time for fun, but make sure you patch
| things first!
| [deleted]
| paddlepop wrote:
| I call this the "Vulnerability view" instead of a "Remediation
| view" and its something I feel a lot of Security people and
| tooling gets wrong when sharing information with those outside
| our bubble.
|
| It is dead easy to export a vulnerability scan or penetration
| test report and throw it at the developers, but you will get
| much better outcomes and better rapport if you tell them what
| they need to do (i.e. patch to version x.x.x) versus telling
| them what is wrong ("the sky is falling!").
| [deleted]
| [deleted]
| jacobr1 wrote:
| Also interesting was this recommendation: "Keep using
| Tailscale!
|
| The speed and quality of Tailscale's response to our report is
| unlike any vendor interaction I have experienced, and suggests
| a deep commitment to keeping their customers safe."
|
| Every product has security issues now and then. The real
| challeng is building robust processes remediate them (and to
| ensure that class of issue doesn't reoccur). The teams that
| deliver, by being transparent and fixing their stuff in a
| timely way, get my business.
| radicaldreamer wrote:
| They should immediately blacklist the affected client versions.
| arc-in-space wrote:
| I... didn't get an email? Very cool to find out by looking at hn
| enneff wrote:
| I did. Are you running an affected node?
| [deleted]
| cassianoleal wrote:
| To potentially save you a click: Who is
| affected? All Windows clients prior to version
| v1.32.3 are affected.
___________________________________________________________________
(page generated 2022-11-21 23:00 UTC)