[HN Gopher] Improving Tailscale via Apple's open source
___________________________________________________________________
Improving Tailscale via Apple's open source
Author : mfiguiere
Score : 269 points
Date : 2023-04-13 17:36 UTC (5 hours ago)
(HTM) web link (tailscale.dev)
(TXT) w3m dump (tailscale.dev)
| omneity wrote:
| Slightly related, I use Zerotier for some personal stuff as a
| managed VPN, including connecting several Apple devices. I never
| see it mentioned in HN (Zerolove for Zerotier lol), does anyone
| who used both knows what am I missing out for not using
| Tailscale?
| stingraycharles wrote:
| We recently migrated away from our old OpenVPN deployment, and
| Zerotier was on the table. In the end we picked Tailscale
| because it's based on open protocols and more transparent in
| how they work.
|
| From a different perspective, it appears Zerotier is investing
| a lot of engineering efforts in maintaining its own protocol,
| with arguable benefits. Tailscale seems to be deploying new
| features on an almost weekly basis and are having huge
| velocity, and to me it just appeared their trade-offs made more
| sense, and I have more confidence in their business doing great
| on a long term.
| 0xbadcafebee wrote:
| Isn't there usually vendor documentation of internal system
| calls? I know Microsoft used to provide it, though I think you
| had to pay for MSDN? I assume Apple has something similar.
|
| In terms of finding a syscall, you don't usually need to dig
| through source code. On Linux, _strace_ and _gdb_ will show you a
| program 's active system calls, and on MacOS it'd be _dtruss_.
| macshome wrote:
| Things like this are documented in the Apple sources and header
| files. Even if there are stubs of documentation they pretty
| much just give you the names of files, or man pages, to go
| read.
|
| Once you get up out of the POSIX layers the Apple documentation
| has improved a lot lately.
| raggi wrote:
| You need both at depth. strace and dtruss translate binary
| information to a textual representation but they're only as
| good as their coverage. When you get closer to the edges of
| common use they fall down. On macOS / iOS using dtruss for all
| programs comes with extra burdens, too.
|
| We (tailscalar here!) have another post out today on more
| throughput optimizations and the story there is related in this
| way: when we started working on that code path the docs didn't
| exist! (There are some docs now, in Linux). But either way the
| stuff is seldom used outside of specialist implementations - so
| when you need to understand what's going on you're headed for
| the source, or a debugger, or a symbolic tracer rather than an
| interpretive one.
| drewg123 wrote:
| Shared / Open source is super helpful when you're trying to
| figure out how to do something unusual that's not on the paved
| path covered by the docs and examples.
|
| In the early 2000s, when doing Mac drivers for an OS-bypass HPC
| NIC (Myrinet, a sort of pre-cursor to inifiband/roce), I spent
| 90% of my time reading the Apple source and about 10% of my time
| coding. In my case the first goal was to figure out how to use
| BSD ioctls rather than Mach based userclient stuff, as we had
| tons of shared kernel and userspace code for ioctls (across
| linux, dec unix, solaris, freebsd, windows, etc). And the second
| was figuring out exactly which iomemory descriptor variant would
| work well for registering memory.
| ignoramous wrote:
| > _Shared / Open source is super helpful when you're trying to
| figure out how to do something unusual that's not on the paved
| path covered by the docs and examples._
|
| Not just the unusual, but the _usual_ too: Just this past month
| or so, a user on Fly.io forums helped debug just why reading
| from _stdin_ / writing to _stdout_ didn 't work for non- _root_
| users (https://community.fly.io/t/10375/8). It all started from
| the fact that Fly.io had open sourced a snapshot-in-time of
| their _init_ process back in 2021.
|
| Also, it isn't uncommon in the Android world for developers to
| routinely find the right APIs to use by reading the OS source
| code.
| iudqnolq wrote:
| A very meaningful point in my learning as a junior developer
| was when I realized I should regularly click go to definition
| on standard library functions.
|
| I needed to tweak the behavior of an Android standard library
| function, so I copied it's source code (~20 lines) and made
| some edits. Blew my mind at the time it was that simple.
| [deleted]
| jamesfmilne wrote:
| Interesting background. I've recently been toying with the idea
| of writing some open-source Mac DriverKit drivers for Mellanox
| cards, so I'm not forced to pay through the nose for ATTO cards
| (which are just rebranded Mellanox cards.)
|
| Sadly all the DriverKit stuff appears to be completely closed
| source. At least its user-space though, so perhaps a bit less
| painful than normal kernel driver development...
| skrtskrt wrote:
| funny side note about Tailscale/iOS: one of the first things I
| tried with Tailscale was TailDrop-ing a random file to my iPad.
|
| I chose a dotfile like .gitconfig, but then the iOS Files app
| wouldn't show it to me. It would show me that there was one item
| of nonzero size in the folder, but I could not see the actual
| file, and now (I assume) I can't delete it either unless I delete
| the folder.
| heyflyguy wrote:
| I love tailscale, but an unusual thing happened when I upgraded
| my home pc to Windows 11. Now my tailscale still connects to all
| of my other computers and servers, but I can't browse any network
| shares. I guess authentication changed in Windows 11 and is not
| backward-compatible. Tailscale worked marvelously for me for
| ages, but Windows 11 for sure threw a wrench in the gears!
| unraveller wrote:
| I had to manually enable workstation service on my win11
| install to get my home network shares working and before that
| in win10 I had force guest mode just to browse freely and
| remote desktop. AllowInsecureGuestAuth 1
| HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWork
| station\Parameters
| heyflyguy wrote:
| You are quite a person and I thank you
| skrtskrt wrote:
| their clients have to do a _lot_ of platform specific
| configuration to make it appear magic- OS /distro-specific
| network configuration is always going to be a hairy, moving
| target - open an issue on their GitHub and they'll probably
| help you start poking at it
| EMIRELADERO wrote:
| Nice blog post! I like the way this problem is approached and
| solved. Peeking under the "table" to see how the internals work
| is actually creative when trying to fix things "upstream" on an
| upper abstraction layer.
|
| Unfortunately, I still don't love Tailscale. I _do_ like it, a
| lot even. But their refusal to open-source all their clients (and
| the server) is baffling, especially considering that they have an
| employee contributing to Headscale, the community-led FOSS
| tailscale server. At that point just open-source the damn thing!
|
| Issues aside, it's still a great product. It actually felt like
| magic when I first used it in a way few technologies have.
| wg0 wrote:
| There is some money (rightly and legitimately) to be made as
| well. We all are Craftsmen. We all deserve some.
|
| There's headscale[1] that fills the gap somewhat.
|
| [1] https://github.com/juanfont/headscale
| bradfitz wrote:
| We'd planned to open source our control server but it wasn't in
| a good shape to release at the time we released the other
| stuff. Then Headscale came along and removed all need for us to
| do so. Headscale is _much_ easier for people to run &
| understand. The Tailscale closed source one is kinda a monster,
| built for a very different scale. We're busy enough without
| also helping people struggling to run our control plane. We'd
| rather focus the community (or the subset of the community that
| wants to run their own server) to use Headscale instead.
|
| And if you don't trust our server, use
| https://tailscale.com/blog/tailnet-lock/ and then you don't
| need to trust us. Or run Headscale. :)
| EMIRELADERO wrote:
| Thank you for your answer.
|
| Honestly I (and I suspect most here) wouldn't mind if the
| server was not easy to set up. The goal here would be
| transparency. It's true that, with lock, a user can run
| Tailscale without having to trust it. But it is still a good
| show of good faith and goodwill to have everything in your
| infrastructure be as transparent as possible, barring actual
| user data and service credentials.
|
| Same concept applies to the proprietary GUI clients. What's
| the rationale for not, at least, making their source code
| publicly available for reproducible builds (or, if those are
| too complicated to implement, the same goodwill and
| transparency I talked about)? You wouldn't even need to
| actually support the source releases.
| bradfitz wrote:
| FWIW, you can run open source Tailscale on macOS and
| Windows without the GUI.
|
| And you can already do reproducible builds of our Windows
| build: the `tailscaled.exe` service and `tailscale.exe` CLI
| are open source. Only the GUI systray client (tailscale-
| ipn.exe) is closed.
|
| For macOS and iOS, our development environment is kinda
| hell. It's great when it's finally working, but hard to get
| it set up and keep it in a happy place ... you have to get
| users into the right Apple teams for the right Network
| Extension Entitlements/notarization/etc, disable SIP to
| work on certain types of builds, be sure to clean up Xcode
| temp folders in ~/Library/ so the system doesn't pick up
| the wrong builds, etc, etc. Then you think things are good
| and in a few months a random keychain cert expires and you
| have to repeat the dance. Which sometimes involves a few
| macOS reboots for some reason.
|
| Yes, maybe we could say that the macOS/iOS/Windows GUI
| source releases are "not supported" but that will stop
| approximately nobody from asking questions anyway and
| consuming time.
|
| Plus I always come back to the question: if you care about
| open source so much, why aren't you running Linux?
|
| The common reply is: but trust! but security! but
| auditability! Know how many corporate/paying customers have
| asked for open source Windows/macOS/iOS GUIs? Zero that
| I've heard about. Their trust relationship is with other
| companies, not with codebases they don't have time to build
| or audit anyway. Or they trust us that all the interesting
| code (non-GUI wrappers) is open source anyway and read
| that.
|
| So, yes, we _could_ open source our GUIs. But it's not
| worth the resulting pain. It'd save me writing comments
| like this one, but then I'd be answering Xcode/mac build
| questions instead, and I'd much prefer writing this.
| freedomben wrote:
| > _Plus I always come back to the question: if you care
| about open source so much, why aren 't you running
| Linux?_
|
| Yes, exactly. I'm one of the biggest FOSS advocates
| around, but if someone is running Mac or Windows, I have
| a hard time accepting criticism from them about closed
| source software. They are obviously pretty comfortable
| with closed-source/proprietary blobs. If you're running
| closed systems, you're voting for closed software. It's
| pretty hypocritical to criticize others to go open when
| you don't.
|
| With my FOSS projects I do try to support macs, and if
| it's relatively easy then windows, but I don't blame
| anyone for taking a "like for like" approach to their
| software. Open for people who vote open, closed for
| people who vote closed. Live by the sword, die by the
| sword, or something like that.
|
| Seems quite fair to me. It's also arguably better for
| Linux/open source, because if more software vendors took
| this approach with their software, we'd see more people
| choosing Linux because it would give us another
| competitive advantage. That leads to more Linux users,
| and the more Linux users there are the better it is for
| the whole community. More software vendors will support
| Linux, for example.
| slimsag wrote:
| You don't need to be vegan to care about animal rights.
|
| I can consume closed-source software and care about FOSS
| at the same time, do not discard people who are on your
| side just because they aren't extreme enough for your
| tastes.
| freedomben wrote:
| > _You don 't need to be vegan to care about animal
| rights._
|
| Yes, but that's not really equivalent to this situation.
| This isn't just a "do you care about FOSS" question like
| the "do you care about animal rights" analogy that you
| raised.
|
| In this situation, a person who pays for and uses closed
| source software (and not just one application, but the
| entire OS), is criticizing another company for making
| closed source software. It would be more equivalent to
| somebody who is a meat purchaser/eater criticizing a
| company for making meat.
| pdpi wrote:
| > Plus I always come back to the question: if you care
| about open source so much, why aren't you running Linux?
|
| Because it's not black and white.
|
| I can easily see myself in a position where I have
| influence over the VPN we use, while not having control
| over the finance people using Windows (because of the
| Windows-only accounting software) and the designers using
| Macs (for all the usual reasons). As a more general
| point, "I have stricter openness requirements for my
| infrastructure than I do for client devices" seems like a
| reasonable position to hold.
|
| I do appreciate that the armchair CIO crowd keeps raising
| auditably/trust/security as being much bigger issues than
| they are in practice, but I have had engagements in the
| past where the client _demanded_ that any softare we
| wrote on their behalf had to be open-sourced. So such
| requirements are in practice unusual but by no means
| unheard of.
| e28eta wrote:
| Have you considered the benefit of open source serving as
| an example of how to make a non-standard use case work?
| </sarcasm>
|
| I think I just wanted to point out the similarity. You
| _have_ to maintain the dev environment, and you _have_ to
| have it documented in some form for new hires. If that
| was public, we could be reading a blog post 6 months from
| now from some company that ran into similar-shaped
| problems in their dev environment, and solved it using
| "the power of open source"
|
| I don't think that's a compelling enough reason for
| someone trying to run a business, but I hope at least
| some people appreciate the irony here.
| ignoramous wrote:
| > _Their trust relationship is with other companies, not
| with codebases they don 't have time to build or audit
| anyway._
|
| Sums up the book, _Selling the Invisible_.
|
| (a pretty good summary:
| https://penniesintofortunes.com/2016/09/20/selling-the-
| invis...)
| Operyl wrote:
| As a paying customer, this is 100% true. Reputation helps
| a lot, the core of the product being opensource for a
| quick skim for some sanity helped. Support being there to
| help with questions about why X was done the way it was
| helped. We don't care or have the time to audit further.
| The company pays _others_ to help audit them, iirc
| Latacora.
| bradfitz wrote:
| Yes, we work with Latacora.
| (https://tailscale.com/blog/latacora-and-tailscale/)
| jzelinskie wrote:
| I don't think that their choice to keep their server software
| proprietary is baffling. They're going to architect it so that
| it's easiest/best for their team to run which is not the same
| what's easiest/best for EVERYONE to run as is the typical goal
| of open source software.
| ceejayoz wrote:
| There's two concerns: "can I run it myself", and "what does
| this security-critical third-party code do?"
|
| Open sourcing helps with the latter, even if the code's not
| very well optimized for general purpose use for the former
| case.
| judge2020 wrote:
| The concern is that people will assume the OSS is for both
| scenarios, so even if you close issues and have a PR bot
| rejecting PRs, you'll still get both support tickets and
| (good|bad) PR from people trying to run the mainline system
| themselves.
|
| Supporting development on a made-for-self-hosting OSS
| project is the most user friendly option and allows them to
| avoid needing to handle the woes of public code.
| daveidol wrote:
| What about open sourcing the code with a README stating
| it is only for transparency, comes with no support, etc.
| and disabling GitHub issues?
| Operyl wrote:
| Users will still just spam the ticketing system. (Their
| support email).
| ehPReth wrote:
| woah, they have an employee contributing to head scale now?
| skrtskrt wrote:
| parent comment was edited, originally said "i thought they
| [Tailscale] hated that thing [Headscale]"
|
| they literally have on their website that they encourage
| people to use Headscale if they want an open source solution
| for the coordination server or just to learn how the
| internals work, they coordinate with Headscale maintainers to
| avoid breakages, and they don't discourage (but also don't
| require) employees contributing to Headscale
| EMIRELADERO wrote:
| > parent comment was edited, originally said "i thought
| they [Tailscale] hated that thing [Headscale]"
|
| I don't know where you got this from. I _did not_ say that
| anywhere.
| skrtskrt wrote:
| you are not the parent comment to my comment - not sure
| if I used "parent comment" clearly
| ehPReth wrote:
| that was me! then i realized in the apps (except iOS?)
| they allow you to specify your own coordinator server...
| so if they hated them then they'd never build that in! so
| i removed that part of my comment before it had any
| replies that I was aware of
|
| edit: see comment below me; iOS now has that ability!
| mihaip wrote:
| Also possible on iOS starting in 1.38.1:
| https://tailscale.com/changelog/#2023-03-14-client
| ehPReth wrote:
| Awesome!
| bradfitz wrote:
| iOS allows that now too as of two releases ago (in
| 1.38.1).
|
| Go to the system Settings app, scroll down down down
| through your apps to Tailscale, and set the "Alternate
| Coordination Server URL".
| ehPReth wrote:
| ah! very cool. thanks for the context :). I must have
| missed the shoutout on the website. It's cool that they
| coordinate with headscale to avoid breakages as well
| i386 wrote:
| Product person here, I've been using and loving Tail scale for
| about a year now and I don't understand when I have to pay and
| why and I really don't understand why any of it is OSS. Where's
| the revenue and where's the moat?
| stingraycharles wrote:
| We're a customer and paying Tailscale a decent amount of $ a
| month and planning on expanding our usage more and more.
| They're probably just using a generous free tier as a growth
| tool.
| spiantino wrote:
| I wonder if this will help with the iOS app's battery drain
| issues. VPNs of any kind come with a battery penalty, but some of
| these issues might have exacerbated that
| neilalexander wrote:
| Using regular Wireguard with on-demand enabled and persistent
| keepalives disabled is about as good as VPN on iOS can possibly
| get. Without the keepalives, there is no idle protocol traffic
| and it is therefore barely consuming any battery power until
| traffic is sent over the tunnel.
___________________________________________________________________
(page generated 2023-04-13 23:00 UTC)