[HN Gopher] Ultra fast Thunderbolt NAS with Apple M1 and Linux
___________________________________________________________________
Ultra fast Thunderbolt NAS with Apple M1 and Linux
Author : cyberge99
Score : 131 points
Date : 2021-08-03 13:24 UTC (1 days ago)
(HTM) web link (chrisbergeron.com)
(TXT) w3m dump (chrisbergeron.com)
| boynamedsue wrote:
| I just wish Apple would offer a native iSCSI initiator.
| FullyFunctional wrote:
| Funny I went down this rathole very shortly after discovering the
| "Thunderbolt networking" change in the Linux kernel. My hope was
| the use the Linux box (with 10 GbE) as a bridge/switch to give my
| Mac cheap 10 GbE access. While point-to-point did hit more or
| less the expected speed, bridging was horribly slow. Ended up
| just getting an Sonnet Solo10G SFP+ adapter (which is great).
| 21com wrote:
| > bridging was horribly slow.
|
| Can you quantify that? (I'm wondering if it might be good-
| enough for my needs.)
| insaneirish wrote:
| There's something weird here: root@nas-04:~ #
| ifconfig thunderbolt0 10.1.1.2 root@nas-04:~ #
| ifconfig thunderbolt0 thunderbolt0:
| flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
| inet 192.168.10.2 netmask 255.0.0.0 broadcast 10.255.255.255
|
| The IP on the interface is different than the one that it was
| theoretically configured with. My suspicion is that perhaps the
| IP used was changed at some point and this is just an editing
| slip.
|
| > Now that the interface has an IP, we have to add a static host
| route entry for it to communicate:
|
| If both sides are configured with an appropriate netmask on the
| same subnet, there's no need for a specific host route.
| cyberge99 wrote:
| Author here: You're correct, it's an editing error. There's
| another where I do ifconfig on the mini, but I pasted from the
| nas. I'll correct the post.
| reacharavindh wrote:
| Not to take away anything from the fun of figuring out
| Thunderbolt on Linux, and such. But, I wonder if a NAS with a 40
| Gbps or even 100Gbps NIC hooked up to a Thunderbolt to Ethernet
| adapter would achieve a better result without extra effort.
| [deleted]
| minhazm wrote:
| The fastest thunderbolt to ethernet adapter I could find is 25
| Gbps and it's $1500. 10 Gbps ones are in the $200+ range as
| well.
| [deleted]
| osmarks wrote:
| Buy an eGPU enclosure and put a NIC in that?
| reacharavindh wrote:
| True. I assumed it was a simple adaptor for TB3 to 40Gbps
| Ethernet or atleast QSFP+ Since TB also advertises 40 Gbps,
| but looks like expensive niche. At $1500, you might as well
| spend time to figure out how to get Linux to work better with
| Thunderbolt card.
|
| For other devices in the network that do not impose
| Thunderbolt though, Melanox 40G NICs are surprisingly cheap.
| lostlogin wrote:
| For 10Gbe it's slightly cheaper if you go to SFP+, but then
| you need sfp cables. I went this route as they are also
| cooler. Then I got sick of Thunderbolt finicky config and
| went for cheap Mellanox PCI cards which are truely excellent
| and basically zero config.
|
| The dongle is QNAP QNA-T310G1S Thunderbolt 3 to 10GbE
| Adaptor, US$169 on Amazon.
| jasonjackson wrote:
| probably, but thunderbolt would cost 5x less for equivalent
| 40Gbps NIC
| schmookeeg wrote:
| This has not been my experience -- I use both thunderbolt 3
| and Mellanox ConnectX-3 40gbps cards, and have been looking
| for an excuse to add a Mini M1 to the homelab.
|
| The cables and cards for PC Thunderbolt retrofit are far
| (like 3-5X!) more expensive than the MLNX cards and optical
| cables.
|
| $0.02 :)
| jsjohnst wrote:
| > The cables and cards for PC Thunderbolt retrofit are far
| (like 3-5X!) more expensive than the MLNX cards and optical
| cables.
|
| I added _new_ Thunderbolt3 PCIe cards (Gigabyte Titan
| Ridge) to two of my 2012 Mac Pros for around $100 each a
| bit over a year ago and the prices have dropped since, so I
| don 't get how that is 3-5x more expensive than MLNX cards.
| Expense was way way lower a factor than the finesse it took
| to get MacOS to support them for me, but it was the only
| path that worked for my needs (different need than OP
| article) and so I endured the pain of getting it working.
| kllrnohj wrote:
| You can find tons of used 10-40Gbps NICs on ebay for $50 or
| less. For a simple point-to-point connection with both sides
| having PCI-E slots that's going to be easier & cheaper to set
| up than thunderbolt will be. It'll also be more reliable &
| more practical since getting 40Gbps over Thunderbolt requires
| extremely short cables, whereas you can get DAC cables for
| 40Gbps QSFP or Infiniband up to 7M or so. And there's no
| confusion around what the cable can actually do, unlike the
| nightmare that is finding the right USB-C cable with the
| right signal integrity requirements at the right distance for
| what you actually need/want.
| xiconfjs wrote:
| this blog uses webp as image format without a fallback option.
| Had to switch to Firefox because Safari wouldn't display the
| pictures.
| cyberge99 wrote:
| Thanks for pointing this out. I'll add a fallback using the
| "picture" element (TIL).
|
| I also use some javascript to delay rendering until the browser
| window stays on an image. That could very well be introducing
| some wonkiness.
| interactivecode wrote:
| you should use the built in lazy loading
| https://caniuse.com/loading-lazy-attr or intersection
| observer for this https://developer.mozilla.org/en-
| US/docs/Web/API/Intersectio...
| mpweiher wrote:
| Odd. Displays fine on my Safari 14.1.2. Checked and the images
| do appear to be web for me as well.
|
| Current Big Sur release, not a Beta.
| definitelyhuman wrote:
| Safari 14.1.2 on latest Catalina doesn't render
| michelb wrote:
| Only safari on big sur has webp support.
| jeroenhd wrote:
| That's very strange. Is this a versioning problem (i.e.
| Catalina not being supported anymore?) or is there some
| kind of hardware restriction?
|
| From what I can find, Catalina should still be supported
| for more than a year from now, right?
|
| Edit: well, that's bizarre. Caniuse.com [0] lists the
| same restriction, WebP is only available from Big Sur
| onwards. And then people wonder why nobody likes
| developing for Safari.
|
| [0]: https://www.caniuse.com/?search=webp
| SahAssar wrote:
| For media formats the support is often dependent on the
| OS. It's been that way forever (ask anyone deploying
| video in early html5 and getting it to work cross-os and
| cross-platform). I guess it's more expected with videos,
| but images are (usually) handled the same.
| jeroenhd wrote:
| Maybe with hardware decoding, but software decoding is
| often done in browsers when hardware support is not
| available.
|
| I suppose Apple doesn't want to include a software
| fallback for platforms where hardware isn't available to
| discourage developers from using the format. It's not
| like Apple would need to write any complicated decoding
| algorithms when there's already an open source
| implementation that's free to use.
| SahAssar wrote:
| H264 didn't work in FF on MacOS and Linux (but it still
| worked on windows) until v34 (and it only started working
| because Cisco donated a license), Ogg Vorbis only works
| on MacOS 11.3 or later in safari (and it also depends on
| which container you use), HEVC in IE/EdgeHTML depended on
| hardware (with no fallback to software whatsoever), AV1
| in FF65 only worked on Windows, AV1 in FF66 only worked
| on Windows and MacOS, chrome on android 2.3 required you
| to specify a m4v without mime-type, but doing it with
| mime-type worked fine on desktop chrome.
|
| I'm just saying it's not unprecedented to have the same
| browser version supporting different audio/video/image
| formats depending on OS or hardware.
|
| For image formats it's not as usual, but if someone is
| making a site only supporting webp then I'd assume they'd
| look up some support matrixes beforehand.
| stalfosknight wrote:
| Works fine in the latest Safari.
| jeffbee wrote:
| Where by "ultra fast" it is meant that it is much slower than a
| directly-attached SSD that occupies the same port.
| renewiltord wrote:
| Well, yes, that is implied by the fact that it says "NAS" in
| the title.
|
| In general, you can usually do faster local than over the
| network. People use a NAS for other reasons.
| dylan604 wrote:
| That SSD has 0 redundancy so you're trading the purpose of the
| NAS by going for speed. If that's what you want, then you're
| wasting time looking at NAS equipment anyways.
|
| So what's your point?
| CoolGuySteve wrote:
| Most SSDs have a lot of redundancy in the form of multiple
| flash chips, it's just not exposed to you, the user, by the
| controller chipset.
|
| There are also PCI-E x16 with multiple NVMe slots. There's
| nothing stopping some company from making something similar
| for Thunderbolt.
|
| And I guess the cost is too high and the connectors would be
| too bulky, but it would be nice to have something like a
| SODIMM for flash that separates the memory from the
| controller.
| dylan604 wrote:
| Are we really comparing the "redundancy" of an SSD internal
| workings to a NAS? You're reaching really far for that.
| Whatever "redundancy" an SSD has is whatever it needs just
| to present a single instance of that data to you. Sounds
| pretty damn ineffecient, but if that's what it takes to
| reliably get 1200MB/s speeds I'm okay with it for
| situations where I need that.
|
| The redundancy a NAS can provide means losing an entire
| drive (or more depending on config) in the unit without
| data loss. Even with that failed drive, the data is still
| accessible. The failed thing can then even be replaced and
| the system can be brought back to the normal full redundant
| state. Your "redundancy" of an SSD doesn't allow for that
| to happen.
|
| I feel like I'm actually feeding trolls with this
| reacharavindh wrote:
| Local storage and Networked storage always have their places.
| Local solid state storage almost always gets you better
| performance because you negate the overhead of network. Unless,
| you're talking fancy arrays connected to saturate 100G NICs.
|
| Local storage falls apart the moment you need that storage
| accessible from another machine...
| systemvoltage wrote:
| Dumb question: Can you use Mac's shared folder for another
| machine to access files?
| filoleg wrote:
| Yep, it is possible. Here is the official support page on
| how to set up a shared folder that would be accessible by a
| any machine using SMB[0], and here is one for access by a
| macOS machine[1] (that is a bit simpler for setting up
| certain advanced config options).
|
| 0. https://support.apple.com/guide/mac-help/share-mac-
| files-wit...
|
| 1. https://support.apple.com/guide/mac-help/set-up-file-
| sharing...
| jeffbee wrote:
| This doesn't say anything about how such storage would be
| accessed by the mac and some other clients at the same time.
| It literally just runs iperf over tb. As a matter of
| architecture it leaves the question of whether it would be
| preferable to directly attach the storage to the Mac and
| export it to the rest of the network from there.
| cyberge99 wrote:
| The maturity of Thunderbolt hardware and software support should
| have TB on a short list of high speed local connectivity, high-
| bandwidth multi-disk access options.
| Teknoman117 wrote:
| This is cool, but my main takeaway is that you can get around
| needing the thunderbolt enablement header by shorting two pins
| together and use it on any motherboard?
|
| What the hell Intel...
| unicornfinder wrote:
| If memory serves the main problem is some Thunderbolt cards
| don't ship with the necessary firmware on them, so would need
| to be used at least once in a supported motherboard before just
| used in any system.
| pronoiac wrote:
| > In this post I discuss how you can upgrade a NAS Server by
| adding Thunderbolt 3 for lightning fast connectivity at 20 or
| 40Gbps.
|
| This is an interesting angle of attack, though I'll note that
| 10Gbps Ethernet was added as an option on the M1 Mac Minis.
| MayeulC wrote:
| here's an idea: use link aggregation yo make it 10 G faster,
| then.
|
| I wonder what's the simplest option. could it be made to work
| with a virtual interface, like wireguard?
| whalesalad wrote:
| Just yesterday I was thinking how badly I want an obscenely fast
| local datastore for Docker. I am managing a handful of clients
| right now who are all using Docker containers for their
| microservice environments and my local machine runs out of disk
| daily.
|
| What I really want is lightning quick _network_ storage, but I
| don 't think it would be feasible to roll out 10g networking in
| my current rental home.
| kilroy123 wrote:
| Just curious what size is your hard drive?
|
| I had this pain with my last macbook. I ALWAYS ran out of disk
| space. Never again! I went with a 1 TB drive for my M1.
| dijit wrote:
| I offload docker to another machine and I'm still using about
| 400GiB of storage on my relatively new MacBook; and I
| basically never use this machine outside of work (and even
| then I didn't use it for like 4 months because I preferred
| linux). it has no local media or anything.
|
| I'm shocked by how much disk I've used actually.
|
| https://imgur.com/a/iHV4IKn
| biztos wrote:
| I got the 2TB drive so I can run out of _even more_ disk
| space! ;-)
| devonkim wrote:
| 10gbe can be done over RJ45 with a direct cable connection and
| is something I've done before in a pinch between my personal
| NAS and a client machine (just be careful about your storage
| protocol because NFS has its pros and cons just like iSCSI and
| FCoE, and with macOS as a client you're likely in for a Bad
| Time like I did).
| vitejose wrote:
| Out of curiosity, what issues did you run into with NFS on
| macOS? I'm looking into building a NAS for a network of
| macOS, Linux, and BSD machines and figured NFS would be my
| first choice.
| devonkim wrote:
| NFSv3 and NFSv4 clients were fine when local machines over
| TCP and always-on like with a previous Mac Mini I had but
| resuming from a laptop was pulling teeth and may or may not
| have been the reason for some of the crashes I got. iSCSI
| in theory should be better but I had a lot of problems
| getting the iSCSI initiator I found to work for myself
| despite being a pretty simple use case I thought, and I
| gave up on the project around then as I found macOS simply
| required more budget for myself to buy an appropriate iSCSI
| initiator. This was over 5 years ago and things may have
| changed between then and now.
| geerlingguy wrote:
| I've had weird issues with some applications and NFS on
| macOS (notably, Final Cut Pro X and Photoshop). And I've
| also had weird issues with SMB and some apps (notably,
| Photoshop again!).
|
| But at least SMB has been pretty stable for years. NFS
| support has had some hiccups. A few 10.14 macOS releases
| broke NFSv3 and NFSv4 for me, but it was fixed in later
| releases.
| 411111111111111 wrote:
| I'm not the author nor am I an apple user anymore, though i
| was at one point.
|
| NFS doesn't have a lot of metadata, so you loose a lot of
| functionality. iPhoto for example could cause major issues
| if you persisted its library on a nfs, potentially making
| the library irreparable.
| yawniek wrote:
| why don't you simply use one or many nvme thunderbolt/usb disks
| and move the caches there?
| ehutch79 wrote:
| Not the op, but if the nas was pre-existing, and already
| serving other network clients, no reason to add a second one.
| cyberge99 wrote:
| Author here: I actually used a thunderbolt nvme and adapter
| for a bit. Then, for kicks I got another one and combined
| them in a raid-0 on macos. Realizing I didn't want to run
| nvme's, in raid-0 over two cables in any sort of production
| (homelab - I use production loosely here), is what sent me
| down the Thunderbolt rabbit hole that precipitated this blog
| post.
| FullyFunctional wrote:
| Not OP, but my files are shared with many hosts. Having local
| files are a PITA to manage coherent state.
| cbushko wrote:
| Would running something like the docker proxy below on an old
| laptop help? TLDR: A caching proxy for Docker;
| allows centralized management of (multiple) registries and
| their authentication; caches images from any registry. Caches
| the potentially huge blob/layer requests (for bandwidth/time
| savings), and optionally caches manifest requests ("pulls") to
| avoid rate-limiting.
|
| https://github.com/rpardini/docker-registry-proxy
| whalesalad wrote:
| At some point you need to build images locally or store
| images locally to run a local docker-compose cluster.
|
| I need a fat block store for that.
| cbushko wrote:
| That is painful for sure.
___________________________________________________________________
(page generated 2021-08-04 23:00 UTC)