[HN Gopher] NTFS read-write GPL kernel driver
___________________________________________________________________
NTFS read-write GPL kernel driver
Author : ButSpider
Score : 161 points
Date : 2020-08-15 16:19 UTC (6 hours ago)
(HTM) web link (lore.kernel.org)
(TXT) w3m dump (lore.kernel.org)
| 0x0 wrote:
| Bad line-wrapping in the first commitmsg does not leave a first
| good impression.
| ufmace wrote:
| Everyone (on Reddit at least) seems to be yakking away at whether
| Paragon Software is good or bad and what other things they've
| done. I just wanna know if this driver is actually good or bad.
| Far as I know, a Linux NTFS driver capable of writing has been a
| long time coming.
| bayesianbot wrote:
| I've been writing to my NTFS volume for around 5-10 years I
| think, out of the box with Linux Mint. (Or maybe an extra
| package from apt-get, not 100%). Extra options needed and
| possible data loss when writing used to be a thing, but a long
| time ago.
| awild wrote:
| You're probably using ntfs3g which is separately installed.
| app4soft wrote:
| > _You 're probably using ntfs3g which is separately
| installed._
|
| Not sure about Linux Mint, but _MX Linux_ & _antiX_ has
| NTFS driver bundled into ISO-images.
| Soulsbane wrote:
| I've been able to read/write NTFS for years via NTFS-3G. Am I
| missing something? Check it out if you haven't already.
| Fronzie wrote:
| This is about a kernel-space driver, as opposed to a FUSE
| (user space) driver. It should allow for better performance.
| lizknope wrote:
| I've been writing to NTFS for at least 10 years. It was the
| FUSE driver but it worked fine and I never had any problem
| other than it seemed to write at half the speed of ext4
| speedgoose wrote:
| NTFS is super slow on windows too. You get better
| performances with ext4 on WSL2 (Windows Subsystem for Linux).
| mehrdadn wrote:
| This is not an NTFS issue. You can verify it by trying a
| different file system.
| risho wrote:
| how is this possible? isn't wsl 2 literally a virtual
| machine image on top of your ntfs drive?
| speedgoose wrote:
| Yes but it's only one big file, the ext4 partition.
| Tuna-Fish wrote:
| And to extend, Windows/NTFS is not particularly slow at
| accessing a large file (actually, until recently they
| have been ahead for async read/write into big files), the
| area where they really suck is anything that has anything
| to do with metadata.
| formerly_proven wrote:
| I've been told this has to do with NTFS using a
| complicated metadata scheme and also all permissions in
| NTFS being backed by a complex ACL-security-descriptor
| scheme. Is that an accurate representation?
| Tuna-Fish wrote:
| I don't believe it is. Windows metadata performance is
| not just bad, it's so bad it beggars belief. I don't
| believe the root cause is any specific technical
| decision, but rather that no-one cares about it. It has
| always been so bad that no-one can count on it, so no-one
| depends on it, so it's something that can safely be
| ignored.
|
| On Linux, and other unices, there have always been
| important workloads where it really matters, so that itch
| got scratched.
| ufmace wrote:
| I had heard the issue was with how Windows/NTFS let a
| bunch of extensions place hooks into reading and writing
| files. Antivirus apps would sometimes put a bunch of
| heavy-ish hooks in that would be terrible for performance
| with lots of tiny files, which many Unix-y workflows
| expect to be able to do quickly.
| opencl wrote:
| The "sometimes" includes the one Microsoft ships by
| default with Windows. It is ridiculous how much faster
| random file access becomes when disabling Defender's real
| time protection feature.
| neurostimulant wrote:
| I think it bypassed windows filesystem filters
| (antivirus, etc) and thus faster because there is nothing
| taxing your read/write operations.
| loeg wrote:
| So far, signs are not super positive:
| https://lore.kernel.org/lkml/87h7t5454n.fsf@suse.com/
| hannob wrote:
| This sounds like "yeah, it has bugs that need to be fixed",
| not fundamental reasons not to merge it. (Though I think they
| should've tested with KASAN+KUBSAN before submitting.)
| loeg wrote:
| Sure, bugs can be fixed. The bugs are one reason not to
| merge this iteration of the patchset. I don't dispute that
| it could eventually be polished into something worth
| merging.
|
| (The second bug in the email linked above is a NULL ptr
| deref during an in-order recursive directory copy into the
| volume. That is real basic stuff; I don't think it required
| ASAN/UBSAN to hit. It shows a lack of even relatively basic
| testing, if I understand correctly.)
| kelnos wrote:
| If a tester can't simply create a volume, mount it, and
| copy a directory tree into it without encountering a null
| pointer dereference, this to me raises serious questions
| about how this driver is tested and the lack of care in
| building it.
| binwiederhier wrote:
| We have licensed this driver and the fsck-like tool.
|
| The driver was deployed to 65,000 Ubuntu-based servers (as far
| as I remember), but we had a ton of issues with it, to the
| point that we removed it entirely and went back to NTFS-3g. We
| were told by paragon that we used a too-old version but at that
| point we had already moved on.
|
| However, their NTFS check tool that ships with the driver (I
| forgot the name, but I can dog it up if anyone is interested)
| is actually super awesome! It is so so so much better than even
| the Windows chkdsk tool itself, and corrects many more issues.
| We use it to fix NTFS errors in millions of backup images. Can
| highly recommend!
|
| Hope this helps.
| loeg wrote:
| Beware, it does not seem functional:
| https://lore.kernel.org/lkml/87h7t5454n.fsf@suse.com/
| [deleted]
| pengaru wrote:
| Does anyone really care anymore since ntfs-3g works well enough?
| formerly_proven wrote:
| ntfs-3g is pretty slow with a max throughput of around 80-140
| MB/s when using an SSD.
| mkup wrote:
| ntfs-3g does not support NTFS journal replay/rollback (e.g.
| after Windows unclean shutdown), also it does not maintain NTFS
| journal for its own operations
| rwmj wrote:
| Although I'd be surprised if this or any Linux NTFS driver
| could do that. It would require reverse engineering the
| journal format, which may even change with every Windows
| release.
| gertrunde wrote:
| I think that may be the point - it works, but not really well
| enough. If you need to use it to shift a large amount of data
| around, the performance is crippling.
| zamadatix wrote:
| I think it'd be nice to have a native driver instead of loading
| fuse and then a userspace driver. Is it going to revolutionize
| my NTFS access and make me toast in the mornings? No. Do I care
| to see it included? Yes.
| pengaru wrote:
| Maybe there's something about your use case where the
| potential performance difference between a kernel driver and
| FUSE driver is hugely relevant and outweighs the security
| problems.
|
| For me, the use cases for NTFS in my Linux-centric life are
| all fraught with trust issues when it comes to knowing what's
| in that filesystem isn't malicious and about to exploit a
| kernel fs driver bug for unfettered ring-0 execution.
|
| It's less of a concern with filesystems you initialized from
| your host, on devices permanently attached to said host, with
| a clear chain of custody, accessed purely from the trusted
| environment. There's a lot of implicit trust when it comes to
| mounting filesystems with a kernel driver, the kernel fs devs
| aren't shy about admitting there's significant trust
| assumptions throughout those drivers - they aren't exactly
| hardened against malicious input, and they aren't isolated
| processes like w/FUSE.
| zamadatix wrote:
| Personally when a box has a sensitive enough use case (e.g.
| secret storage hosts) that attacks via the local filesystem
| are a design concern I simply don't include either FUSE or
| additional filesystem modules nor do I plan on mounting
| additional storage after boot. I configure everything with
| minimal possible surface regardless of if it could be
| constructed with more convenience.
|
| I suppose some people may treat every box with that level
| of security profile but I don't think it should come as
| much of a surprise that level of hardening is not the
| normal use case considered when new drivers are staged in
| the kernel tree.
| formerly_proven wrote:
| > Maybe there's something about your use case where the
| potential performance difference between a kernel driver
| and FUSE driver is hugely relevant and outweighs the
| security problems.
|
| The idea of handling I/O errors is pretty recent in most
| Linux file systems, I don't think security concerns are
| much of an issue with adding another file system - the
| kernel is probably full of easily exploitable holes anyway.
| nitrogen wrote:
| I haven't checked in a really long time, but years ago ntfs-3g
| had a really suboptimal allocation pattern for spinning drives,
| and optimizations were only present in the commercial version.
| I don't know if the block allocation strategy was owned by
| ntfs-3g or if it was using something in-kernel.
|
| If I'm remembering the details right (and that's a _big_ if),
| large files written sequentially would have their blocks
| scattered all over the place, making e.g. large uncompressed
| videos written from Linux too slow to play due to seek times,
| when if they 'd been written sequentially they would have been
| fine.
| pengaru wrote:
| ntfs-3g is entirely implemented in userspace, and is far more
| trivial to hack on and test if anyone wanted to improve its
| file allocation strategies than an in-kernel driver would be.
|
| Years ago when working on a commercial virtualization product
| I had to support initializing sparsely populated ntfs
| filesystems where only the filesystem metadata was written
| out, with allocated file extent ranges logged externally and
| not actually populated in the sparse ntfs filesystem in a
| file. I ended up deriving a small library for doing this from
| the libntfs-3g code, I don't think it took more than a
| weekend to do, the code was quite sane.
|
| What I recall from that experience was that the file extent
| allocations on an empty filesystem was perfectly efficient,
| since you'd have to go out of your way to make it bad.
|
| The real trouble was when the filesystem was used enough for
| its free space to become fragmented. At that point the
| allocation strategy was quite poor and you would end up with
| extremely fragmented large files. IIRC it was really just a
| naive sequential find first free blocks kind of thing. This
| aspect is consistent with your memory.
|
| I think if we wanted to improve that aspect of linux ntfs
| support we're better off working from ntfs-3g.
|
| Not to mention there are some significant down sides to in-
| kernel filesystem drivers, they are rather problematic when
| it comes to containers/namespaces and the security story is a
| total shit show.
| sam_lowry_ wrote:
| Argh, Russian hackers, again!
| gigatexal wrote:
| Given the initial testing people on the mailing list have done
| and the issues that have arisen I hope the kernel maintainers so
| f no and push back until said issues are fixed and code quality
| rises.
| m000 wrote:
| Any guesses why Paragon decided to GPL the driver now?
| CameronNemo wrote:
| They didn't. Samsung wrote this driver and Microsoft proposed
| its inclusion into the kernel.
| rwmj wrote:
| Are you thinking about exFAT?
| CameronNemo wrote:
| Yep. My bad.
| rbanffy wrote:
| I haven't used a dual boot Linux/Windows machine in ages. How
| popular is this setting these days?
|
| On Linux, my most common scenario of deploying Windows is in
| small VMs and, when needed, mounting host folders from them. It's
| true these VMs will probably not run some PC software well, but I
| have a Windows PC for that. The other way around we have WSL and
| WSL2 on Windows, again kind of erasing the need to dual boot from
| the Windows user side.
|
| The money for them here is in support. Maybe they have licensees
| that bind them to support contracts and merging software into
| mainline Linux counts as a "best effort" to make others chip in
| at keeping it functional as the kernel evolves.
| SahAssar wrote:
| Dual booting is common for people playing games. While
| steamplay/proton works well enough for some games a lot need
| either a lot of hacking or a full blown windows to run. You can
| do GPU passthrough VMs if you have two GPUs but I have a
| miniITX board and no iGPU so that is not an option for me.
|
| Personally when dual-booting I don't share anything at all
| between the OS'es, but my understanding is that most do.
| mPReDiToR wrote:
| Was there something terrible about NTFS-3G?
| m000 wrote:
| Uhhhmmm... Speed? Stability? NTFS-3G has been a lifesaver for
| occasional use, but with anything remotely I/O intensive, it
| starts to crack.
| krzyk wrote:
| Why would you use NTFS on linux for anything other than
| occasional use?
| mimi89999 wrote:
| To deploy Windows from WIM imeges. Windows PE isn't
| something one would even consider using.
| ed25519FUUU wrote:
| Why should it matter?
| q3k wrote:
| Why not? It's a good filesystem, why not use it?
| nwallin wrote:
| It's kind of a garbage filesystem though.
|
| HN gets all bent out of shape about minor differences
| between btrfs and zfs. Like how zfs supports raid5/6 but
| btrfs raid5/6 support is immature. Meanwhile NTFS doesn't
| even support checksumming, multiple block devices, copy
| on write, the performance on SSDs is garbage, etc etc.
| nicoburns wrote:
| It's the only non-FAT filesystem with good cross-OS
| compatibiity.
| m000 wrote:
| This. Now, you've made me wonder if in the near future
| you could have both Windows and Linux installed on the
| same _-gasp-_ partition.
| brnt wrote:
| I had for years. Usung Wubi.
| Fnoord wrote:
| To make backups of data on a Windows partition. Playing
| games residing on a Windows partition. The main purpose I
| can think of though is forensics (images might contain
| NTFS).
| rwmj wrote:
| Stability hasn't been an issue for us. Virt-v2v uses ntfs-3g
| intensively and we have migrated a large fraction of a
| million VMs, many (the Windows ones) all using ntfs-3g. We've
| not had problems with ntfs-3g at all.
|
| I'm still interested to see if an in-kernel driver can be
| better.
|
| (Disclaimer: I have contributed a few patches to ntfs-3g
| upstream)
| rwmj wrote:
| The main problems are there have been no released versions
| since 2017, combined with an annoying "open core" development
| model where some nice to have features have been held back from
| the free version (although not ones we care about too much).
| For our use case (virt-v2v) speed is not an issue, but data
| integrity when reading, writing and modifying many dozens of
| files absolutely is, and ntfs-3g has been very reliable.
|
| (Disclaimer: I've made occasional contributions to ntfs-3g
| upstream)
| ilkkao wrote:
| Has this been merged already?
| the8472 wrote:
| It was posted yesterday and the replies indicate issues with
| the patch. So as these things usually go it'll probably take a
| long time until it's ready for inclusion.
| KindOne wrote:
| Discussion on reddit:
|
| https://www.reddit.com/r/linux/comments/i9mdqg/ntfs_readwrit...
|
| When exFAT was announced about 5 months ago Paragon Software
| posted a rant against that.
|
| Rant is posted here:
|
| https://news.ycombinator.com/item?id=22688058
| tomc1985 wrote:
| Paragon, purveyors of fine commercial filesystem drivers that
| corrupt my hard drives? LOL
| mehrdadn wrote:
| It's easy to laugh at them but I've had corruptions from
| Linux's existing NTFS implementation too.
| Quekid5 wrote:
| I guess it's not that surprising really[0], but it is quite
| amusing in combination with the company name.
|
| [0] If you tie your company's worth to a proprietary
| technology.
___________________________________________________________________
(page generated 2020-08-15 23:00 UTC)