[HN Gopher] Node_modules: One character saved 50 GB of disk space
___________________________________________________________________
Node_modules: One character saved 50 GB of disk space
Author : feross
Score : 38 points
Date : 2022-09-30 21:02 UTC (1 hours ago)
(HTM) web link (mainmatter.com)
(TXT) w3m dump (mainmatter.com)
| bricss wrote:
| > npm dedup
| morelisp wrote:
| > I'd rather have two versions of the same dependency in a
| project, than not be able to use the dependency at all.
|
| This is where a poor choice was made and everything went wrong.
| lucideer wrote:
| This is an extremely clickbait title.
|
| The 50GB figure is the number of Node modules a developer has
| installed on their local machine that are duplicated between
| multiple Node projects/repos checked out on that machine. Even
| for the notoriously bloated JS ecosystem that seems well above
| average, even assuming any given developer has >1 project checked
| out at once.
|
| For reference, I'm a developer primarily working in JS. I use an
| old 2017 MB Air with the smallest disk (120GB) and have many many
| Node projects checked out (including random GH FOSS I've
| contributed to once). I don't use pnpm & I've never had disk
| space issues.
|
| Don't get me wrong, pnpm is cool. I've started trying it out and
| will likely convert a a lot of stuff. But 50GB is extreme even
| for Node.
| runme wrote:
| Is there any need to have so many dependencies in the first
| place? Seems like slovenly development practices are to blame.
| latchkey wrote:
| Kind of yes... not all dependencies are direct for the app, a
| lot are just dev dependencies. Just to get eslint/prettier to
| warn, auto format and cleanup my code when I save a file, it is
| 13 direct dev dependencies in my project [0].
|
| [0] https://github.com/lookfirst/mui-
| rff/blob/master/package.jso...
| vlunkr wrote:
| My projects tend to have vastly more dev dependencies as
| well. Especially with something like create-react-app. It's a
| pretty sad ecosystem.
| eric4smith wrote:
| The worst part about all these node modules is the little small
| silly ones that do something really inane - like to just get the
| current year.
|
| I said the same thing about some ruby gems years ago and
| thankfully that's a little bit sane now.
|
| I don't use JS that often. But recently I looked at the
| dependencies for some library I was using and I was astonished at
| the literally hundreds of tiny modules that were being used.
|
| And it gets even worse - those tiny little modules have their
| dependencies too.
|
| Amazing.
| encryptluks2 wrote:
| Imagine how easy it would be to exploit one of those too.
| simonw wrote:
| Right - every single dependency adds potentially another
| human maintainer who, if bribed or threatened, could release
| an update that exploits your project.
| rglover wrote:
| I don't understand why you're getting downvoted. I'm a JS
| developer (and framework developer) and what you describe is a
| serious problem. It's great that there's so many problems
| solved, but some stuff is just a one or two liner that should
| be in your own app's /lib, not a dependency.
|
| This is one reason I like Deno's idea of having a standard
| library (I just wish Ryan would have proposed that for Node
| directly instead of creating a brand new runtime).
| monlockandkey wrote:
| The solution to all this dependency mess if for NPM to make a
| standard library. Yeah that sounds crazy and weird. But they
| are in the best place to make a unified standard library for
| Javascript. This would bypass all the junk transitive
| dependencies and have more libraries rely on a centralised
| but standard library.
| torgard wrote:
| Someone linked me to 1-liners[0], which is - you guessed it - a
| bunch of one-liners. I think it's nice to have as a reference.
| But a dependency? Really?
|
| My least favorite is assign.
|
| Not only does JavaScript feature that natively (though I
| suppose the library may predate widespread support for
| Object.assign), the 1-liner assign _flips the order of the
| parameters!_ assign({ a: true }, { a: false
| }) -> { a: true } Object.assign({ a: true }, { a: false
| }) -> { a: false }
|
| And most of them are just straight-up pointless! Like, let's
| introduce a dependency for _decrement_ lol
|
| EDIT: In looking up whether the 1-liners assign predated
| widespread Object.assign support, I found that their
| implementation - confusingly named extend[1] at first -
| _literally used Object.assign from the very beginning._ And
| they _still_ chose to mess with the parameter order. For shame
| lol
|
| [0] https://github.com/1-liners/1-liners
|
| [1]
| https://github.com/1-liners/1-liners/blob/7c1f8d51df4b4b3e0a...
| alpaca128 wrote:
| A project I've been working on has roughly 40 dependencies. If
| you run `npm install` it'll pull about 1050 npm packages.
|
| Change one minor version number and everything breaks. Forget
| one dependency and npm will not tell you that a dependency is
| missing but instead it'll complain that Steam has a broken link
| in the home directory (this is a known open issue for years and
| the only two solutions are to uninstall Steam or to use a
| Docker container)
|
| Needless to say I am not a fan of large web projects.
| comprev wrote:
| Anyone who's run a CI platform for more than a few devs and
| NodeJS projects quickly bumps into inode problems unless they
| thought about build server filesystems in advance. Very quickly
| you end up with hundreds of thousands of minuscule files
| filling up the disk.
| kayodelycaon wrote:
| Maybe I'm out of date, but I thought hardlinks were generally a
| bad idea.
| denysonique wrote:
| This is where a file system with compression support becomes
| useful, such as ZFS or APFS.
| the_duke wrote:
| Or btrfs, which can also deduplicate files.
| mostlylikeable wrote:
| Fwiw, it looks like you can use pnpm as nodeLinker with yarn
| berry. I'm a fan of yarn's plugin support and the extensibility
| that provides, so I'll likely be trying that with pnpm as the
| linker and see how that goes.
| ordiel wrote:
| I am just amazed that this is still an issue for JS when it has
| basically being fixed for other languages, being such a largely
| used language with such a fanatic user base claiming "its the
| best" I would expect this no longer being an issue nor all the
| security issues that come with it
| taion wrote:
| It's not really "fixed" for other languages in general. The
| support in Node for multiple different versions of transitive
| dependencies is actually quite nice. In Python, for example,
| you simply can't have multiple versions of transitive
| dependencies, and this can lead to issues with commonly-used
| utility packages. I've seen issues like this come up with
| utility libraries like six or boto and its variants. Likewise
| with larger libraries like numpy.
|
| As someone who's worked pretty heavily in both ecosystems -
| it's definitely not something I think about every day on the
| Python side, but Python dependency conflicts are very
| annoying... while in Node they're mostly not a big deal except
| in a small set of cases where peer dependencies show up.
| viraptor wrote:
| A different / additional thing you can do on a few systems is
| compress the node_packages specifically. Afsctool on macos,
| chatter +c on btrfs, folder properties on Windows - you don't
| have to have compression enabled on the whole drive to use that.
|
| Since node_modules is mostly text, this has amazing results and
| can be applied to the deduplicated pnpm store as well.
| thijser wrote:
| The TL;DR is: use pnpm instead of npm.
| latchkey wrote:
| tl;dr: pnpm
| gadflyinyoureye wrote:
| Sadly my team had to go back to npm. Pnpm has issues in
| resolving dependences. Also the build box didn't have it. There
| were enough quirks to make it hard to write scripts that ran
| smoothly between npm and pnpm.
| allynood wrote:
| Good tl;dr !
| michaelsalim wrote:
| Thanks! I thought it was because of a bug or some interesting
| behavior.
___________________________________________________________________
(page generated 2022-09-30 23:00 UTC)