[HN Gopher] Debian Running on Rust Coreutils
___________________________________________________________________
Debian Running on Rust Coreutils
Author : todsacerdoti
Score : 445 points
Date : 2021-03-09 09:01 UTC (13 hours ago)
(HTM) web link (sylvestre.ledru.info)
(TXT) w3m dump (sylvestre.ledru.info)
| hntestacc wrote:
| Some testing...
|
| Some testing...Some testing...Some testing...Some testing...
| swiley wrote:
| It needs a less portable compiler and uses MIT instead of GPL.
|
| Otherwise that would be pretty exciting.
| mschuster91 wrote:
| > Or, uglier, overriding the files with the new ones.
|
| There's a less ugly solution - Debian/Ubuntu have the battle-
| hardened "alternatives" system:
| https://wiki.debian.org/DebianAlternatives
| andrewshadura wrote:
| Not quite, alternatives are co-operative, since all packages
| need to opt in.
|
| In this case, diversions would be more suitable, but in this
| case probably not necessary, since at the moment it's only for
| testing.
| tyrion wrote:
| The goal of modernizing coreutils is great, and doing it in rust
| is even better. However, it makes me very sad that this is
| licensed under the MIT licence.
|
| Being licensed under the GPL is an essential part of the GNU
| project and their philosophy. Of course everyone is free to do as
| they please. However, IMHO, if one appreciates the GNU project
| and the ideals it stands for, then, maybe, it could be preferable
| not to rewrite parts of it under weaker licences that go directly
| against their mission!
| stjohnswarts wrote:
| I think people should be able to license under whatever they
| choose to license under. Diversity is good for the most part.
| mouldysammich wrote:
| Yeah I really agree here. Especially when it comes to coreutil
| implementations. I've seen multiple implemented under MIT/BSD
| and it always makes me a little sad.
|
| I have access to an incredibly high quality easy to use router
| firmware in the form of OpenWrt by virtue of GPLd coreutils. I
| wish FOSS developers weren't so afraid of it these days
| coldtea wrote:
| > _Being licensed under the GPL is an essential part of the GNU
| project and their philosophy._
|
| Having "won the desktop from Windows" and "finished a next-gen
| OS (Hurd)" was also part of that plan, but it didn't really pan
| out, did it?
|
| I'm glad for MIT software since I can use it in more places I
| can use GPL (which companies often wont touch, in it's later
| license version).
| AlchemistCamp wrote:
| Do the places you work forbid the use of Linux or the GNU
| Coreutils?
|
| In what context are you using Rust Coreutils? Windows? MacOS?
| BSD?
| coldtea wrote:
| > _Do the places you work forbid the use of Linux or the
| GNU Coreutils?_
|
| They allow them. I had cases where clients forbade some GPL
| code (banks, etc) - worse with GPL3.
| wizzwizz4 wrote:
| They'll stop doing that if most code is GPLv3. If most
| code is GPLv3, we're all better off - but while people
| are still prepared to work around GPL-rejecting
| companies, those companies can still reject the GPL.
|
| It's the tragedy of the Commons.
| Master_Odin wrote:
| Alternatively, if more OSS was GPLv3, we might see
| companies more default to recreating the software in
| house and nobody really wins.
| ben_bai wrote:
| The GPL license usage is falling, and even Linux is GPLv2
| and rejects GPLv3. Some Linux code is dual licenses GPLv2
| and MIT.
|
| ApacheV2, MPL, MIT, ISC and BSD licenses are gaining in
| popularity.
| raegis wrote:
| Do they forbid _using_ the code, or incorporating the
| code into their own works? I hear of places forbidding
| the use of GPL code (even in-house) where they use
| proprietary software like Microsoft Word with no worries.
| ben_bai wrote:
| Because Microsoft doesn't demand you open source your
| plugins. GNU software _might_ force you to.
|
| Corporations choosing to not use GPL software mostly
| don't know enough about it, to know it's ok. So they ban
| it outright.
| teddyh wrote:
| > _GNU software might force you to._
|
| No, they might not. Just like Microsoft, they _might_
| demand that you don't release your plugins. _Addidionaly_
| , GNU software gives you the _alternative_ of releasing
| your plugins. But you don't have to, and they can't make
| you do it.
| robocat wrote:
| Your argument seems to imply that GNU was the progenitor of the
| utilities. A quick sample of the first 10 utilities from the
| first column of http://maizure.org/projects/decoded-gnu-
| coreutils/ on my machine showed 80% of the utilities were
| copied from existing sh utilities. Sure, GNU has added
| features, but there is nothing unseemly about creating new
| utilities using a different licence that are largely compatible
| - that is exactly how GNU coreutils came to life in the 90s.
|
| arch - GNU
|
| chgrp - Descended from chgrp introduced in Version 6 UNIX
| (1975)
|
| comm - Descended from comm in Version 2 UNIX (1972)
|
| dd - Descended from dd introduced in Version 5 UNIX (1974)
|
| du - Descended from du introduced in Version 1 UNIX (1971)
|
| factor - Descended from sort in Version 4 UNIX (1973)
|
| head - Descended from head included in System V (1985)
|
| join - Descended from join introduced in Version 7 UNIX (1979)
|
| ls - Spiritually linked with LISTF from CTSS (1963)
|
| mktemp - GNU
| anamexis wrote:
| Where does their argument imply that GNU was the progenitor
| of those utilites?
| augusto-moura wrote:
| I think he/she commented on the wrong parent comment, it
| makes a lot more sense about the current top comment
| vfsignal wrote:
| Agreed. It would also be good of this Rust project to state
| that all of the utils are original works and no one has peeked
| at the C sources. Perhaps the maintainers of the C versions
| should do the same in reverse.
| amenod wrote:
| > Perhaps the maintainers of the C versions should do the
| same in reverse.
|
| Nitpick: they don't need to, they can just attribute the Rust
| developers if needed. They can also just take MIT code and
| publish it under GPL (reverse is not true).
| flangert wrote:
| Taking MIT code and re-licensing it under the GPL will
| cause a huge uproar:
|
| http://undeadly.org/cgi?action=article&sid=20070913014315
|
| I'm not sure that is even legal.
| ben_bai wrote:
| It's doable if you can get OK from every single copyright
| owner of the source files.
|
| LLVM did it. Switching from "University of Illinois/NCSA"
| to "APL 2.0".
|
| But not without some big discussions.
| moocowtruck wrote:
| https://github.com/uutils/coreutils/search?q=gnu "This is the
| way it was implemented in GNU split." i dunno maybe there are
| cases where they copy more than behavior... but interesting
| to look through
| okl wrote:
| There are more examples, here is one: // a
| few global constants as used in the GNU implementation
| danudey wrote:
| It's pretty easy to ask someone "Hey, can you go take a
| look at GNU split and see how they handle this case?" and
| then implement your own clean-room solution afterwards.
|
| Alternately, to solve the problem yourself and mention it
| to someone and have them say "that's how GNU split does it,
| so why not?"
| eximius wrote:
| Isn't that, by definition, not a clean-room solution?
| moonchild wrote:
| 'Clean-room reverse engineering' refers to the practice
| of having one person look at disassembled source code,
| spec sheets, actual behaviour, etc. of some program;
| describe that behaviour to a second person; and having
| the second person implement the described behaviour.
|
| https://en.wikipedia.org/wiki/Chinese_wall#Reverse_engine
| eri...
| eximius wrote:
| Ah, that's interesting! I didn't know that was a
| sufficient insulation of IP.
| gregwebs wrote:
| Richard Stalman and others that believe the GPL is generally
| the best license are not against ever using the MIT license.
| Stalman has been willing to be pragmatic and support tactical
| choices of other licenses besides GPL when licensing.
|
| So it is worth having some deeper thoughts about what the
| implications are for different licenses of coreutils. When does
| using coreutils create a derivative work that requires GPL?
| mrob wrote:
| Tactical considerations exist, but even so, a copyleft
| license might have better long term results in promoting
| software Freedom. The FSF supported a non-copyleft license
| for Ogg Vorbis, because at the time it seemed reasonable that
| this was the best way to fight the (at the time) patent-
| encumbered MP3 format. But in practice it had little benefit:
|
| "However, we must recognize that this strategy did not
| succeed for Ogg Vorbis. Even after changing the copyright
| license to permit easy inclusion of that library code in
| proprietary applications, proprietary developers generally
| did not include it. The sacrifice made in the choice of
| license ultimately won us little."[0]
|
| [0] https://www.gnu.org/licenses/license-recommendations.html
| moonchild wrote:
| I don't know when that was written, but vorbis is used
| pervasively today, including in proprietary applications.
| oehtXRwMkIs wrote:
| I think this is more of a reflection of mainstream OSS culture
| in general. Rust development currently is heavily tied to
| proprietary services (e.g. GitHub required for contributions,
| issues, crates.io login, CI, etc., Discord and Slack
| communities rather than Matrix, Zulip) and likes to license
| things under MIT. I think the lack of awareness/care for the
| big picture of fighting for software freedom is simply not
| there just as it is missing from mainstream OSS culture.
| Because convenience and network effect are king.
| pdimitar wrote:
| You do have a point but in case you are saddened by this
| phenomena, let me just point out that we live in a VERY
| different age compared to when GNU coreutils were born.
| Nowadays you only get a few minutes -- or hours if you are
| lucky -- to answer a ton of fundamental questions like "where
| do we host code?" and "how do we communicate on this
| project?" or "how do we do CI/CD?" etc.
|
| The people in the past had all the time in the world to
| tinker and invent. Maybe I am mistaken though, past is
| usually looked through rose-tinted glasses right?
|
| But the fact remains: nowadays answering the above questions
| is beyond my pay grade: in fact it's beyond anyone's pay
| grade. Services like GitHub are deemed a commodity and
| questioning that status quo is a career danger.
|
| I really do wish we start over on most of the items you
| enumerated. But I am not paid to do it. In fact I am paid to
| quickly select tools and never invent any -- except when they
| solve a pressing business need and are specific enough for
| the organization; in that case it's not only okay but a
| requirement.
|
| Beyond anything else however, we practically have no choice.
| If I don't host a new company project on GitHub I'll
| eventually be fired and replaced with somebody who will.
| JulianMorrison wrote:
| In the case of coreutils, there is really no issue with
| presence or absence of GPL. cp is just cp and nobody is going
| to hang a proprietary extension off the side.
| eeZah7Ux wrote:
| > it could be preferable not to rewrite parts of it under
| weaker licences that go directly against their mission
|
| +1. It's very sad to see user freedom being thrown away as a
| goal and replaced by software that becomes unpaid labor for
| FAANGs. Especially since the SaaS takeover.
| imhoguy wrote:
| GPL does not protect against SaaS provider creating
| derivative as there is no further distribution of binary or
| source code. Only AGPL (and other more strict licenses)
| addresses SaaS derivatives issue.
| hntestacc wrote:
| some more testing...
|
| some more testing...some more testing...some more testing...
| est31 wrote:
| That's pretty cool. I believe that Rust is the future of systems
| programming. Has someone tried to use the Rust coreutils on Nix
| OS?
| sylvestre wrote:
| There is a package for nix:
| https://gitlab.tryp.io/nix/nixpkgs/-/tree/bodhi/pkgs/tools/m...
| (but not up to date)
| siraben wrote:
| I went to check[0] who was the maintainer for uutils-
| coreutils on Nixpkgs, and realized it was me! Thanks for the
| reminder, will push a PR update.
|
| [0] https://repology.org/project/uutils-coreutils/versions
| siraben wrote:
| Update has been pushed and merged, will show up in the
| binary caches within the next few days :)
| siraben wrote:
| They are in Nixpkgs, presumably it would just be a matter of
| replacing coreutils in the stdenv[0] with uutils, then
| attempting to build, say, Firefox with the new stdenv.
|
| [0]
| https://github.com/edolstra/nixpkgs/blob/master/pkgs/stdenv/...
| skore wrote:
| I just wish someone would RIIR Gnu make.
|
| I use it heavily and think it's extremely underappreciated, so
| instead of reinventing it, I would like to build on it. But -
| trying to extend the old C codebase is daunting. I'd even be
| happy with a reduced featureset that avoids the exotic stuff that
| is either outdated or no longer useful. The core syntax of a
| Makefile is just so close to perfect.
|
| (I wrote about some of this in the remake repo:
| https://github.com/rocky/remake/issues/114 )
| boris wrote:
| Allow me to suggest build2 (the build system part) as another
| alternative. You would probably be first interested in ad hoc
| recipes: https://build2.org/release/0.13.0.xhtml#adhoc-recipe
| skore wrote:
| Very VERY interesting!
|
| The one thing that I'm currently struggling to find
| information on is dynamic dependency handling (dyndep in
| ninja terms).
|
| Is that something that build2 covers as well? Any resource
| you could point me to?
| boris wrote:
| Dynamic dependencies are supported by build2 (this is, for
| example, how we handle auto-generated C/C++ headers) but
| it's not something that is exposed in ad hoc recipes that
| are written in Buildscript yet (but is exposed in those
| written in C++). Our plan is to start with the ability to
| "ingest" make dependency declarations since quite a few
| tools are able to produce those. Would something like this
| cover your use-cases?
| skore wrote:
| Very likely, yes. I'm generating my own makefiles
| anyways, so generating something that build2 could
| consume should be possible, too.
|
| Would it be appropriate to open a github issue for
| discussing this further? I would like to share some
| example for how my current setup is working and having
| the github syntax available would be helpful.
| boris wrote:
| Yes, please, that would be very helpful:
| https://github.com/build2/build2/issues
| skore wrote:
| "issue" filed. My apologies for the lack of brevity.
|
| https://github.com/build2/build2/issues/135
| layoutIfNeeded wrote:
| In case anyone would attempt this: please make quoting right.
| deknos wrote:
| https://github.com/taskctl/taskctl
| skore wrote:
| That looks very good! I wonder if I can make it fit my file-
| based approach.
|
| edit: Yeah, doesn't seem like it.
| JoshTriplett wrote:
| Someone is working on this, and could use help:
| https://github.com/michaelmelanson/fab-rs
|
| I would _love_ to see this completed to the point of passing
| the GNU make testsuite. Having make as a modular library would
| be wildly useful.
| coldtea wrote:
| > _The core syntax of a Makefile is just so close to perfect._
|
| That's about the last thing I would call it.
| stefan_ wrote:
| There is kati from Android:
|
| https://github.com/google/kati/blob/master/INTERNALS.md
|
| This takes your existing Makefile(s) and produces a ninja build
| file. Not sure if Android still uses it or it is all soong now.
| dataflow wrote:
| You had me until
|
| > The core syntax of a Makefile is just so close to perfect.
|
| I'd argue what is close to perfect is much of the underlying
| model/semantics, and what is terrible is very much the syntax.
| I've long wanted to make something similar to Make but simply
| with better syntax...
| rwmj wrote:
| I had a go at making an experimental tool with a better
| syntax, fixing a lot of the obvious problems (whitespace, $
| having a double meaning, etc).
| https://rwmj.wordpress.com/2020/01/14/goals-an-
| experimental-...
| dataflow wrote:
| Wow, cool! Thanks for sharing! I'll have to take a look
| when I get the chance.
| skore wrote:
| To me, the core of the syntax is: target:
| prereq | ooprereq recipe
|
| In my eyes, the most important thing when building something
| that is complex is the dependency graph and it makes sense to
| make the syntax for defining the graph as simple as possible.
| I think the make syntax just nails it and most of the other
| approaches I have seen so far add complexity without any
| benefit. In fact, most of the complexity they introduce seems
| to stem from confusion on the side of the developer being
| unable to simplify what they're trying to express.
|
| At the level of variable handling and so forth, make is
| slightly annoying but manageable.
|
| Anything beyond that - yeah, I'm with you, a lot of that is
| terrible.
| megous wrote:
| you might like ninja build then
| thechao wrote:
| I really like the idea of Ninja; I really want Ninja to
| be a single C(++) file that I can build with a trivial
| command line. I don't want a single C(++) file because of
| "elegance" so much that I want to ship Ninja as an inline
| dependency but now I've got two problems: I have to build
| my code _and_ I have to build Ninja. If Ninja was a
| single file: boom! Trivial build line.
|
| Other than that, I think Ninja is perfect.
| stormbrew wrote:
| You could probably post-process samurai (a rewrite of
| ninja into C) into a single-file:
| https://github.com/michaelforney/samurai
| skore wrote:
| I will check it out, thank you for the suggestion - what
| I glanced at seems very interesting.
| eru wrote:
| Shake would also be interesting to look at. It has much
| more sophisticated dependency graph handling.
|
| Gnu Make is almost pathetic.
|
| See eg http://simonmar.github.io/bib/papers/shake.pdf
| dahfizz wrote:
| This is their minimal example to compile a single C file:
|
| 1 module Main(main) where
|
| 2 import Development.Shake
|
| 3 import System.FilePath
|
| 4
|
| 5 main :: IO ()
|
| 6 main = shake shakeOptions $ do
|
| 7 want ["foo.o"]
|
| 8
|
| 9 "*.o" %> \out - do
|
| 10 let src = out -<.> "c"
|
| 11 need [src]
|
| 12 cmd "gcc -c" src "-o" out
|
| Maybe there is a need for super sophisticated graph
| handling, but for most use cases the way more complicated
| syntax of Shake is not a worthy tradeof.
| rkangel wrote:
| The syntax has one major insurmountable problem - it uses
| space as a separator and so doesn't allow spaces in target
| names and therefore doesn't allow spaces in filenames.
|
| Yes, there are workarounds for some situations, but none of
| them work in all cases and no-one ever applies them
| consistently.
| dahfizz wrote:
| That isn't a "major insurmountable" problem. Its a minor
| annoyance - just don't have spaces in your file names.
| rkangel wrote:
| Or in your directory names both in your project, or in
| any parent folder. There's absolutely no reason why
| Windows shouldn't be able to call the folder "My
| Documents" but it broke a whole generation of builds that
| people tried to do in their home directory.
|
| Spaces in filenames is a perfectly reasonable thing to
| want to do. They are supported on all three major
| platforms, but make just throws up it's hands and doesn't
| care. Yes, it is workable around, but it would be nice if
| it wasn't.
|
| It's an annoyance with make overall, but if we're talking
| about the syntax specifically then yes I would call that
| a "major insurmountable problem".
| steveklabnik wrote:
| One time, I had a work-issued laptop where they filled
| out my user account. They used spaces in my username.
| This meant that I had a space in every single path.
|
| It caused so many issues. But mostly with developer
| tooling, like Make, that assumes these sorts of things.
| Most programs worked just fine.
| a1369209993 wrote:
| > One time, I had a work-issued laptop [which I did not
| have root on].
|
| Well there's your problem right there. That kind of shit
| needs to stop, and making companies that perpetrate it
| less able to develop software (and thus less profitable)
| is one of the only things people who don't directly
| interact with such a company can do to discourage it.
| (Not that that's in any way a _good_ strategy, but to the
| tiny extent it has any bearing on makefile syntax in the
| first place, it 's a argument _against_ supporting spaces
| in filenames.)
| Ericson2314 wrote:
| So you want ninja?
| cat199 wrote:
| > I've long wanted to make something similar to Make but
| simply with better syntax...
|
| If you haven't, I'd suggest having a look at pmake (sometimes
| packaged as 'bmake' since it is the base 'make'
| implementation on BSDs) - the core 'make' portions are still
| the same (like gmake extends core 'make' as well), but the
| more script-like 'dynamic' parts are much nicer than gnumake
| in my opinion. It also supports the notion of 'makefile
| libraries' which are directories containing make snippets
| which can be #include's into other client projects.
|
| freebsd examples:
|
| manual: https://www.freebsd.org/cgi/man.cgi?query=make&apropo
| s=0&sek...
|
| makefile library used by the system to build itself (good
| examples): https://cgit.freebsd.org/src/tree/share/mk
| dataflow wrote:
| Looks interesting, thanks for sharing!
| nicoburns wrote:
| Have you seen Just? https://github.com/casey/just
|
| It's make-like but supposedly improves on some of the arcane
| aspects of make (I can't judge how successful it is at that as
| I've never used make in anger).
| bluejekyll wrote:
| One of the reasons I know I still rely on Make instead of
| Just, is about how available make is, and pretty much any
| tool (be it a new implementation of make in Rust or Just)
| will have to deal with the fact that make is already a
| default on so many systems.
|
| That makes make a great entry point for any project. Does
| anyone have a general suggestion about how to work around
| this? The goal being, sync a project and not need to install
| anything to get going. One thing I do sometimes is to use
| make as the entry point, and it has an init target to install
| the necessary tools to get going (depends on who I know to be
| the target audience).
| andoriyu wrote:
| yeah, until you meet the system with different make, and
| now you have to install GNU/Make.
| berkes wrote:
| This sounds like what I do: Makefile with distinct steps:
| install, build, test, lint, deploy. Those steps call
| underlying tools, depending on the platform, framework,
| language.
|
| E.g. in one project it will install rbenv, bundle, npm,
| yarn etc. Call Rake, npx, that odd docker wrapper, etc. Or
| deploy with git in one project and through capistrano or
| ansible in another.
|
| As a dev, in daily mode, all you need is 'make lint && make
| test && make deploy'. All the underlying tools and their
| intricacies are only needed when it fails or when you want
| to change stuff.
| bluejekyll wrote:
| Yeah. Exactly. I do this when I think there will be users
| who don't already have all the dev tools installed, and I
| don't want them to have to download and install
| everything.
| candied_scarf wrote:
| i wont use this because the author sucks up all justfiles
| files with no way for a user to opt out and isnt fully clear
| on this. use it on some private or company thing that
| accidentally goes public, oops he has your file now in his
| repo.
|
| if it was opt-in to him monitoring i might be more open.
| maybe i should fork and change the filename to keep this from
| happening
| cytzol wrote:
| I love Just, but I love it _because_ it 's not a build system
| -- I already have a build system, I just need a standard
| "entry point" to all my projects so each command a user could
| use to interact with the _actual_ build system is in the same
| place for each project. I wouldn 't build, say, a C project
| with Just and Just alone, where it tracks the dependencies
| between files and only rebuilds what's necessary.
|
| So while it's replaced one use of Make for me, I can't
| rightly call it a Make replacement.
| nicoburns wrote:
| Would you build a C project with make? One might argue that
| C could do with a better build system like other languages
| have. I don't use C much, but I hear that these exist.
| skore wrote:
| I've seen it before but discarded it because since my
| workflow is heavily file-based, I DO need makes capacity as a
| build system, not just a task runner. Will check it out to
| see whether that changed.
|
| edit: Yeah, I can't see how I can make it work for a file-
| based approach.
| jamwaffles wrote:
| We use it a bit for https://github.com/embedded-
| graphics/embedded-graphics and I have to say I love it. It's
| just the right balance between configuration
| variables/constants and "a folder full of bash scripts". I
| highly recommend giving Just a try.
| matheusmoreira wrote:
| Yeah, GNU Make is actually a functional programming language.
| When I discovered this, I almost reinvented parts of autoconf
| in GNU Make. It's like trying to write a complex project with C
| preprocessor macros! It was a lot of fun.
|
| GNU Make is hurt by its minimalism. Doing anything interesting
| in pure GNU Make is a herculean effort. Even something simple
| like recursing into directories in order to build up a list of
| source files is extremely hard. Most people don't even try,
| they would rather keep their source code tree flat than deal
| with GNU Make. I wanted to support any directory structure so I
| implemented a simple version of find as a pure GNU Make
| function!
|
| At the same time, a reduced feature set would be nice. GNU Make
| ships with a ton of old rules enabled by default. The no-
| builtin-rules and no-builtin-variables options let you disable
| this stuff. Makes it a lot easier to understand the print-data-
| base output.
|
| > The core syntax of a Makefile is just so close to perfect.
|
| It's very simple syntax but it has its pain points. Significant
| spaces effectively rules out spaces in file names. It also
| makes it much harder to format custom functions.
|
| Speaking of functions, why can't we call user-defined functions
| directly? We're forced to use the call function with the custom
| function's name as parameter. Things quickly get out of hand
| when you're building new functions on top of existing ones. I
| actually looked up the GNU Make source code, I remember
| comments and a discussion about this... It was possible to do
| it but they didn't want to because then they'd have to think
| about users when introducing new built-in functions. Oh well...
| moonchild wrote:
| > something simple like recursing into directories in order
| to build up a list of source files is extremely hard
|
| What's wrong with shelling out to find?
| jancsika wrote:
| > tldr: Rust/coreutils ( https://github.com/uutils/coreutils/ )
| is now available in Debian,
|
| Is it just my version of Firefox, or does anybody else get a line
| wrapping justification algo that ends the first line with "Debia"
| and begins the next line with "n"?
|
| It appears hyperlinks are split opportunistically to match the
| line justification.
|
| Two things:
|
| * I have _never_ seen a single web page do this
|
| * Something smells _soooo_ right that the only page I 've ever
| seen with this weirdo readability regression involves the word
| "Debian." It's as if when I read the word "Debian" in a
| blog/article, I get this sixth sense shiver that _something_ soon
| will be broken with a default setting because "nobody ever said
| you couldn't do it the other way."
|
| Edit: it's as if this choice were made specifically to keep me
| from clicking the FF Reader button. Because now I'm scanning over
| the entire document counting the number of words in hyperlinks
| that are broken across lines-- I see 8, which includes another
| instance of "Debian"-- this time it's broken into "D" and
| "ebian."
|
| Edit2: OMG the text doesn't wrap to remain in the viewport when I
| zoom in. I successfully see "Debia" at the end of the first line
| _every time_ , and if I zoom in far enough I can enter horizontal
| scrollbar readability hell.
| gpm wrote:
| I don't know why, but this is in the css, it appears to be
| intentional a { word-break:
| break-all; }
| andrewshadura wrote:
| This development is great news for automotive, since modern
| coreutils can't be used there because of the anti-tivoisation
| clause conflicting with government regulations, which forces
| vendors to use old unsupported versions.
| m45t3r wrote:
| Wouldn't busybox (GPLv2) or toybox (BSD) be a better choice
| than an very incomplete Rust rewrite?
|
| To start, both of them were write with embedded applications in
| mind, busybox is feature complete since distros like Alpine and
| OpenWRT use it (not sure about Toybox, but I know Android ships
| it).
|
| BTW, both of them cover way more tooling than coreutils too.
| Busybox for example have a primitive init system and a vi/vim
| clone. You can also of course disable them during build time.
| deknos wrote:
| well, then i hope the new rust tools will do this too.
| jgilias wrote:
| I have to deal with government regulated certification in a
| different industry. In our case there's a hard requirement to
| make it impossible for the end-user, which is some business, to
| run modified software on the devices. Reason being, the devices
| carry out financial transactions and the possibility of running
| modified software on them would potentially enable fraud.
| Basically, the whole point of certification in this case is to
| verify that the software doesn't have provisions for doing
| anything fraudulent and then making sure it stays that way. So
| GPLv3 anti-tivoization requirements and the certification
| requirements are mutually exclusive. I imagine it's similar in
| the automotive industry for safety reasons.
| mschuster91 wrote:
| > In our case there's a hard requirement to make it
| impossible for the end-user, which is some business, to run
| modified software on the devices.
|
| I guess you're working on credit card terminals?
|
| Anyway: why not have a secure hardware element / TPM that has
| a GPIO with a pull-up resistor that can be queried by the
| device? Then, have the bootloader check as part of the boot
| if the TPM attests with a digital signature that the GPIO is
| still high, and the application also regularly checking the
| TPM? Or an e-fuse similar to Samsung's Knox Guard?
|
| That way, a user can remove the pullup resistor (and thus, as
| it's a hardware modification, has to break the seal of the
| device) to "unlock" custom firmware loading to fulfill the
| GPL requirement, but at the same time your application can be
| reasonably certain at run-time the device hasn't been
| tampered with.
| jgilias wrote:
| In our case tampering with the sealed parts of the hardware
| by the user would be illegal. I'm not sure if providing an
| illegal method to load the firmware fulfills the GPLv3
| requirements.
|
| EDIT: So, I don't think this is actually a technical
| question. It's a legal question that boils down to the fact
| that a premise and its inverse can not both be true at the
| same time.
| varajelle wrote:
| > make it impossible for the end-user, which is some
| business, to run modified software on the devices.
|
| Looks like that device might not be covered by the
| tiviozation close since it might not be an user product as
| defined in the GPL
|
| > A "User Product" is either (1) a "consumer product", which
| means any tangible personal property which is normally used
| for personal, family, or household purposes, or (2) anything
| designed or sold for incorporation into a dwelling.
|
| If you sell to business, that should be alright.
|
| Also, it is fine to make it impossible to modify, if you also
| don't have this possibility:
|
| > this requirement does not apply if neither you nor any
| third party retains the ability to install modified object
| code on the User Product
| jgilias wrote:
| Thanks! This is a very useful clarification. I wonder how
| the last paragraph relates to automotive. I guess they
| still want to be able to do updates.
| enriquto wrote:
| This is a sad state of affairs. The world does not become a
| better place by rewriting working copylefted code into
| permissive licenses. Especially when the main goal of the
| rewrite is to ease corporate "theft" of these codes. We are
| walking several steps backwards with these rewrites (as much as
| I love the concept of rewriting common tools in a safer
| language).
|
| EDIT: I'm actually outraged by this. I _want_ to love the rust
| rewrite of coreutils, but I don 't understand why they had to
| taint this beautiful work with stupid license politics.
| flomo wrote:
| Coreutils is mostly UNIX/POSIX standard utilities which have
| been implemented under many closed and open source licenses
| for decades now. This is not where you build the GNU/Moat.
| scaladev wrote:
| But GNU coreutils are ahead of all proprietary alternatives
| in both speed and features, are they not? I've personally
| not had the opportunity to work on AIX, or Solaris, or
| whatever (there are no machines like that where I live),
| but that's what I always read: "we got a new AIX/HP-
| UX/Solaris/whatever server and the first thing I did was to
| build GNU coreutils and replace the stock crap with it".
| stonemetal12 wrote:
| >...first thing I did...
|
| How do you know if the platform tools are faster\slower
| more featureful or less if the first thing you did is
| replace them?
|
| GNU coreutils won on portability of knowledge, no need to
| learn how the platform's tools work when I can just roll
| with GNU. Which is great when you are dealing with
| multiple different varieties of Unix that are all
| slightly different. In general the base os tools where
| like all software that competes in the same space, it did
| somethings better and somethings worse.
| jka wrote:
| It's disappointing but it's not worth assuming any malice or
| continuing flamewars. There's been some discussion on the
| project's GitHub page in the past about licensing.
|
| What's worthwhile is to share and talk about why the license
| selection may result in proprietary and bloated applications
| built upon the stack, and the resulting inefficiencies and
| security issues that could occur years in the future as a
| result. Perhaps it'd even be possible to speculate about what
| incentives create and support the conditions for those.
|
| If enough people agree and understand the situation, perhaps
| we'll see change. If they don't, we'll continue on for a
| number of years and maybe those hypothetical problems will
| occur, or maybe they won't and the concerns will have been
| unjustified.
| jka wrote:
| (and FWIW: I also think it's excellent to see core
| utilities rewritten in rust)
| Thiez wrote:
| Are you also outraged by the existence of clang, as it has a
| permissive license and competes with GCC?
|
| From where I'm standing it looks like _you_ are the one
| introducing the stupid license politics. The Rust Coreutils
| developers have chosen a license that they like, and you are
| the one making arguments about how terrible this is for the
| world and how they are tainting the GNU Coreutils.
| konjin wrote:
| >Are you also outraged by the existence of clang, as it has
| a permissive license and competes with GCC?
|
| Yes.
| dralley wrote:
| Maybe you should be outraged by GCC's deliberate
| sandbagging and less deliberate stagnation that allowed
| LLVM to swoop in and be a legitimately more useful (if
| not "better") compiler.
|
| Stallman himself was largely responsible for GCC not
| exposing intermediate representation for analysis.
|
| https://www.reddit.com/r/programming/comments/2rtumb/curr
| ent...
|
| There is a world in which rustc might have been built on
| gcc instead - a world where GCC was interested in being a
| platfom rather than a Maginot line trying to prevent any
| possible theoretical use by proprietary software even if
| it means preventing free software from doing the same.
| konjin wrote:
| GCC is a great C compiler and most of us are happy to
| keep it that way. The constant drumbeat of 'different
| different different' is not something I'm interested in.
| That I can compile C code written before I was born is
| more than I can ask for. In the same period a dozen safe
| languages have come and gone with no one even thinking
| about them any more.
| littlestymaar wrote:
| > that allowed LLVM to swoop in and be a legitimately
| more useful (if not "better") compiler.
|
| Interestingly enough, if Stallman didn't miss the
| email[1], LLVM was intended by Latner to be given to the
| FSF[2]. So yes RMS did commit a management mistake, but
| it's not the one you think.
|
| > The patch I'm working on is GPL licensed and copyright
| will be assigned to the FSF under the standard Apple
| copyright assignment. Initially, I intend to link the
| LLVM libraries in from the existing LLVM distribution,
| mainly to simplify my work. This code is licensed under a
| BSD-like license [8], and LLVM itself will not initially
| be assigned to the FSF. _If people are seriously in favor
| of LLVM being a long-term part of GCC, I personally
| believe that the LLVM community would agree to assign the
| copyright of LLVM itself to the FSF and we can work
| through these details_.
|
| [1] https://lists.gnu.org/archive/html/emacs-
| devel/2015-02/msg00... [2] https://gcc.gnu.org/legacy-
| ml/gcc/2005-11/msg00888.html
| dralley wrote:
| Yes but didn't he later submit the patches and they were
| rejected anyways?
|
| https://lists.gnu.org/archive/html/emacs-
| devel/2015-02/msg00...
| enriquto wrote:
| > Stallman himself was largely responsible for GCC not
| exposing intermediate representation for analysis.
|
| This is wildly offtopic for this thread, but it is not
| clear that Stallman's stance on this issue was an error.
| The wide existence of LLVM-based compilers for
| proprietary architectures is, for many of us, a tragedy
| that Stallman correctly anticipated. His insightful and
| courageous steering of the GCC development was a crucial
| step in avoiding this problem for GCC.
| dralley wrote:
| At the expense of forcing people that would love to use
| GCC to produce open-source tools and research, into using
| LLVM.
|
| See the email thread I linked for several such examples.
|
| https://lists.gnu.org/archive/html/emacs-
| devel/2015-01/msg00...
|
| https://lists.gnu.org/archive/html/emacs-
| devel/2015-01/msg00...
|
| https://lists.gnu.org/archive/html/emacs-
| devel/2015-01/msg00...
|
| https://lists.gnu.org/archive/html/emacs-
| devel/2015-02/msg00...
|
| https://lists.gnu.org/archive/html/emacs-
| devel/2015-02/msg00...
|
| Most compiler research and tooling development now
| happens in the LLVM ecosystem, precisely because they
| dragged their feet for so long.
| dagmx wrote:
| Isn't it also precluding several non proprietary
| compilers as well though? In turn making LLVM more
| popular and making it even more appealing to propriety
| compiler authors.
| mst wrote:
| I definitely believe that for the first N years of gcc
| the policy was absolutely necessary to drag vendors into
| actually contributing back.
|
| Whether that's _still_ true in terms of the trade-offs
| being worth it to maintain the policy in the current day,
| and if it 's no longer true what value of N accurately
| describes when that changed, is something that I think
| people can reasonably disagree about.
|
| (my extremely boring take being "there's so many
| counterfactuals here I'm really not sure")
| enriquto wrote:
| > Are you also outraged by the existence of clang
|
| If clang wast named "Rust GNU Compiler Collection rewrite"
| I would certainly be!
| layoutIfNeeded wrote:
| >I don't understand why they had to taint this beautiful work
| with stupid license politics
|
| It's pretty simple reasoning: Rust evangelists want to
| maximize Rust usage. Period. Replacing GPL-licensed tools
| with alternatives written in Rust is an easy way to bootstrap
| this. Compare with: paperclip maximizer
| (https://www.lesswrong.com/tag/paperclip-maximizer)
| q3k wrote:
| What is the conflict between GPL3's anti-tivoisation clause and
| government regulations?
| andrewshadura wrote:
| Oftentimes they require vendors to only allow certified
| software to be installed for safety or compliance reasons.
| bestouff wrote:
| ... and how is GPLv3 incompatible with certification ? I
| don't get it.
| andrewshadura wrote:
| Because it doesn't allow users to install software
| modified by them, which is a requirement in GPLv3.
| littlestymaar wrote:
| And then the user loses the certification, as you said in
| another reply[1] the "user" you're talking about are
| other companies, so I don't see why they would decide to
| break the certification they got from their supplier just
| for fun.
|
| Edit since the parent was in fact speaking about the end-
| user, which I misunderstood: I don't see the problem
| either. The manufacturer has no obligation to prevent the
| end user from updating his car's software. There is no
| locks that prevents the car owner to just disable his
| airbag[2], or remove the safety belt. It's illegal to do
| so in most countries, and if the user do do and injure
| himself or somebody else because of that modification,
| they are on their own. I don't think it should be any
| different for software actually.
|
| [1]: https://news.ycombinator.com/item?id=26397176 [2]
| Edit: in fact, this is a bad example, because you need to
| be able to disable the airbag to put an infant car seat
| next to the driver.
| detaro wrote:
| No, the user that matters for GPL is the car owner. The
| point in the linked comment is that given how the market
| works, there is little incentive to work towards changing
| regulation (or even just interpretation of or belief
| about the regulation, I have clue if there are actually
| countries where this is impossible, but know for sure
| it's a widespread belief in the industry), because the
| company applying for certification is not the one getting
| annoyed by having to avoid GPLv3.
| andrewshadura wrote:
| I obviously meant end users in this case.
|
| The chain of software delivery often looks like this:
|
| Small subcontractor delivers parts of system- big company
| provides ready to use solution - hardware vendor uses the
| solution and gets their devices certified - end user uses
| the final product
|
| In this case, the hardware vendor is mostly interested in
| having their devices work as intended. Everyone up the
| delivery chain has to meet their requirements in some way
| to basically get paid. That's not a position where it's
| easy to make demands regarding certifications, since the
| hardware vendor may just go to someone else.
| africanboy wrote:
| Not a lawyer, but I don't think that's true.
|
| GPLv3 says that manufacturers have to release all the
| information needed to run modified software on the
| device, it doesn't mean that there is one (and one only)
| certified version that can legally run on the device for
| safety reasons.
|
| GPLv3 in this case would force manufacturers to release
| the information so that the owner of the car could run
| modified software, but legally if you do it, you, the
| user, not the manufacturer, are violating the law.
|
| It's the same thing that happens with electronic
| blueprints, you can modify the HW, it will void the
| warranty if you do it.
| --------------------------------------------
|
| _Protecting Your Right to Tinker_
|
| _Tivoization is a dangerous attempt to curtail users '
| freedom: the right to modify your software will become
| meaningless if none of your computers let you do it.
| GPLv3 stops tivoization by requiring the distributor to
| provide you with whatever information or data is
| necessary to install modified software on the device.
| This may be as simple as a set of instructions, or it may
| include special data such as cryptographic keys or
| information about how to bypass an integrity check in the
| hardware. It will depend on how the hardware was designed
| --but no matter what information you need, you must be
| able to get it._
|
| _This requirement is limited in scope. Distributors are
| still allowed to use cryptographic keys for any purpose,
| and they 'll only be required to disclose a key if you
| need it to modify GPLed software on the device they gave
| you. The GNU Project itself uses GnuPG to prove the
| integrity of all the software on its FTP site, and
| measures like that are beneficial to users. GPLv3 does
| not stop people from using cryptography; we wouldn't want
| it to. It only stops people from taking away the rights
| that the license provides you--whether through patent
| law, technology, or any other means._
|
| https://www.gnu.org/licenses/quick-guide-gplv3.html
| detaro wrote:
| The key point is if certification is possible for a
| device that allows arbitrary software to be run or not.
| If it is, we have your scenario. If it isn't, it's not
| possible. I don't know if the former case is true for all
| countries, I certainly know that the market overall
| believes it's not, or not worth the hassle of arguing it
| with other companies and regulators.
| africanboy wrote:
| As I understand it the manufacturers need to release only
| the information, nowhere GPLv3 says that the manufacturer
| should make the process of running custom software easy
| or economically viable, just that the information should
| be available.
|
| But as I've said I'm no law expert and I wouldn't put my
| hand on fire about it.
| matkoniecz wrote:
| Why certifying specific version of software would
| conflict with GPLv3?
| henearkr wrote:
| What kind of obstacles to the certification process do GPL3
| softs typically encounter?
| andrewshadura wrote:
| You as a user can't get your modified software certified,
| hence you cannot install it, your GPLv3 rights are
| violated.
| henearkr wrote:
| I think that's a problem of interpretation more than a
| hard obstacle.
|
| In many projects there is the requirement of a fixed
| release for 3rd party dependencies, versions for which
| all tests have been checked to pass (this is what is done
| in NodeJS with packages.json). There is even a
| requirement of reproducible build sometimes (like with
| the ongoing project to reach full reproducibility in
| Debian builds).
|
| Wouldn't these fit the same thinking pattern as the
| requirements of certification of software for the
| industry?
|
| I'd love to hear RMS on this subject, maybe he would,
| too, say that the solution exists inside of GPL3 rather
| than outside of it.
| andrewshadura wrote:
| I'm sure there are ways to solve it, but they probably
| require both lawyering and developing technical
| solutions, which unfortunately the industry isn't much
| interested in doing, partly because there isn't enough
| pressure directed at them to change the course of things.
| Such pressure, in my understanding has to come from the
| regulators, but for that to happen they need to be
| convinced this is the right thing to do, and that isn't
| an easy task.
| gspr wrote:
| I'm confused. Surely you can install the modified
| software? It's just that now the local jurisdiction may
| not allow you to use your car on public roads. That's a
| bummer, but a political problem that exists between the
| user and his government, not a software problem between
| the user and the car manufacturer?
| bregma wrote:
| If the firmware is burned into ROMs at the manufacturing
| plant, how do you propose the car owner installs their
| modified version? Do we need to force all parts
| manufacturers to put ROM burners hardware and
| (replacable) firmware into every part, and OEMS to
| provide consolidated hardware interfaces to allow that,
| and car manufacturers to dedicate all the infrastructure
| required to allow access to all the OEMS who allow access
| to all the parts for all the various firmware updates?
|
| Sure. Starting price for a car would be several years
| salary but at least you could abide by GPLv3 licensing
| should any part manufacturer choose to use it.
| detaro wrote:
| > _If the firmware is burned into ROMs at the
| manufacturing plant,_
|
| then you do not need to make it modifiable under GPLv3.
| It explicitly states that.
|
| anyways, it's generally not, expect any Linux system in a
| car to run from flash memory.
| bregma wrote:
| Very little outside of the in-dash infotainment system
| runs on Linux in a car.
|
| I deal with manufacturers and OEMs putting software in
| cars for a living. I am not speculating, I am just
| reporting the reasons they tell me they will not accept
| any GPLv3 software for anything that gets loaded into
| their target devices.
| varajelle wrote:
| Either;
|
| 1. They are lying, and spreading FUD about regulations
| and licenses as an excuse to hide the real reason why
| they don't want to let user install their software. (Eg,
| because options costs extra and if it was free software
| one could install it for free)
|
| 2. Or, they are mistaken and don't understand the
| license. Maybe the cost to use alternative is less than
| the cost of figuring out.
|
| 3. They are right and that is a sad reality that the
| government give more power to companies than power to end
| users for things they owe.
|
| Since you mentioned something about the ROM which was
| clearly false, that could very well be option 1 or 2.
|
| I mean, why would the gouvernement want to restrict users
| to update the GPS software or the media player?
| detaro wrote:
| Not if it is a certification requirement that the system
| prevents the installation of modified software that's not
| certified, which is what's claimed.
| Blikkentrekker wrote:
| _GPLv3_ banned _tivoization_ , -- a practice whereby
| hardware certificates are used to prohibit the
| installation of modified software on hardware.
|
| Some governments require that automotive manufacturers
| implement these kinds of certificates to prohibit the
| installation of custom software in automotive
| applications for safety reasons, as they do not wish that
| users could install their own, potentially buggy
| software, at the potential cost of human lives.
|
| Older coreutils were licensed under _GPLv2_ , which has
| no such restriction.
| cue_the_strings wrote:
| You should finish that thought: and the manufacturers won't
| invest their money to certify something that's open source
| and can be used by everyone, including their competitors.
|
| If there was a will to do something about it, GPL3 wouldn't
| be in the way.
| CJefferson wrote:
| No, GPLv3 requires you let users replace the software --
| in some cases there are safety checks in the software
| which could be turned off if the user installed an
| alternative they built themselves.
|
| That isn't allowed -- you fail verification if you give
| users a documented way to disable required safety
| features.
| andrewshadura wrote:
| No, manufacturers want to use free software, but they are
| forced to avoid GPLv3 for this reason alone.
|
| Unfortunately, challenging the status quo is difficult
| when your customers are not the final users, but other
| companies: if you don't deliver them a free software
| firmware without GPLv3 components, someone else will, or
| someone else will deliver something that's completely
| proprietary.
| africanboy wrote:
| > No, manufacturers want to use free software, but they
| are forced to avoid GPLv3 for this reason alone.
|
| good.
|
| It means that GPLv3 works as intended.
|
| EDIT: as intended by the software authors, that chose
| freely GPLv3 as license, they were not forced to.
|
| I'm quite sure they knew what they were doing.
|
| If car manufacturers want to use GPLv3 software, they
| simply need to respect the license the author released
| their software under or rewrite the software.
| andrewshadura wrote:
| If by that you mean preventing free software from being
| more widely used, I agree it does. However, I disagree
| with that aim.
| andrey_utkin wrote:
| What's the point of wishing for existing FOSS artifacts
| (supposedly) "being more widely used" if it stops giving
| the foundational freedoms to the user?
| shawnz wrote:
| Because Stallman's specific conception of "foundational
| freedoms" doesn't actually necessarily align with what
| gives the best utility for users.
| andrey_utkin wrote:
| Free and open source software is not all about "best
| utility for users". It's about freedoms being valued
| extremely highly; people self-select for being or not
| being the consumers of it.
| shawnz wrote:
| What is a "freedom" is highly subjective. For example the
| GPL doesn't enable the "freedom" to create proprietary
| derivative products or tivoized products. Why is that
| better than the "freedom" to flash their firmware? It is
| just a matter of opinion.
| africanboy wrote:
| > If by that you mean preventing free software from being
| more widely used
|
| That's false.
|
| It simply prevents the Tivoization.
|
| GPLv3 was created exactly with the purpose of preventing
| free software from becoming a commodity.
|
| There's a cost involved when you use free software:
|
| - the software must stay free
|
| - if you include software licensed under a FOSS license,
| you have to adhere to the license terms
|
| simple as that.
|
| If the authors of software X or Y chose the GPLv3 as
| license I imagine they were completely aware and agreed
| to the terms of the license they used, including the
| limitations it enforces.
| eek04_ wrote:
| > If the authors of software X or Y chose the GPLv3 as
| license I imagine they were completely aware and agreed
| to the terms of the license they used, including the
| limitations it enforces.
|
| That is certainly not universally true. It is common, but
| not universally true.
|
| And we should assume that the authors were NOT aware of
| real ecosystem implications of licenses - because nobody
| understands these in detail. I've been trying to
| understand them since the mid 1990s when I started
| contributing in a BSD environment, and I won't say that I
| properly understand them. I understand _parts_ of them,
| but I don 't understand all of them.
| zokula wrote:
| It stops being free software after being widely because
| it's not free software anymore.
| deknos wrote:
| > challenging the status quo is difficult
|
| and always is. and people like you are part of the
| problem, since you want to enable them in their ways.
|
| Why? why cannot we just enable companies to behave mor
| ethically?
| andrewshadura wrote:
| > Why? why cannot we just enable companies to behave mor
| ethically?
|
| Please read the rest of the sentence, it answers the
| question.
|
| > people like you are part of the problem
|
| Thank you for turning the licensing question into ad
| hominem.
| bottled_poe wrote:
| I think you are both wrong. The purpose of a
| certification is to mitigate risk. If the licensor of the
| software has no money, there is no one to sue, and the
| risk is not mitigated. I'm not advocating this mentality.
| drran wrote:
| Manufacturers want to use free as beer software, but want
| to avoid expenses required to upkeep free software, or
| share these expenses with competitors.
|
| As developer of free software, I don't care about those,
| who don't care about me.
| Blikkentrekker wrote:
| Some governments require that one cannot install one's own
| software on automotives for safety reasons?
|
| That is quite interesting, but since one can modify one's car
| to begin with to make it unsafe, it is also rather futile.
|
| Rather, a sensible system would be that after such
| modifications, a car would have to pass inspection again to be
| deemed road-worthy. -- one may change the software, but one
| must pay to have it certified again ere it be allowed on public
| roads, and if that not be an option, one can always simply
| drive it on private property only.
| andrewshadura wrote:
| I agree with your view, but as I mentioned in another thread,
| bringing the change is very difficult, especially when you're
| not in a position with sufficient power.
| littlestymaar wrote:
| > That is quite interesting, but since one can modify one's
| car to begin with to make it unsafe, it is also rather
| futile.
|
| In France, if you want to modify your car (in a significant
| way, not specified by the car manufacturer) you need to send
| your car to a specific administration were engineer will
| inspect your vehicle before you can get it a plate number.
| This is called [Passage aux
| Mines](<<https://fr.wikipedia.org/wiki/Passage_aux_Mines>>).
| You typically have to do it when you decide to repurpose a
| cargo van as a camping van.
|
| That would shock me if you had to do the same after updating
| the embedded software of your car.
| dividedbyzero wrote:
| In Germany, it's called a TuV-Inspektion by the Technische
| Uberwachungsverein. Even unmodified cars have to get
| certified in regular intervals to ensure they're still
| street-worthy. Of course you may still tinker with your
| car, and do all you want with it on private land, and you
| can get quite a few modifications certified, but flashing
| some firmware image from Github on safety-critical systems
| definitely wouldn't get through.
| africanboy wrote:
| Same in Italy, but that is government enforcing their
| rules, you can modify the software on your car if you want
| to, it' s simply illegal.
|
| You can disable airbags, ABS and every other electronic
| circuit in the car and it's all explained in the car
| manual.
|
| There might be a good reason to do it (for example the
| airbag is malfunctioning)
|
| It has nothing to do with the terms of GPLv3
| Lev1a wrote:
| > (for example the airbag is malfunctioning)
|
| Then the car/vehicle is no longer road-worthy/road-safe
| and has to be repaired before going on public roads
| again. If the airbag isn't e.g. discovered in a repair
| shop, during an inspection or any other place where the
| car can be fixed without rejoining the public roads it
| has to be safely towed/transported to a place where it
| CAN be fixed.
|
| And incidentally it's not a problem of "muh car == muh
| freedom".
|
| If you want to drive a vehicle not safe by the standards
| everyone has to adhere to you're free to do that on
| private property.
|
| If you want to drive a vehicle on public roads where
| probably nobody knows about anything stupid you've done
| to the vehicle, i.e. a 1-2+ tonne lump of metal, glass
| and plastic probably with some sharp points/edges moving
| at a significant speed quite close to other people
| without any protection whatsoever. It's just horrifying
| accidents waiting to happen.
|
| Let's take the example of someone fucking around with the
| airbags in the car. If they're not known good, they might
| as well go off at any moment during the drive, possibly
| incapacitating the driver while the car is moving at
| normal road speeds, making the car veer around wildly and
| generally being an extreme hazard. There is a reason cars
| have to certified and adhere to standards of road-
| safety/road-worthiness.
|
| P.S. When a topic like this comes up I always have to
| think back to my father when e.g. a commercial/TV-
| programm about super-/hyper-cars came on. He almost
| always said "Dafur brauchte man eigentlich einen
| Waffenschein." ("One SHOULD need a weapons license for
| that thing.") in the sense that in our country (Germany)
| prospective gun owners need a license which IIRC requires
| amongst other things a psychological
| examination/certificate to ensure that no irresponsible,
| no mentally-ill, no mentally-challenged etc. people get
| the license to own guns. MEANING driving around on public
| roads with something that's essentially a road-going mix
| of a missile and a door wedge should probably something
| like an idiot-test. I mean it's already standard practice
| to deny RENTALS of cars above a certain power threshold
| to people under something like 23 or 25.
| littlestymaar wrote:
| FYI: you have the right to disable the airbags in your
| car, and it fact it's needed to put an infant seat in.
| Lev1a wrote:
| Sure, on the front PASSENGER seat and not permanently.
| Screwing around with software control etc. of airbags and
| other security features is (I hope) very much illegal and
| a reason for taking away the operating license of that
| particular car.
| GuB-42 wrote:
| Is it a hard requirement or is it about vendors being
| overzealous because it also fit their interest?
|
| I know that the law requires some radio equipment to make it
| hard for users to modify so that they can't mess with some
| parts of the spectrum.
|
| But for cars, the responsibility is typically of the user, that
| is, if you modify your car in a way that is not what the
| manufacturer certified, you can but you have to do by the rules
| or face the consequences, the manufacturer is not responsible.
|
| But by locking down software hard, it is a way for
| manufacturers to make sure that they can't be held responsible
| and the side effect of locking down features is a nice bonus.
| ognarb wrote:
| I find it a bit strange that the rewrite is licensed under MIT.
| Are they doing a clean room reimplement or are they rewriting
| the core utils by looking at the source code?
| arp242 wrote:
| In my experience looking at the source code of programs
| written in a different language is rarely helpful. It's only
| useful to glean some general ideas or algorithms. But
| rewriting with one eye on some C code? Rarely helpful and
| usually counterproductive.
| justinclift wrote:
| > It's only useful to glean some general ideas or
| algorithms.
|
| Sure, but they'll also have an easier time of ensuring
| they're "pretty close" to fully reproducing the original.
|
| At least, in comparison to having to reverse something
| black box. :)
| nindalf wrote:
| They're trying to be compliant to the interface of the
| original, not the implementation. Now whether the
| interface itself can be copyrighted ... the SCOTUS will
| weigh in on that soon (https://en.wikipedia.org/wiki/Goog
| le_LLC_v._Oracle_America,_...)
| curt15 wrote:
| If the SCOTUS says yes, would the original GNU coreutils
| even retain their GPL license given that they
| reimplemented pre-existing, non-GPL utilities?
| codys wrote:
| Not necessarily. The non-GPL utilities the GNU coreutils
| are based on are presumably licensed as MIT, ISC, Public
| Domain, or similar. If that is the case, those licensing
| regimes don't forbid adding extra restrictions, so adding
| the restrictions of GPL would be permitted, and the new
| derived (again, assuming interfaces are copyrightable)
| work would operate as being essentially GPL licensed.
| Things get murky as there are proprietary utilities too
| that coreutils are probably based on, but then there are
| also specifications of behavior (SUS, Posix) which might
| or might not sever the "derivation path" of the work, or
| might introduce further licensing constraints.
| nojito wrote:
| That case has no bearing on this because they aren't
| deciding if "interfaces can be copyrighted"
| throwaway17_17 wrote:
| Do you view APIs as something other than an interface? A
| ruling from the Supreme Court may not reach the ultimate
| issue, but if it does, what assumptions would you make
| concerning the difference between an API and a program's
| interface with the user?
| nindalf wrote:
| An Application Programming Interface is ... an interface.
| As this very link points out, there are many programs
| that use coreutils via this interface.
|
| Rust coreutils replacing GNU coreutils in these programs
| by implementing the same interface is the same issue.
|
| I'm not sure if it's possible to spell it out clearer
| than this.
| abstractbeliefs wrote:
| I understand your point of view, but it's important to remember
| the end outcome:
|
| Users will be forced to use outdated, buggy software, or
| vendors will write minimal closed implementations covering only
| what they need, without the benefit of the accumulated good
| work in foss (bugs, lack of features, etc)
| kungito wrote:
| These are edge cases. It's hard to tell how many people give
| in to the rules of the license while before they wouldn't
| Blikkentrekker wrote:
| I'm more so interested in performance and size comparisons.
|
| Most different implementations of the coreutils have different
| goals in mind. _GNU_ 's implementation, as is common of _GNU_ is
| to treat memory for performance, and _busybox_ 's attempts to
| achieve the converse. -- so what is the purpose of this one?
| simply being written in _Rust_?
|
| How does it compare to the established ones?
| sylvestre wrote:
| Significantly bigger than the C version.
|
| Rust: 73M source:
| https://packages.debian.org/experimental/rust-coreutils
|
| C: 17M https://packages.debian.org/unstable/coreutils
|
| Note that no size optimization that been done and there is a
| lot ways to do it in Rust https://github.com/johnthagen/min-
| sized-rust
|
| In term of performance, well, it depends on the binary. For
| example for cp, most of the time is spending copying the files.
| For the factor command, some work happened to make it faster
| https://github.com/uutils/coreutils/commits/master/src/uu/fa...
| (getting closer).
|
| Using Rust also opens some great capabilities like parallelism.
| But this makes sense only for complex commands. For example,
| doing a parallel "df" isn't super interesting (I tried it was
| too expensive just to start threads to do it).
|
| Anyway, performances should be a focus but only after
| correctness is implemented.
| gpm wrote:
| > For example for cp, most of the time is spending copying
| the files.
|
| As it turns out, cp can be made significantly faster by using
| (very) modern kernel APIs: https://wheybags.com/blog/wcp.html
| 20 512MiB files wcp 3.97s, 2579.59 MiB/s
| cp 8.44s, 1213.38 MiB/s rsync 17.26s, 593.33
| MiB/s
| arp242 wrote:
| You can use it as a single binary that's linked to the actual
| tool name, like busybox. That binary is just 9.6M (0.0.4 from
| their GitHub, Linux/amd64).
|
| The Debian version ships with separate binaries though (109
| of them), and is much larger. There is probably a lot of
| duplication as many tools use the same Rust libraries for
| flag parsing and whatnot which are compiled in the binaries.
| This is why those 73M compresses down to just 8M.
| alerighi wrote:
| First, symlinks are not really portable, for example they
| don't work on Windows, or in ancient UNIX systems.
|
| Also, that way every command executes with the same
| privileges because it's the same binary. And at least in
| Linux capabilities are bounded to a binary. So for utility
| that necessitate of special capabilities (e.g. ping,
| passwd, sudo, etc) this approach will not work.
|
| Yes, it works on busybox since everything is owned by root
| on the typical system so nobody cares about privileges.
|
| Finally, you need to load in memory every time you run a
| single binary all the thing needed by all the other
| binaries, even for the `true` binary. To me is stupid.
|
| The solution exists, and exists since ever, and it's
| encapsulating common functionalities in shared libraries,
| and have very small binaries linked to only the libraries
| they use. Unfortunately Rust choose not to support them for
| stupid reasons like problems that were resolved decades
| ago.
| _flux wrote:
| > For example, doing a parallel "df" isn't super interesting
|
| Actually I would enjoy that. Sometimes slow filesystems (e.g.
| stuck nfs) prevent the whole list from being displayed, or
| displayed only up to some point.
|
| If ctrl-c let me break out the command yet list all the
| collected data (preserving non-parallel order), it would be
| nice at times.
|
| Bonus: list skipped mountpoints to stderr..
| viraptor wrote:
| > For example for cp, most of the time is spending copying
| the files.
|
| There's a lot of ways to "copy a file" though. Finding the
| right buffer sizes, making concurrent read/write ops for
| different backend devices, and many other things can
| influence this. A good cp is not trivial.
| felixr wrote:
| From looking at the implementation of yes [0], I would say it
| follows closely the GNU coreutils implementation [1]. It
| allocates and re-uses a buffer to speed things up [2] and does
| not implement the old simple version [3] (or the even simpler
| OpenBSD [4])
|
| [0]:
| https://github.com/uutils/coreutils/blob/master/src/uu/yes/s...
|
| [1]:
| https://github.com/coreutils/coreutils/blob/master/src/yes.c
|
| [2]:
| http://github.com//coreutils/coreutils/commit/35217221c211f3...
|
| [3]:
| https://github.com/coreutils/coreutils/blob/ccbd1d7dc5189f46...
|
| [4]:
| https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c
| dastx wrote:
| I'd love to know too. It seems like it's nearly at feature
| parity but if performance is an issue, it's not really a
| replacement.
| Blikkentrekker wrote:
| _GNU_ coreutils are quite fast by the way.
|
| https://lists.freebsd.org/pipermail/freebsd-
| current/2010-Aug...
|
| This is an interesting article that outlines why _GNU_ grep
| outperforms almost any other grep quite significantly, at the
| cost of more memory.
| codethief wrote:
| That statement surprises me because my impression had
| always been that GNU grep is rather slow compared to other
| (more modern) tools, see e.g. the benchmarks on
| https://github.com/BurntSushi/ripgrep and
| https://blog.burntsushi.net/ripgrep/ .
| burntsushi wrote:
| Look at the dates on the material you're reading. That
| should resolve most of your confusion.
|
| Also, to be clear, in a strict apples to apples
| comparison, GNU grep and ripgrep will tend to have
| comparable performance. ripgrep does edge it out a bit in
| many cases, but it isn't always earth shattering.
| However, if you don't limit yourself to apples-to-apples
| comparisons and instead look at the user experience, then
| yes, ripgrep will likely be "a lot" faster. Primarily
| because of automatic parallelism and its "smart"
| filtering.
| duckerude wrote:
| GNU grep is so fast that it set the bar for later
| implementations.
|
| In your second link ripgrep is even marketed as having
| "the raw performance of GNU grep" (though it managed to
| exceed it).
| arp242 wrote:
| When that was written in 2010 it probably _was_ the
| fastest grep. It 's not like GNU grep is completely
| devoid of any performance optimisations.
| rcxdude wrote:
| It's a bit out of date (that email predates ripgrep).
| ripgrep goes even further in terms of skipping work, as
| well as focusing on parallelism.
| Freaky wrote:
| Having written a Rust wc implementation a few years ago
| (https://github.com/Freaky/cw), I had a look at theirs.
|
| It's pretty naive - a simple linewise read_until loop, a
| conditional to avoid word splitting and such if it's not
| needed, and for some reason it collects results into an array
| and prints when it's done rather than printing as it goes.
|
| It doesn't support --files0-from like GNU wc, so isn't a drop-
| in replacement from that perspective. It also has the sadly
| common Rust trope of only supporting filenames that are valid
| UTF-8.
|
| It doesn't seem overly slow considering its simplicity -
| usually trading blows with GNU and BSD wc. Perhaps the most
| glaring omission is the lack of a fast path for -c, which
| should reduce to a stat() call. Also unfortunate not to use the
| excellent bytecount crate to provide a very fast -l/m path.
|
| The read_until loop also makes its memory use unpredictable
| compared with other wc's. If you run it on /dev/zero it will
| try to eat your computer.
| tavianator wrote:
| GNU wc doesn't seem to have that -c fast path. I can tell
| because files in /proc report 0 for st_size even though
| they're not empty: $ wc -c
| /proc/self/cmdline 25 /proc/self/cmdline $
| stat -c '%s' /proc/self/cmdline 0
| Freaky wrote:
| It certainly has a fast path, it just mixes it up with
| lseek() for filesystems like /proc: https://github.com/core
| utils/coreutils/blob/9de1d153f82243ae...
|
| FreeBSD just uses fstat:
| https://github.com/freebsd/freebsd-
| src/blob/e4b8deb222278b2a...
| twic wrote:
| > uutils is an attempt at writing universal (as in cross-
| platform) CLI utilities in Rust. This repository is intended to
| aggregate GNU coreutils rewrites.
|
| > Why?
|
| > Many GNU, Linux and other utilities are useful, and obviously
| some effort has been spent in the past to port them to Windows.
| However, those projects are either old and abandoned, are
| hosted on CVS (which makes it more difficult for new
| contributors to contribute to them), are written in platform-
| specific C, or suffer from other issues.
|
| > Rust provides a good, platform-agnostic way of writing
| systems utilities that are easy to compile anywhere, and this
| is as good a way as any to try and learn it.
|
| [1] https://github.com/uutils/coreutils/
| steerablesafe wrote:
| > Rust provides a good, platform-agnostic way of writing
| systems utilities that are easy to compile anywhere, and this
| is as good a way as any to try and learn it.
|
| For some value of "anywhere".
| justinclift wrote:
| Also note the MIT License, which _could_ mean it gets picked
| up by places unhappy with GNU in general. Maybe future macOS?
| vletal wrote:
| I am so excited about rust slowly sneaking into the C territory.
| I am looking forward for the point in time when it is going to
| get to Linux kernel.
| joseluisq wrote:
| Another step forward to add Rust language support to the Linux
| kernel: https://github.com/Rust-for-Linux/linux
| simion314 wrote:
| My opinion is that Rust community focuses on a new modern
| kernel, and emulate Linux op top if needed(like Windows does
| it). In theory you would get a cleaner architecture and cleaner
| code base, no need to worry about regression and you might
| finish the task in 10 years. Open source applications could run
| directly on top of this kernel and Linux closed source apps
| could run on the emulation/translation layer.
| littlestymaar wrote:
| How about the drivers though...
|
| There's already a Rust OS[1] built on top of a brand new Rust
| kernel, but even if it gets all the traction possible, it
| won't have proper hardware support before several decades
| (see the current state of Linux, which is better than ever,
| but still far from perfect).
|
| [1]: https://www.redox-os.org/
| simion314 wrote:
| You will get some drivers at most, so maybe you could
| architect this drivers in such a way you could re-use part
| of teh code on the alternative kernels.
|
| What is missing is someone with big pockets to pay very
| competent engineers to work on such kernels full time and
| setup a decent test lab with different hardware. Hobbyist
| might take a long time to finish the job.
| pilif wrote:
| sooner than you might think: https://lwn.net/Articles/829858/
| joseluisq wrote:
| Yeah, Barriers to in-tree Rust [video] -
| https://www.youtube.com/watch?v=FFjV9f_Ub9o&t=2045s
| nindalf wrote:
| The biggest barrier is probably going to be the diff
| between architectures supported by Linux and those
| supported by Rust. It'll take several years for Linux to
| prune some of the lesser used architectures [1] and also
| years for Rust to add support on either LLVM [2] or via a
| WIP GCC frontend [3].
|
| So while most people involved seem to supportive, including
| Torvalds and GKH, it will take time for any non-driver code
| to be written in Rust.
|
| [1] - https://lore.kernel.org/lkml/CAK8P3a2VW8T+yYUG1pn1yR-
| 5eU4jJX...
|
| [2] - https://github.com/fishinabarrel/linux-kernel-module-
| rust/is...
|
| [3] - https://rust-gcc.github.io
| littlestymaar wrote:
| This is addressed in the questions in the aforementioned
| video[1] and it's actually not that much of a blocker at
| this point. There's plenty of arch-specific stuff in the
| kernel (drivers, but not only), so Rust can (and will) be
| used there. And there's no point waiting for complete
| architecture compatibility before starting to deploy
| _some_ Rust in the kernel.
|
| Of course it means that they aren't going to rewrite the
| core kernel logic in Rust tomorrow, but they wouldn't do
| it anyway even if GCC supported Rust.
|
| [1]: https://youtu.be/FFjV9f_Ub9o?t=2930
| wyldfire wrote:
| LLVM just landed an experimental m68k backend, this week
| IIRC. That's some progress, anyways.
| j1elo wrote:
| _sed_ seriously needs a Perl-regex mode... it 's stuck in the
| Basic/Extended Regex syntaxes, which feel prehistoric and don't
| support the syntax that you will learn in practically _any_
| modern regex tutorial.
|
| Maybe they could take the opportunity to add that. Otherwise I
| end up needing to sed -i 's/sed -i/perl -pi
| -e/g' *
| bewuethr wrote:
| Sed isn't part of the coreutils, though, so this would be
| outside of the scope of at least this project.
|
| There is https://github.com/chmln/sd written in Rust, but it's
| far from a sed replacement - it's reducing it to search and
| replace for fixed strings, as far as I can tell.
| chubot wrote:
| FWIW this is one reason I designed the Eggex sublanguage of Oil
| to translate to extended regexps: so you can use it with egrep,
| awk, and sed --regexp-extended.
|
| https://www.oilshell.org/release/latest/doc/eggex.html
|
| It really annoys me to have to remember multiple syntaxes for
| regexes in shell scripts!
|
| Also, even if you added Perl syntax to sed today, it wouldn't
| be available everywhere. Oil isn't available everywhere either
| but there are some advantages to having the syntax in the shell
| rather than in every tool.
|
| (It's also easy to translate eggex to the BRE syntax or Perl
| syntax, but nobody has done so yet)
| eru wrote:
| Keep in mind that Perl regexen can not be recognized in linear
| time.
| jjice wrote:
| While true, you can get similar syntax without backtracking
| in linear time, which is what the Rust regex crate and
| ripgrep use.
| tyingq wrote:
| It looks like they did consider it:
| https://debbugs.gnu.org/cgi/bugreport.cgi?bug=22801
| stevekemp wrote:
| Funny to see this here, as I remember years ago an effort to
| rewrite the coreutils package in perl:
|
| https://metacpan.org/release/CWEST/ppt-0.14
|
| Back then it was a simple way to get some things running under
| Windows, and I guess it would have also been called "memory safe"
| albeit not in a way that rust is!
| hu3 wrote:
| > Packaging in Debian isn't a trivial or even simple task. It
| requires uploading independently all the dependencies in the
| archive. Rust, with its new ecosystem and small crates, is making
| this task significantly harder.
|
| I wonder if this can be made simpler in order to spur adoption.
| jhoechtl wrote:
| Coreutils sound like they are important enough if a bug is
| spotted and the original author is unreachable, another should be
| able to jump in in short time.
|
| Do we already have enough quality Rust workforce to guarantee
| that?
| reacharavindh wrote:
| If it is MIT licensed, could MacOS get it instead of the aging
| utils there?
| toast0 wrote:
| Apple barely ever updated the BSD userland.
|
| The right thing to do is to install your own version of tools
| you need. The OS vendor tools are only for configuring the OS
| and downloading your tools.
| unrequited wrote:
| Off topic - but this might of interest.
|
| http://maizure.org/projects/decoded-gnu-coreutils/
| galactus wrote:
| Wow, that's great
| sylvestre wrote:
| Wahou; this is terrific work! It would have saved me hours of
| work if I had known that before! Thanks for sharing!
| DarmokJalad1701 wrote:
| Does this mean we will no longer have pedants going around
| telling everyone to say "GEE-EN-YOU-SLASH-Linux" ?
| littlestymaar wrote:
| Not until the glibc too gets rewritten in Rust ;)
|
| A project existed[1], but was abandoned since then, so I don't
| think it will happen anytime soon.
|
| [1]: https://github.com/japaric/steed
| rapsey wrote:
| https://gitlab.redox-os.org/redox-os/relibc
| zokula wrote:
| Too bad about the crappy license.
| littlestymaar wrote:
| Oh, I didn't know it supported Linux also! Thanks
| mprovost wrote:
| There's a great opportunity to not just reimplement the GNU
| coreutils but to actually rethink them. The way I install a Linux
| system these days and then immediately download ripgrep or fd
| reminds me of how the first thing I used to do on a Solaris
| system was install bash and the GNU coreutils. There's a rich
| history of doing this - BSD Unix started as a bunch of utilities
| and patches to the AT&T version, it only became a freestanding
| distro with its own kernel later on. And the GNU coreutils go way
| back before the Linux kernel - you just installed them on a
| commercial Unix to make it more usable. In both cases they went
| beyond just drop-in replacements and started adding lots of new
| features and flags. Things have been pretty static for a long
| time so it's good to see some innovations coming.
| est31 wrote:
| The GNU coreutils are the lowest common denominator of most GNU
| systems, and thus a lot of software targets them including the
| GNU extensions.
|
| Replacements like rg or fd are often not even compatible to the
| unix originals let alone the GNU extensions. Yes, some of the
| GNU tools are badly in need of UI improvements, but you'd have
| to abandon compatibility with a great deal of scripts.
|
| I think both a coreutils rewrite as well as some end user
| facing software like rg has its place on modern unix systems.
| I'm a very happy user of rg! But I'd like some more respect for
| tradition by some of those tools. For example, "fd" in a unix
| setting refers to file descriptors. They should rename IMO.
| adrianN wrote:
| Not being compatible is often the point of tools like
| ripgrep.
| thesuperbigfrog wrote:
| If a Rust rewrite of Coreutils is not backward compatible,
| it is not a replacement and the current GNU Coreutils would
| still be needed for backward compatibility.
|
| There is a long history of newer, better tools supporting
| backward compatibility to be a replacement for their
| predecessors:
|
| - bash and zsh support backward compatibility to be
| replacements for sh.
|
| - vim has compatibility mode to be a replacement for vi.
|
| If the Rust implementation is not backward compatible, it
| should not be called "Coreutils".
| GoblinSlayer wrote:
| What you call "Coreutils" should be called GNU/Coreutils.
| thesuperbigfrog wrote:
| The Rust implementation that we are discussing is MIT
| licensed and as far as I know not part of GNU.
|
| I do not believe that "Coreutils" is trademarked by GNU.
|
| There are other projects that use the name "Coreutils"
| that are not part of GNU:
|
| https://github.com/DiegoMagdaleno/BSDCoreUtils
| entropicdrifter wrote:
| I think the person you're replying to was just memeing on
| the Richard Stallman essay that has the line, "What you
| call Linux should really be called GNU/Linux"
| anand-bala wrote:
| The Rust rewrite of Coreutils (this post essentially)
| _is_ meant to be a drop in replacement for GNU/Coreutils.
| OTOH, ripgrep and fd are not "drop-in replacements".
| Rather, they are "replacements" that devs can use if they
| want to.
| steerablesafe wrote:
| It's not the point of ripgrep. If ripgrep were only
| marginally better at common grepping tasks then it could
| hardly justify not being compatible. It aims to be much
| better though, so it can justify it well. But not being
| compatible is not the point or the goal of the project, I
| believe.
| burntsushi wrote:
| Yes, this is correct in some sense. I didn't set out to
| create an "incompatible grep." I set out to build a
| "better" grep, where "better" could include things like
| being incompatible.
|
| Of course, ripgrep is still ultimately a lot more similar
| to grep than it is different. Where possible, I used the
| same flag names and prescribed similar behavior---if it
| made sense. Because it's good to lean on existing
| experiences. It makes it easier for folks to migrate.
|
| So it's often a balancing act. But yes, POSIX
| compatibility is certainly a non-goal of ripgrep.
| danudey wrote:
| Slightly unrelated, but ripgrep is probably the best
| improvement to my day-to-day workflow in years. Thanks so
| much for making such a great tool!
| coldtea wrote:
| That said, thanks for making ripgrep so good. I use it
| 100s of times every day for 3-4 years now... (and iirc,
| it's the default engine in VS Code as well).
| Cullinet wrote:
| tldr : ripgrep immediately strikes me as holding
| incredible medium term potential effects routed via the
| needs particular to openVMS; particularly complimenting
| the dramatically improved economics of the x64 port and
| the profound potential to attract highly experienced
| critical systems developers to FOSS via Rust breathing
| air into the VMS world. Oh and to burntsushi : I'm not
| worthy dude - excellent and thank you for ripgrep which I
| have only just encountered!
|
| ed : within 30s of learning about ripgrep I learned that
| VSCode's search routine calls ripgrep and this has just
| gifted me leverage to introduce Rust in my company...
| we're a vms shop.. searching for rust and vms gets you
| the delightful statement that since Rust isn't a
| international standard there's no guarantee it'll be
| around in.. oh about right now...the real reason why any
| language doesn't last long on vms is if the ffi
| implementation is shoddy. but that's another story that
| turns into a book the moment I try explaining how much
| value vms ffi brings... my brain suddenly just now
| thought "slingshot the critical systems devs outta vms on
| a Rust hook and things will happen ".
|
| (longer explicative comment in profile - please do check
| it out if you'd like to know why this really made my day
| and some)
| burntsushi wrote:
| Thanks for the kind words. I don't know much about
| openVMS. I read your profile bio. I'm not sure I quite
| grok all the details, but I would like to caution you:
| ripgrep is still, roughly speaking, a "grep." That is, it
| exhaustively searches through corpora. While it has some
| "smart filtering," it's not going to do much better than
| other tools when you're searching corpora that don't fit
| into memory.
|
| I do have plans to add an indexing mode to ripgrep[1],
| but it will probably be years before that lands. (I have
| a newborn at home, and my free-time coding has almost
| completely stopped.)
|
| [1] - https://github.com/BurntSushi/ripgrep/issues/1497
| Cullinet wrote:
| hi! thanks for your reply, and notes gratefully taken! in
| fact it's avoiding touching memory that's important for
| vms clusters since the model is shared storage but
| nothing else. the language support hopefully leans
| towards implementating a initial subset to poc. there's a
| serious problem going forward for the vms fs with a major
| adoption and migration from ODS5 recently abandoned.
| where my thinking is going is towards extending RMS the
| record management system which was extended to become
| what's now Oracle RDB. RDB still uses RMS and behaves
| cooperatively with fs tools and semantics. without more
| concentrated contemplation I'm stretching to venture
| there's a text search tool in the making, but I can say
| that is going to be carefully investigated and I imagine
| quite quickly because of the potential. halving licence
| costs per cpu and going from 8 to 28 cores with features
| separately very expensive in Oracle iXX.X is behind a lot
| of new commitment. Because everything is obedient to the
| system TM, that'll require integration, but let you run
| the ripgrep function on the raw database from low memory
| clients ~ so my tentative elevator pitch is going..
| orra wrote:
| Not having to specify "-rHInE" on every invocation is one
| killer reason to use ripgrep, over grep. It's a concrete
| advantage of you not having 100% compatibility as a goal.
|
| (The other killer reason is the blitzing speed.)
| ses1984 wrote:
| You can solve `-rHInE` with just an alias though.
| shakow wrote:
| Then you go back to incompatibility.
| vbernat wrote:
| Aliases are only active in your interactive shell.
| orra wrote:
| True, but then you have the occasional headscratcher that
| the command behaves differently in your script than when
| you run it manually.
| shakow wrote:
| Exactly, if grep behaves differently in your scripts and
| your shell, there's no point in not using ripgrep (or ag,
| or silver searcher, or whatever strikes your fancy).
| andoriyu wrote:
| The difference is - your script works out of the box, or
| you need to install ripgrep.
| Sebb767 wrote:
| If that is the only incompatibility, it would be easy to
| make a patch that checks if it is called as "grep" and
| default to "not -rHInE" - so one could ship ripgrep as
| default and yet have backwards compatibility. Some busy
| boxes already do that iirc.
|
| EDIT: So I've quickly looked into it and it seems nobody
| did an extensive comparison to the grep feature set or
| the POSIX specification. If I have some time later this
| week I might do this and check whether something like
| this would be viable.
| burntsushi wrote:
| It's not even close to the only incompatibility. :-)
| That's a nominal one. If that were really the only thing,
| then sure, I would provide a way to make ripgrep work in
| a POSIX compatible manner.
|
| There are lots of incompatibilities. The regex engine
| itself is probably one of the hardest to fix. The
| incompatibilities range from surface level syntactic
| differences all the way down to how match regions
| themselves are determined, or even the feature sets
| themselves (BREs for example allow the use of
| backreferences).
|
| Then of course there's locale support. ripgrep takes a
| more "modern" approach: it ignores locale support and
| instead just provides what level 1 of UTS#18 specifies.
| (Unicode aware case insensitive matches, Unicode aware
| character classes, lots of Unicode properties available
| via \p{..}, and so on.)
| Sebb767 wrote:
| Pity! I did look; only "-E" and "-s" diverge from the
| POSIX standard parameter-wise. But making significant
| changes to the pattern engine is probably not worth it.
|
| Thanks anyway, I enjoy rg quite a lot :)
| nicoburns wrote:
| It's worth noting that the implementation of ripgrep has
| been split up into a whole bunch of modular components.
| So it wouldn't be out of the question for someone to
| piece those together into a GNU-compatible grep
| implementation.
| orra wrote:
| True, though is there any point? ripgrep's homegrown
| regex engine only supports true regular expressions.
|
| To give backreference support, ripgrep can optionally use
| PCRE. But PCRE already comes with its own drop in grep
| replacement...
| burntsushi wrote:
| To the extent that you want to get a POSIX compatible
| regex engine working with ripgrep, you could patch it to
| use a POSIX compatible regex engine. The simplest way
| might be to implement the requisite interfaces by using,
| say, the regex engine that gets shipped with libc. This
| might end up being quite slow, but it is very doable.
|
| But still, that only solves the incompatibilities with
| the regex engine. There are many others. The extent to
| which ripgrep is compatible with POSIX grep is that I
| used flag names similar to GNU grep where I could. I have
| never taken a fine toothed comb over the POSIX grep spec
| and tried to emulate the parts that I thought were
| reasonable. Since POSIX wasn't and won't be a part of
| ripgrep's core design, it's likely there are many other
| things that are incompatible.
|
| A POSIX grep can theoretically be built with a pretty
| small amount of code. Check out busybox's grep
| implementation, for example.
|
| While building a POSIX grep in Rust sounds like fun, I do
| think you'd have a difficult time with adoption. GNU grep
| isn't a great source of critical CVEs, it works pretty
| well as-is and is actively maintained. So there just
| isn't a lot of reason to. Driving adoption is much easier
| when you can offer something new to users, and in order
| to do that, you either need to break with POSIX or make
| it completely opt-in. (I do think a good reason to build
| a POSIX grep in Rust is if you want to provide a complete
| user-land for an OS in Rust, perhaps if only for
| development purposes.)
| Sebb767 wrote:
| Well, the reasons I see for being POSIX-compatible would
| be:
|
| 1. Distributions could adopt rg as default and ship with
| it only, adding features at nearly no cost
|
| 2. The performance advantage over "traditional" grep
|
| Number 1 is basically how bash became the default; since
| it is a superset of sh (or close enough at least),
| distributions could offer the feature set at no
| disadvantage. Shipping it by default would allow scripts
| on that distribution to take advantage of rg and,
| arguably, improve the situation for most users at no
| cost.
|
| If one builds two programs in one with a switch, you're
| effectively shipping optional software, but in a single
| binary, which makes point 1 pretty moot. If you then also
| fall back on another engine, point 2 is moot as well - so
| the only point where this would actually be useful is if
| rg could become a good enough superset of grep that it
| would provide sufficient advatages (most greps _already_
| provide a larger superset of POSIX, though). Everything
| else would just add unnecessary complexity, in my
| opinion.
|
| But it would have been nice :)
| burntsushi wrote:
| Ah I see. Yeah, that's a good point. But it's a very very
| steep hill to climb. In theory it would be nice though.
| There's just a ton of work to do to hit POSIX
| compatibility _and_ simultaneously be as good at GNU grep
| at other things. For example, the simplest way to get the
| regex engine to be POSIX compatible would be to use an
| existing POSIX compatible regex engine, like the one
| found in libc. But that regex engine is generally
| regarded as quite slow AIUI, and is presumably why GNU
| grep bundles it 's entire own regex engine just to speed
| things up in a lot of common cases. So to climb this
| hill, you'd either need to follow in GNU grep's footsteps
| _or_ build a faster POSIX compatible regex engine. Either
| way, you're committing yourself to writing a regex
| engine.
| slimsag wrote:
| I didn't look closely, but Oniguruma is pretty dang fast
| and has drop-in POSIX syntax + ABI compatability as a
| compile-time option. Could maybe use that.
| jftuga wrote:
| I would love it if --crlf was enabled by default with the
| Windows version. This would make using ^ and $ easier.
| jpitz wrote:
| Make a config file for ripgrep and enable it.
|
| https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.m
| d#c...
| alerighi wrote:
| Another thing to mention: there is not only the compatibility
| at the side of software (scripts), that to me is easy to
| maintain (just keep the old software with the new compatible
| one and you are done).
|
| The most important thing is compatibility with humans. The
| main reason because I tend to use a pretty standard setup is
| because I know that these tools are the standard, I need to
| ssh into a server to resolve some issue, well I have a system
| that I'm familiar with.
|
| If for example on my system I have fancy tools then I would
| stop using the classic UNIX tools, and then every time I
| would have to connect to another system (quite often) I would
| type wrong commands because they are different or totally not
| present or have to install the new fancy tools on the new
| system (and it's not always the case, for example if we talk
| about embedded devices with 8Mb of flash everything more than
| busybox is not possible).
|
| To me the GNU tools have the same problem, they got you used
| to non POSIX standard things (like putting flags after
| positional arguments) that hits you when you have to do some
| work on systems without them. And yes, there still exist
| system without the GNU tools, MacOS for example, or embedded
| devices, some old UNIX server that is still running, etc.
|
| Last thing, if we need to look at the future... better change
| everything. Let's be honest, if I could chose to switch
| entirely on a new shell and tools, and magically have it
| installed on every system that I use, it would probably be
| PowerShell. For the simple reason that we are no longer in
| the '80 where everything was a stream of text, and a shell
| that can manipulate complex objects is fantastic.
| pdimitar wrote:
| > _Last thing, if we need to look at the future... better
| change everything._
|
| Absolutely. We had decades to work with a fairly stable set
| of tools and they are not going anywhere. Whoever needs
| them, they are there and likely will be for several more
| decades.
|
| I am gradually migrating all my everyday DevOps workflows
| (I am a senior programmer and being good with tooling is
| practically mandatory for my work) to various Rust tools:
| ls => exa, find => fd, grep => rg, and more and more are
| joining each month. I am very happy about it! They are
| usually faster and work more predictably and (AFAICT) have
| no hidden surprises depending on the OS you use them in.
|
| > _we are no longer in the '80 where everything was a
| stream of text, and a shell that can manipulate complex
| objects is fantastic._
|
| Absolutely (again). We need a modern cross-platform shell
| that does that. There are few interesting projects out
| there but so far it seems that the community is unwilling
| to adopt them.
|
| I am personally also guilty of this: I like zsh fine even
| though the scripting language is veeeeery far from what I'd
| want to use in a shell.
|
| Not sure how that particular innovation will explode but
| IMO something has to give at one point. Pretending
| everything is a text and wasting trillions of CPU hours in
| constantly parsing stuff is just irresponsible in so many
| ways (ecological included).
| throwaway894345 wrote:
| > you'd have to abandon compatibility with a great deal of
| scripts.
|
| Having tried to reproducibly package various software, the C
| layer (and especially the coreutils layer) are the absolute
| worst, and I wouldn't shed a tear if we started afresh with
| something more holistically designed.
| asguy wrote:
| Absolute worst compared to what? Have you tried packaging
| software on Windows?
|
| If you have something better in mind, please implement it.
| It'd get used.
| throwaway894345 wrote:
| "Worst" compared to packaging software in other
| languages.
|
| > If you have something better in mind, please implement
| it. It'd get used.
|
| Nonsense. It's not a dearth of better options that causes
| C folks to clutch their autotools and bash scripts and
| punting-altogether-on-dependency-management; we've had
| better options for decades. Change will happen, but it
| will take decades as individual packages become less
| popular in favor of newer alternatives with better
| features--build hygiene will improve because these newer
| projects are much more likely to be built in Rust or
| "disciplined C" by developers who are increasingly of a
| younger generation, less wed to the Old Ways of building
| software.
| asguy wrote:
| > It's not a dearth of better options that causes C folks
| to clutch their autotools and bash scripts and punting-
| altogether-on-dependency-management; we've had better
| options for decades.
|
| I build and package golang/Python/rust/C/C++ binaries
| using GNU make, bash, and Debian's tooling. I have
| dependency management, parallel builds, and things are
| reproducible. I do it that way, because I need a glue
| that will deploy to VMs, containers, bare metal,
| whatever. I don't use it because I'm scared of other
| tooling. I use it because I haven't seen a better
| environment for getting software I work on, into
| heterogeneous environments.
|
| I'm not attached to the old way; I'm attached to ways
| that I can be productive with.
| Spivak wrote:
| Look I _loathe_ autotools as a developer but it has the
| advantage of being extremely battle-tested and comes with
| lots of tooling to package it out of the box. In RPM spec
| files any autotools project can be packaged with more or
| less a single macro.
| eeZah7Ux wrote:
| Debian has been building GNU coreutils and tons of more C
| reproducibly for years.
| throwaway894345 wrote:
| And it takes a tremendous amount of effort, discipline,
| expertise, and deep, intimate familiarity with the entire
| dependency tree. This is the whole problem. We should be
| able to build software without needing to be intimately
| familiar with every package in the tree and how it wants
| us to build it, what its implicit dependencies are, etc.
|
| For example, for the overwhelming majority of Rust and Go
| projects, there are _explicit_ dependency trees in which
| all nodes are built roughly the same way such that a
| builder can do "cargo build" and get a binary. No need
| to understand each node's bespoke, cobbled-together build
| system or to figure out which undocumented dependencies
| are causing your build to fail (or where those
| dependencies need to be installed on the system or how to
| resolve conflicts with other parts of the dependency
| tree).
| vmchale wrote:
| fd is better IMO, plus it is more cross-platform. Same flags
| on Linux and Mac
| dralley wrote:
| fd doesn't do the same thing at all. fd matches against
| filenames, rg matches against file contents.
| simias wrote:
| I think both approaches are interesting and valuable. Yes
| ripgrep is great for interactive usage, but in scripts I'll
| still be using grep for the foreseeable future.
|
| I think the situation with GNU coreutils on Solaris/BSDs is a
| bit different: a lot of the time the BSD/Solaris tools on one
| hand and the GNU tools on the other were incompatible in some
| ways. Some flags would be different (GNU ls vs. FreeBSD ls for
| instance have pretty significant flag differences) and then you
| have things like Make who have pretty profound syntax
| differences between flavours.
|
| As a result if you needed to write portable scripts you either
| had to go through a painful amount of testing and special
| casing for the various dialects, or you'd just mandate that
| people installed the GNU version of the tools on the system and
| use that. That's why you can pretty much assume that any BSD
| system in the wild these days has at the very least gmake and
| bash installed, just because it's used by third party packages.
|
| So IMO people used GNU on non-GNU systems mainly for
| portability or because they came from GNU-world and they were
| used to them.
|
| I know that there's a common idea that GNU tools were more
| fully featured or ergonomic than BSD equivalents, but I'm not
| entirely sold on that. I think it's mostly because people these
| days learn the GNU tools first, and they're likely to notice
| when a BSD equivalent doesn't support a particular feature
| while they're unlikely to notice the opposite because they
| simply won't use BSD-isms.
|
| For instance for a long time I was frustrated with GNU tar
| because it wouldn't automatically figure out when a compression
| algorithm was used (bz, gz or xz in general) and automatically
| Do The Right Thing. FreeBSD tar however did it just fine.
|
| Similarly I can get FreeBSD ls to do pretty much everything GNU
| ls can do, but you have to use different flags sometimes. If
| you don't take the time to learn the BSDisms you'll just think
| that the program is more limited or doesn't support all the
| features of the GNU version.
|
| An other example out of the top of my head is "fstat" which I
| find massively more usable than lsof, which is a lot more
| clunky with worse defaults IMO. It's mainly that lsof mixes
| disk and network resources by default, while fstat is dedicated
| to local resources and you use other programs like netstat to
| monitor network resources. Since I rarely want to monitor both
| at the same time when I'm debugging something, I find that it
| makes more sense to split those apart.
| mprovost wrote:
| You're right, there were a lot of compatibility problems in
| the past because tools reused the same names but didn't
| behave the same. Sometimes they were completely different
| like ps - the BSD and System V versions are totally
| incompatible. There were attempts to unify some tools with
| something new, for example pax (which was supposed to replace
| the incompatible tar and cpio, but which basically nobody
| uses). These days most tools are getting new names (ripgrep,
| exa) where in the past you'd just call it the same thing. I
| think that decision has freed people up from having to
| maintain backwards compatibility with the GBU tools - which
| weren't always backwards compatible with the tools of the
| same name that they replaced, as you point out. Once you lose
| the requirement to be compatible you can actually rethink how
| the tool should be used and not worry about a decision that
| was made by someone in the 70s.
| simias wrote:
| Ah yeah I forgot about `ps`, that's definitely one of the
| worst offenders.
|
| I completely agree with you overall point btw, I just don't
| think that's in an either/or type of situation. Rewriting
| the existing standard utils in Rust could provide _some_
| benefits, but it 's also great to have utils like ripgrep
| who break compatibility for better ergonomy.
| JoshTriplett wrote:
| One of the standard things coreutils does right that many
| other implementations do wrong: after running a command with
| a filename argument, hit up, space, dash, some option you
| want to add, enter. coreutils handles the option you added.
| Many other implementations either complain or treat the
| option as another file.
|
| That was the original feature that made me want coreutils on
| other systems.
| wahern wrote:
| That's a matter of taste. Argument permutation is evil,
| IMHO. It's also dangerous. If someone can't be bothered to
| order their arguments, they also can't be bothered to use
| the "--" option terminator, which means permutation is a
| correctness and security headache.
|
| But it's the behavior on GNU systems, and it's even the
| behavior of most applications using getopt_long on other
| non-GNU, non-Linux systems (because getopt_long permuted by
| default, and getopt_long is a de facto standard now). So it
| should be supported.
| JoshTriplett wrote:
| If you're writing a script, perhaps.
|
| I'm talking about _interactive command-line usage_ , for
| which the ability to put an argument at the end provides
| more user-friendliness.
| wahern wrote:
| But the command doesn't know that, and in general best
| practice is (or was) for commands to not alter their
| behavior based on whether they're attached to a terminal
| or not.
|
| I won't deny the convenience. (Technically jumping
| backwards across arguments in the line editor is trivial,
| but I admit I keep forgetting the command sequence.) But
| from a software programming standpoint, the benefit isn't
| worth the cost, IMO.
|
| And there are more costs than meet the eye. Have you ever
| tried to implement argument permutation? You can throw
| together a compliant getopt or getopt_long in
| surprisingly few lines of code.[1] Toss in argument
| permutation and the complexity explodes, both in SLoC and
| asymptoptic runtime cost (though you can trade the latter
| for the former to some extent).
|
| [1] Example:
| https://github.com/wahern/lunix/blob/master/src/unix-
| getopt....
| JoshTriplett wrote:
| > But the command doesn't know that, and in general best
| practice is (or was) for commands to not alter their
| behavior based on whether they're attached to a terminal
| or not.
|
| I completely agree; most commands should behave the same
| on the command-line and in scripts, because many scripts
| will start out of command-line experimentation. That's
| one of the good and bad things about shell scripting.
|
| > Have you ever tried to implement argument permutation?
| You can throw together a compliant getopt or getopt_long
| in surprisingly few lines of code. Toss in argument
| permutation and the complexity explodes, both in SLoC and
| asymptoptic runtime cost (though you can trade the latter
| for the former to some extent).
|
| "surprisingly few lines of code" doesn't seem like a
| critical property for a library that needs implementing
| once and can then be reused many times. "No more
| complexity than necessary to implement the required
| features" seems like a more useful property.
|
| I've used many command-line processors in various
| languages, all of which have supported passing flags
| after arguments. There are many libraries available for
| this. I don't think anyone should reimplement command-
| line processing in the course of building a normal
| command-line tool.
|
| I personally don't think permutation (in the style of
| getopt and getopt_long, at least in their default mode)
| is the right approach. Don't rearrange the command line
| to look like all the arguments come first. Just parse the
| command line and process everything wherever it is. You
| can either parse it into a separate structure, or make
| two passes over the arguments; neither one is going to
| add substantial cost to a command-line tool.
|
| So, this is _only_ painful for someone who needs to
| reimplement a fully compatible implementation of getopt
| or getopt_long. And there are enough of those out there
| that it should be possible to reuse one of the existing
| ones rather than writing a new one.
| anthk wrote:
| > BSD system in the wild these days has at the very least
| gmake and bash installed, just because it's used by third
| party packages.
|
| No. Gmake, maybe. But not bash.
___________________________________________________________________
(page generated 2021-03-09 23:00 UTC)