[HN Gopher] Weird architectures weren't supported to begin with
___________________________________________________________________
Weird architectures weren't supported to begin with
Author : woodruffw
Score : 219 points
Date : 2021-02-28 16:22 UTC (6 hours ago)
(HTM) web link (blog.yossarian.net)
(TXT) w3m dump (blog.yossarian.net)
| h2odragon wrote:
| If I fire up an Alpha CPU today, I'm not expecting that the
| latest versions of all my favorite free software is going to run
| on it. Asking maintainers to "fix it for me" then would be
| unreasonable. Part of running hardware that far out of date is
| hunting for the last versions of anything that supported it;
| whether that was FOSS or commercial its still the way the world
| is and has been.
|
| It'll be interesting to see how the Rust community responds to
| this; are they so eager to absorb everything that they'll put
| effort into supporting niches to get more users,or will they take
| the opportunity to be "opinionated" and exclusive and shed the
| effort of catering to obsolescence?
| cbmuser wrote:
| > If I fire up an Alpha CPU today, I'm not expecting that the
| latest versions of all my favorite free software is going to
| run on it. Asking maintainers to "fix it for me" then would be
| unreasonable.
|
| No one does that. What people complain about is that code that
| used to be perfectly portable for years suddenly becomes locked
| to a very limited set of targets with the argument that memory
| safety is more important than anything else.
|
| > It'll be interesting to see how the Rust community responds
| to this; are they so eager to absorb everything that they'll
| put effort into supporting niches to get more users,or will
| they take the opportunity to be "opinionated" and exclusive and
| shed the effort of catering to obsolescence?
|
| Rust just needs to help get one of the several alternative Rust
| implementations based on gcc officially supports similar to
| gccgo.
|
| Then the portability issue will have been fixed once and for
| all and Rust will be chosen even for code on obscure targets
| such as Elbrus 2000, Sunway or Tricore.
| pornel wrote:
| The code that used to be perfectly portable still exists,
| fork it and keep using it.
|
| But if you're asking for a project to be _maintained_ , then
| it means you want maintainers to put more work to keep new
| code working on an old platform. Constraints of old/niche
| platforms cause extra work for developers when adding new
| features or improving security.
| JulianMorrison wrote:
| I think the point this article is making is that the
| portability was a sham and a mirage from the get go.
|
| Yes, it might _compile_ on an oddball architecture, but a lot
| of that would be the autotools build system and the C
| compiler fudging around important details that could easily
| leave you with something that kinda-sorta runs but is a wide
| open security flaw. Or that runs for a while and then breaks
| in ways nobody could have predicted because you aren 't meant
| to try to run cryptographic software on something out of the
| museum of historical computing.
| rodgerd wrote:
| > What people complain about is that code that used to be
| perfectly portable
|
| Literally the whole point of the article is that this
| assertion is bullshit. It was never perfectly portable. It
| merely happened to compile, and maybe work, and maybe
| actually was secure.
|
| > Rust just needs to help get one of the several alternative
| Rust implementations based on gcc officially supports similar
| to gccgo.
|
| Who is "Rust"? Why would they pour money and effort into
| this? Would the gcc community finally ignore rms' demands to
| make gcc as hostile as possible to implementing new front
| ends?
|
| > Then the portability issue will have been fixed once and
| for all and Rust will be chosen even for code on obscure
| targets such as Elbrus 2000, Sunway or Tricore.
|
| This does not solve the problem the author describes.
| longcommonname wrote:
| They can pin an older version and have it work.
| Ar-Curunir wrote:
| > Then the portability issue will have been fixed once and
| for all
|
| That's just not true. These targets may make different
| assumptions about various low level things such as memory
| ordering, byte-width, behavior on overflow, etc. While C
| might be okay to defer to the architecture on these
| questions, Rust is more strict.
|
| I personally think you'll just end up with a bunch of broken
| binaries.
| msbarnett wrote:
| > What people complain about is that code that used to be
| perfectly portable for years suddenly becomes locked to a
| very limited set of targets with the argument that memory
| safety is more important than anything else
|
| As TFA points out, this is a mistaken understanding of the
| situation. What we have here is code that gave the _illusion_
| of being "perfectly portable" (while not actually being
| written to target or tested against the peculiarities of
| niche architectures like Itanium and PA-RISC it happened to
| successfully compile on) being replaced with a new version
| that only build on machines its authors have actually given
| any consideration to the security properties of.
|
| That this inconveniences people is obvious. Why they imagine
| this is a net security loss for them is less obvious - the
| older C versions still exist, and any concerns that they're
| missing out on new security updates are swamped out by the
| fact that the older versions may well never have behaved
| securely because nobody from the project was ever writing the
| code with PA-RISC's memory and instruction ordering
| properties in mind to begin with.
| guerrilla wrote:
| Your question makes me wonder what niche Rust is trying to
| fill. Anything that can't bootstrap literally everything then
| will never replace C.
| qbasic_forever wrote:
| The niche of software that doesn't take over your computer
| with malware because you made an off by one error.
| ithkuil wrote:
| Wonder if rust (or llvm) could just have a C backend as a
| fallback for unsupported architectures. Perhaps some stuff
| would be slow but likely faster than flat out emulation
| steveklabnik wrote:
| llvm had a C backend at one point, but my understanding is
| that it bitrotted and was removed. I think there's been
| some work to bring it back? Not 100% sure.
| ithkuil wrote:
| TIL https://github.com/thepowersgang/mrustc
| nicoburns wrote:
| I think the primary goal is Mac/Windows/Linux/BSD/iOS/Android
| on x86/ARM (32bit and 64bit) variants. That covers 99% of
| consumer and server computing. And aside from anything else:
| making that secure would be a huge win.
|
| But it's not like Rust doesn't have wide platform support.
| For example, it's already possible to run Rust on Risc-V. And
| it's is improving all the time.
| h2odragon wrote:
| I take no position on Rust other than interested observer;
| thus the question. There's several factions there, i think
| the "lets make a better language and spread it everywhere so
| it gets used" faction ares going to be opposed by the
| "opinionated zealots of Correct Thinking" and i wonder which
| gets steamrolled.
| guerrilla wrote:
| What's the "Correct Thinking"?
| arithmomachist wrote:
| > Companies should be paying for this directly: if
| pyca/cryptography actually broke on HPPA or IA-64, then HP or
| Intel or whoever should be forking over money to get it fixed or
| using their own horde of engineers to fix it themselves.
|
| This about sums it up.
| ris wrote:
| Problem is that ARM & AArch64 were considered a "weird
| architecture" by the non-Android linux stack until really _very_
| recently, and the migration to a new architecture is _still_ not
| plain-sailing in many peoples ' experience. Without the
| assumption that most open source package authors are implicitly
| trying to be architecture-independent to some degree, we will
| literally all be stuck on x86_64 for the rest of our lives (the
| migration _to_ amd64 itself I remember as being a number of years
| of people working on "unsupported" stuff FWIW).
|
| For users on "weird architectures" to be petitioning that a move
| the pyca authors are making is causing inconvenience to them is
| perfectly reasonable in my eyes.
| rodgerd wrote:
| > Problem is that ARM & AArch64 were considered a "weird
| architecture"
|
| The ARM world is a blizzard of proprietary, undocumented
| implementations with limited support for the upstream kernels,
| often can boot only a vendor-specific distro that is quickly
| abandoned, and full of boards that blink in and out of
| existence at the drop of a hat. It absolutely is a weird
| architecture.
|
| > For users on "weird architectures" to be petitioning that a
| move the pyca authors are making is causing inconvenience to
| them is perfectly reasonable in my eyes.
|
| Yes, this is exactly the sense of entitlement that the author
| is talking about when he describes the destruction of people's
| interest in working on open source.
| ris wrote:
| > this is exactly the sense of entitlement
|
| Now come on with your "entitlement". It's not as though we're
| talking about some random people who made some little package
| for their own use and decided to make it available in case
| anyone else found it useful, and now the community demands
| from them are becoming too much and are something they never
| asked for. This is a group that have named themselves the
| Python Cryptographic Authority and have chosen the prominent
| pypi package name of just "cryptography". They couldn't have
| done any more to encourage the broader community to depend on
| it and make it a core part of their stack.
|
| In comparison, I couldn't imagine the python core team (also
| largely unpaid) doing this with one of their stdlib modules
| and then dismissing those objecting as "entitled".
|
| (FWIW I'm not particularly interested in taking a side in
| this issue, but think your labelling as "entitled" is
| unhelpful)
| darksaints wrote:
| I agree with the idea that it is entitled. Hell, Python
| itself is only directly supported on a couple of
| architectures and operating systems. It has even fewer
| "tier 1" targets than Rust does! It is made available in
| source format only for other packagers to use as they see
| fit, _but it is not python 's responsibility to support
| it_. Why should a library maintainer feel any obligation to
| support platforms that the language doesn't provide first
| class support for?
| simias wrote:
| I don't think it's a good comparison. The architectures
| mentioned in the pyca/cryptography repo are:
|
| - Alpha: introduced 1992, discontinued in the early 2000s
|
| - HP/PA: introduced 1986, discontinued in 2008
|
| - Itanium: introduced 2001, end of life 2021
|
| - s390: introduced 1990, discontinued in ~1998
|
| - m68k: introduced 1979, still in use in some embedded systems
| but not developed at Motorola since 1994.
|
| ARM was once not as popular as it is nowadays but it was never
| moribund and in my experience has always had decent tooling and
| compiler support. Furthermore I'm sure that if tomorrow HP/PA
| makes a comeback for some reason, LLVM will add support for it.
| Out of the list I'd argue that the only two who may be worth
| supporting are Motorola 68k and maybe Itanium but even then
| it's ultra niche.
|
| I personally maintain software that runs on old/niche DSPs and
| I like emulation, so I can definitely feel the pain of people
| who find new release of software breaking on some of the niche
| arch they use (I tried running Rust on MIPS-I but couldn't get
| it to work properly because of lack of support in LLVM for
| instance). These architectures are dead or dying, not up-and-
| coming like, say, RISC-V which has gained some momentum lately.
|
| But while I sympathize with people who are concerned by this
| sort of breakage, it's simply not reasonable to expect these
| open source projects to maintain backward compatibility with
| CPUs that haven't been manufactured in decades. As TFA points
| out it's a huge maintenance burden: you need to regression test
| against these architectures you may know nothing about, you may
| not have an easy way to fix the bugs that arise etc...
|
| >open source groups should not be unconditionally supporting
| the ecosystem for a >large corporation's hardware and/or
| platforms.
|
| Preach. Intel is dropping Itanium, HP dropped HP/PA a long time
| ago. Why should volunteers be expected to provide support for
| free instead?
|
| It's like users who complain that software drops support for
| Windows 7 when MS themselves don't support the OS anymore.
| ndesaulniers wrote:
| SystemZ, what was s390x/390 seems relatively well supported
| by IBM and Red Hat in my experience.
| PurpleFoxy wrote:
| Is it not reasonable to throw this back at the CPU makers? If
| you want to bring out a new cpu architecture, port all the
| compilers to it before you start selling it.
| monocasa wrote:
| FWIW, s390 wasn't really discontinued in 1998. There's still
| new s390 chips being designed and used.
| zozbot234 wrote:
| s390 is the 31-bit-only variant, that has been discontinued
| for some time. Modern variants are 64-bit based, and still
| supported.
|
| All that being said, it's quite worthwhile to include these
| "dead" architectures in LLVM and Rust, if only for
| educational reasons. That need not imply the high level of
| support one would expect for, e.g. RISC-V.
| monocasa wrote:
| Lots of 32 bit code still gets run on these machines.
| ndesaulniers wrote:
| Two architectures currently being added to LLVM are m68k
| and csky. I don't think either are that new (I thought
| csky was, but it was explained to me by Linux kernel
| architecture folks that it has old roots from Motorola,
| with folks from alibaba using that for 32b but moving to
| riscv for 64b).
| darksaints wrote:
| But it is unreasonable, for one of two reasons:
|
| 1) If the architecture is in active production, there is
| someone somewhere trying to make money by selling it. If they
| are intent on only supporting proprietary compilers, they need
| to accept the consequences of that decision: users won't use
| their hardware because they can't use the software that they
| want to use. If they want the architecture to be widely used,
| they have a fiduciary obligation to ensure that they have
| reliable and well tested backends to major compilers.
|
| 2) If the user is using old architectures that are no longer in
| production or no longer supported, there isn't ever any
| reasonable expectation of continuing software support. You're
| stuck with old software, full stop.
|
| In the case of your objection, AArch64 and ARM manufacturers
| have the obligation to develop openly available backends for
| their architectures. And they've taken that seriously, as
| should any newcomer architectures.
| zozbot234 wrote:
| > If the user is using old architectures that are no longer
| in production or no longer supported, there isn't ever any
| reasonable expectation of continuing software support. You're
| stuck with old software, full stop.
|
| That's not a very reasonable POV. Many of these architectures
| are very well understood and very easily supported via
| emulation. There's no need to run them on actual hardware,
| especially if you aren't dealing with anything close to bare-
| metal quirks.
| steveklabnik wrote:
| Incidentally, aarch64-unknown-linux-gnu became a Tier 1
| supported platform in Rust recently, in part because of the
| support of ARM themselves.
|
| (My day job involves a lot of Tier 2 ARM work, and I don't
| personally run into any more bugs than Tier 1 platforms. YMMV.)
| ncmncm wrote:
| It Seems To Me that a little more attention to cross-compilation
| support would go a long way toward extending support to all the
| weird architectures.
|
| Really, running a language toolchain on the target machine, where
| the target machine isn't what everybody has, is largely
| pointless. We each have loads of machines to run toolchains on.
|
| Early in the life of a language, with a compiler meta-circularly
| written in it, the compiler is the biggest program in the
| language, so may seem like an effective early test of a port.
| But, _it must be said_ , meta-circularity is, in large degree,
| wankage. There are much better choices to demonstrate the merits
| of your new language than coding its compiler; a compiler is not
| the persuasive demonstration of language merit it once was.
| Writing your language's compiler in an unstable and unreliable
| new language, making it subject to its own bugs and early
| inefficiencies, is a poor way to help a new and fragile user
| community up to speed.
|
| How did we get on this meta-circularity fetish, anyway? It's easy
| to see how it happened with C, but aping what C did is no formula
| for success, anymore. C is, or once was, a pretty simple language
| to compile, so its compiler wouldn't much benefit from being
| implemented in a more powerful language, if there had been any.
| That cannot be said of modern languages, where all the help you
| can give them is just enough to make a usable compiler. Notably,
| LLVM is coded in (a dialect of) C++, even the project's C
| compiler.
|
| A cross-toolchain, coded in C++, built to run on Linux amd64 and
| nowhere else, ought to be all you need to bring up full support
| for any language on any architecture. As we say about low Earth
| orbit, it puts you halfway to anywhere in the universe. Even
| where you don't have Linux, you have a VM you can run it in.
| Cross-development skills take some investment, but they pay
| forever after. The overwhelming majority of places you could run
| code nowadays can't meaningfully host a development environment
| anyway. Cross skills enable you to program the microcontrollers
| that manage about everything in the world, today, and phones, and
| HPC supercomputers alike.
|
| A cross-built program can not only be built on your desktop host
| machine. It can be debugged from there, running in an emulator or
| on the actual target hardware. The debugger runs on the host,
| with access to all the source code. What we call a "debug stub",
| a tiny fragment of code is inserted in the program or target
| environment that implements the minimal primitives a debugger
| needs: mostly just peek, poke, and run; maybe watchpoints if the
| target supports that. The debugger talks to the stub over a
| network port, or a serial cable, or by blinking lights, if
| necessary.
|
| Linux on amd64 didn't take over the world, with no marketing
| budget to help, by being the best place possible place to run
| programs. (It's just adequate, and better at it than Windows.)
| Linux got there by being the best possible place to develop
| software. Make it the best place to develop in your language,
| whatever the target, and you are halfway to anywhere.
| ndesaulniers wrote:
| This comes up a lot with my work on compiling the Linux kernel
| with LLVM. So much so I've made a tier list to describe what I
| picture as a Venn diagram of support:
| https://clangbuiltlinux.github.io/
|
| Also recently seen on this topic:
|
| https://people.gnome.org/~federico/blog/librsvg-rust-and-non...
|
| https://www.reddit.com/r/rust/comments/lfysy9/pythons_crypto...
| eqvinox wrote:
| Does the "powerpc" listed really mean "ppc64le" and the "s390"
| really mean "s390x"? Certainly _feels_ like it, but I 'm not
| sure...
|
| If you're already making such a nice overview, could you maybe
| not mar it with such ambiguity? Kinda defeats the purpose :(
|
| [Ed.: even "x86" is confusing. x86_64 does work, right?]
| ndesaulniers wrote:
| No. The kernel doesn't make such distinctions. The convention
| used is what the kernel sources call them under arch/. See
| arm64 vs aarch64 for example.
| eqvinox wrote:
| I guess that means that overview is for kernel developers.
| Those directory names are meaningless to me as an user...
|
| FWIW, I'm actually asking out of a real need. I have two
| powerpc boxes (no, not ppc64le) here, which we use for
| testing big-endian compatibility. But last I checked, LLVM
| supported neither 32-bit PowerPC nor big-endian PowerPC.
|
| Actually... I guess the real argument is that the kernel
| directory names aren't sufficient. Because they specify
| architecture families, not architectures.
|
| [And, funnily enough, LLVM _does_ actually seem to support
| 32-bit BE PowerPC. Now I 'm really confused.]
| ndesaulniers wrote:
| > But last I checked, LLVM supported neither 32-bit
| PowerPC nor big-endian PowerPC.
|
| How long ago did you check? I've been working on this for
| 3-4 years, and we've had coverage of kernel builds for
| 32b BE ppc for at least 2 years.
|
| I just checked our coverage, looks like we test ppc64,
| ppc64le, and ppc (32b BE). So no coverage of 32b LE atm,
| but I think llvm recently gained support for the relevant
| triple.
|
| Try it and let us know if it boots!
| gsnedders wrote:
| "The absence of official builds means that" [incomplete sentence]
| woodruffw wrote:
| Whoops, that was a transposition. Thanks, fixed.
| [deleted]
| nerdponx wrote:
| _No free work for platforms that only corporations are using. No,
| this doesn't violate the open-source ethos; nothing about OSS
| says that you have to bend over backwards to support a corporate
| platform that you didn't care about in the first place._
| nickysielicki wrote:
| There's a weird mental leap that rust evangelists, militant
| atheists, and hyper-progressives have in common. I like rust as
| much as the next guy but the zealotry is unbelievable.
|
| It's almost getting to the point that I'm rooting for rust to
| fail and for C++ to just continue to get better, just so that
| these people who claim to take memory safety so seriously might
| shut the fuck up about it.
|
| The gentoo folks "complaining" about this have contributed more
| to LLVM (and, by extension, rust) than 95% of the people arguing
| that they're in the wrong.
|
| edit: what I ultimately have an issue with is tearing down the
| past because we see a brighter future on the horizon. The baby is
| being thrown out with the bath water.
| ojnabieoot wrote:
| I feel like the author is not being that zealous! More to the
| point, he is not really focusing on memory safety so much as
| his frustration with the nature of the complaints RE: Rust.
| Perhaps he is being pithy and snarky enough to hurt people's
| feelings, but his points about C being "organizationally"
| unsafe are well-established and not controversial.
|
| And his broader point is one that C evangelists should take
| seriously - the fact that C can be compiled on all sorts of
| esoteric architecture does not mean that every open-source C
| program is supported on those architectures. There is a serious
| risk with using a crypto library on instruction sets the author
| isn't supporting, and the responsibility is on the package
| distributor or consumer (depending), not the maintainer.
| cbmuser wrote:
| The point is: If Rust is supposed to replace as _the_ systems
| programming language, it needs to be as portable as C.
|
| There is no point in arguing what architectures are
| considered obscure and which are not since there a lot of
| fields of applications in industry and research which use
| architectures most people never heard of such as Elbrus 2000
| or Sunway.
| pornel wrote:
| Rust doesn't need to replace 100% of C, just like C hasn't
| replaced 100% of Fortran or Pascal.
| DasIch wrote:
| C is more portable than Rust when you define portable as
| "it compiles". If you define portable to mean "works
| correctly" I expect C is about as portable as Rust in the
| context of most applications.
| the_only_law wrote:
| I didn't think it was possible but the anti rust crowd managed
| to become even more obnoxious than the Rust evangelism strike
| force
| staticassertion wrote:
| > It's almost getting to the point that I'm rooting for rust to
| fail and for C++ to just continue to get better, just so that
| these people who claim to take memory safety so seriously might
| shut the fuck up about it.
|
| lol idk kiiiinda sounds like you might be the one taking things
| too seriously?
| steveklabnik wrote:
| Rust has contributed a lot to LLVM, see here for a recent
| summary:
| https://twitter.com/pcwalton/status/1366058442276790274
| wizzwizz4 wrote:
| True, but most Rust zealots (including me, though I'm
| reformed) haven't contributed anything to LLVM.
|
| Most Rust developers aren't Rust zealots. It's largely people
| who are new to the language, I think.
| steveklabnik wrote:
| The original person said "evangelists" and then slid into
| "zealotry." And I find a lot of anti-Rust folks seem to
| think anyone who likes Rust is a "zealot." YMMV.
|
| I think that this is just a hard conversation to have, with
| a ton of different groups who all want different things,
| and have different incentives.
| nickysielicki wrote:
| I like rust. What I don't like is people who pretend that
| unsafe rust is any better than C++ or C.
| wizzwizz4 wrote:
| Unsafe Rust is harder to predict the behaviour of than C
| - at least, when you're doing completely off the wall
| stuff like re-using the stack in two threads. You can
| keep it contained, though, and so long as your unsafe
| Rust is keeping the language's invariants, the code is
| safe; therefore, you know where to start looking when
| there's trouble to be found.
|
| I think it depends on what you're trying to do.
| cbmuser wrote:
| What Rust needs is an alternative, gcc-based implementation
| similar to gccgo.
|
| This will solve the problem immediately and allow Rust to be
| used on a much greater variety of targets, including obscure
| targets such as Tricore which are used in the automotive
| sector.
|
| Only if Rust code runs everywhere, it will also be deployed
| everywhere.
| steveklabnik wrote:
| Yes, you and I have spoken about this a few times over the
| years :) Glad to see the m68k work is still ongoing.
|
| I too am pro getting Rust into gcc. We'll see how the
| effort goes.
| josefx wrote:
| And the moment that exists most of the issues the author
| currently ascribes to C will also be true for Rust.
| oivey wrote:
| Rust is on LLVM for a reason: implementing new languages on
| top of GCC is hard. It seems much more reasonable for
| unusual architectures to contribute to LLVM. If companies
| want to sell their own architecture, they should provide
| the support themselves, rather completely relying on free
| contributions from the OSS community.
| [deleted]
| alisonkisk wrote:
| Posted in wrong thread? This comment seems off topic.
| nickysielicki wrote:
| A great deal of this article is about C being a "cancer", "a
| perpetually unsafe development ecosystem", whose cancerous
| properties are enabled by the absolute horror of it being
| portable.
|
| The implied replacement is Rust.
| oivey wrote:
| At least part of the point was that C only appears to be
| portable. The abstract machine concept itself is leaky,
| lesser used compilers have serious bugs, build scripts
| aren't portable across architectures, dependency management
| is poor, etc.
| alisonkisk wrote:
| It's a "cancer" in the sense that it spreads everywhere and
| people use C software even in environments where it was
| never intended to work, and get hurt by it.
| woodruffw wrote:
| > A great deal of this article is about C being a "cancer",
| "a perpetually unsafe development ecosystem", whose
| cancerous properties are enabled by the absolute horror of
| it being portable.
|
| Author here: my sibling already explained the language, but
| I specifically chose "cancer" because I remember seeing the
| UNIX-haters handbook use that phrase (I'm not exactly a
| UNIX hater, but it's always stuck with me). I write C and
| C++ professionally (and I like it that way!), and I think
| it's perfectly fair and accurate to refer to their spread
| as cancerous.
|
| > The implied replacement is Rust.
|
| No. Rust was an example. The replacement is any memory-safe
| language; Rust just happens to have done a great job
| getting much of the tooling right from the get-go.
| timidger wrote:
| That is not a good reason for choosing such charged
| language. You can get your point across without comparing
| a venerable (if flawed) programming language to one of
| the most feared class of diseases that kills millions
| each year.
|
| I think there are good points you make that people should
| hear, but less people will listen if you turn them away
| by making grotesque comparisons.
| woodruffw wrote:
| > You can get your point across without comparing a
| venerable (if flawed) programming language to one of the
| most feared class of diseases that kills millions each
| year.
|
| I'm going to be pedantic with this: calling something "a
| cancer" is not comparing it to cancer. The phrase "X is a
| cancer" is bombastic and inflammatory, which is
| intentional. It's neither a simile nor grotesque, at
| least in my dialect of English.
|
| As for the actual point: I see no reason to venerate C.
| Nobody should labor under the false pretense that we, as
| an industry, are brilliant enough to reliably write safe,
| cross-platform C. We haven't managed to do it for the
| last 50 years and, given the current state of static
| analysis on C, I don't have any particular hope for the
| next 50. I'm going to keep on writing it, but I don't
| intend to venerate it or convey that expectation on
| anyone else.
| avereveard wrote:
| > Your user base is unhappy
|
| It isn't the owner user base if it comes from an unsupported
| third party build.
| Lammy wrote:
| While technically true, that doesn't mean users will understand
| that and send their bug reports to the right place. For some
| historical examples of maintainer burnout and community
| clashes, see QuodLibet vs Gentoo in 2005/2005:
| https://bugs.gentoo.org/101619 / https://bugs.gentoo.org/124595
|
| And GAIM (Pidgin) vs Gentoo in 2003
| (https://bugs.gentoo.org/35890), where we can see that nothing
| ever really changes, RE: "You don't happen to be using an
| architecture other than x86, do you?"
| woodruffw wrote:
| > It isn't the owner user base if it comes from an unsupported
| third party build.
|
| Maybe this is strictly true, but it isn't how the
| maintainer/packager model has historically worked: packagers
| _do_ point users to the upstream for troubleshooting, and the
| upstream points prospective users to the packagers for
| installation. It 's a marriage of convenience (and philosophy),
| but it doesn't imply the arbitrary patches and/or changes to
| the project itself are somehow supported by their upstreams.
| alisonkisk wrote:
| No. Package manager should never send a bug _reporter_
| upstream. It 's the package manger's job to investigate the
| issue and file a bug with upstream including proper context,
| unless the bug is reported on multiple distributions.
| thesuperbigfrog wrote:
| It seems like the easiest answer is to fork the cryptography
| library:
|
| - current maintainers and those who are on supported
| architectures can use the Rust implementation. The current
| maintainers no longer want to maintain the C implementation and
| that is their prerogative as this article describes.
|
| - new maintainers and those on unsupported architectures can
| continue to use the C implementation. Not everyone in the current
| user base (to include some distribution maintainers) is able to
| use the new Rust implementation at this time, but they still need
| the library.
|
| It's not ideal, but it seems like the only practical way ahead
| that meets everyone's needs.
| Chyzwar wrote:
| There is a high chance that there is no pyca/cryptography users
| on these niche platforms that Gentoo is trying to support. If
| there are any, they should pay for support not expect
| maintainers to support theirs fridge platform.
| viraptor wrote:
| > support theirs fridge platform.
|
| I love this typo in context of s390!
| JoeAltmaier wrote:
| Don't know about crypto. But any open source package, ported to a
| new environment (one not explicitly tested by the maintainter) is
| going to have issues. That's the nature of software.
|
| E.g. I just added pnglib to an embedded assembly. Has a display
| among other things. Wanted to put compressed PNG images into
| (limited) flash, then decompress into (plentiful) RAM at boot.
|
| Of course pnglib didn't build for my environment. Never mind it
| has a C library. There are 50+ compile switches in pnglib, and I
| was setting them in a pattern that wasn't part of the tested
| configurations. It didn't even compile. Once it compiled (after
| some source changes) it didn't run. More source changes and it
| would run until it hit some endianness issues. Fix those, and it
| would do (just) what I wanted, which was to decompress certain
| PNG image formats with limited features, once.
|
| No problem. That was my goal, achieved. But at no time did I
| blame the maintainers for not anticipating my use case.
|
| I would say this: maintainers, flag compile-time options that
| aren't tried both ways in your test environments. To give me some
| chance of estimating how hard my job is going to be.
| alisonkisk wrote:
| This yet another example of why gift economy members need to
| understand that _you don 't owe anyone support_. If you publish
| As-Is, thank you. If you promise to support the OS/Hardware in
| your lab, thank you. If you accept patches from users with weird
| use-cases, thank you.
|
| If you get too much bug spam, you need to set up filters and auto
| replies and volunteer helpers to help you find the reports you
| care about.
|
| _You don 't owe anyone support_.
| dfox wrote:
| I think that main issue in this is the pyca/cryptography library
| itself and its unfortunate name. It bundles three only marginally
| related things (high-level symmetric encryption API, X.509
| handling and bunch of cryptographic primitives) things into one
| library that is first result on google for "python cryptography".
|
| The end result of this is that another libraries which need only
| some small part (which is probably better provided by some other
| python crypto package) depend on the whole library.
| reidacdc wrote:
| I'm not hugely knowledgeable about LLVM, but it was my
| understanding that a major benefit of it was separating out high-
| level from low-level concerns.
|
| There are lots of practical issues, obviously, but the dream was
| that the LLVM intermediate representation would take the role of
| C in the author's description, with the major difference that
| nobody has to actually hand-write anything in it.
|
| So from that point of view, isn't the robust, long-term solution
| to this and related issues to build LLVM back-ends for all the
| "weird" architectures?
|
| I'm surprised it's not mentioned as a possible way forward.
| qbasic_forever wrote:
| Yes, the reality is all those weird architectures stopped being
| developed and actively supported 20 years ago. If someone wants
| to get some old Itaniums and Alphas and such together to hack
| in support, go wild and have fun. But the reality is the venn
| diagram interesection of people knowledgeable enough to create
| a LLVM backend, people motivated to support vintage
| architectures, and people that actually have the hardware
| available and ready to test is basically zero. If people in the
| community are passionate about it happening then organize and
| make it a reality.
| rodgerd wrote:
| It's also that - per the article - it goes beyond "willing
| and able to implement in LLVM". Because that doesn't let the
| Rust devs or the Python devs test on those vintage platforms.
| kps wrote:
| The problem is that LLVM IR is not stable across releases, so
| keeping an out-of-tree back end up to date would be expensive.
| [deleted]
| tom_mellior wrote:
| LLVM IR is not platform independent:
| https://releases.llvm.org/8.0.0/docs/FAQ.html#can-i-compile-...
|
| There is no "compile to LLVM once, run anywhere with an LLVM
| backend". If your C _frontend_ doesn 't know that you want to
| compile for System/390, it will not be able to generate LLVM
| code that you can expect to turn into a working System/390
| binary.
| Fordec wrote:
| I do think that we're going to see more of this. This is just a
| relatively early example. Rust and LLVM bring things to the table
| that make them inevitable if we as an ecosystem, on the whole,
| value privacy and security. This is where the rubber of the ethos
| hits the road. C for all it's good work and history, is a leaky
| mess and the source of so many zero-days, especially the bad bad
| state actor level ones.
|
| If we are to move to a more abstracted and safer system, the
| ideas behind LLVM are just going to be a fact of life in years to
| come. The solution to this is to either fund greater LLVM
| integration (or similar that isn't on the scene yet) or accept
| the status-quo. I choose the former, but I sincerely hope the
| future direction leaves as few out in the cold as possible
| through the effort of smart people. But protecting hobbyists is a
| stretch goal in my mind compared to improving the security and
| privacy of the global interconnected world we're in.
| titzer wrote:
| I thought very hard about this problem as I've developed Virgil
| [https://github.com/titzer/virgil] over the years. Bootstrapping
| any system, not the least of which a new programming language is
| a hard problem.
|
| Virgil has an (almost) hermetic build. The compiler binaries for
| a stable version are checked into the repository. At any given
| revision, that stable compiler can compile the source code in the
| repo to produce a new compiler binary, for any of the supported
| platforms. That stable binary is therefore a cross-compiler.
| There are stable binaries for each of the supported stable
| platforms (x86-darwin, x86-linux, JVM), and there are more
| platforms that are in the works (x86-64-linux, wasm), but don't
| have stable binaries.
|
| What do you need to run one of the stable binaries?
|
| 1. JVM: any Java 5 compliant JVM
|
| 2. x86-linux: a 32-bit Linux kernel
|
| 3. x86-darwin: a 32-bit Darwin kernel*
|
| [*] sadly, no longer supported past Mavericks, thanks Apple
|
| The (native) compiler binaries are statically-linked, so they
| don't need any runtime libraries, DLLs, .so, etc.
|
| Also, nothing depends on having a compiler for any other
| language, or even much of a shell. There is test code in C, but
| no runtime system or other services. The entire system is self-
| hosted.
|
| I think this is a decent solution, but it has limitations. For
| one, since stable executables absolutely need to be checked in,
| it's not good to rev stable too often, since it will bloat the
| git repo. Also, checking in binaries that are all cross-compilers
| for every platform grows like O(n^2). It would be better to check
| in just one binary per platform, that contains an interpreter
| capable of running the compiler from source to bootstrap itself.
| I guess I'll get to that at platform #4.
| breakfastduck wrote:
| Question from a point of ignorance - why would you target 32bit
| for something that is being actively developed?
|
| Are we not at a point where 64bit should be the expected
| target?
| jcelerier wrote:
| Webassembly for instance is a 32 bit target. So are most arm
| hobby boards (even if hw is 64 bit, they ship with 32 bit
| OSes)
| eqvinox wrote:
| A lot of "IoT" devices are 32bit. And there's no reason for
| them not to be. (There's 8bit ones too.)
| titzer wrote:
| I bootstrapped on the JVM first, and then the first native
| bootstrap was around 2011. I am almost finished with my
| x86-64-linux port.
| skybrian wrote:
| I'm wondering if bootstrapping from WebAssembly would make
| sense someday, under the assumption that everyone has a
| browser? (Though a stand-alone interpreter is preferable.)
| titzer wrote:
| That's not a bad long-term plan (if there is a lightweight
| standalone Wasm interpreter), but Wasm is not quite
| ubiquitous enough. Hopefully!
| SAI_Peregrinus wrote:
| Packagers often fork software, but confusingly don't give the
| fork a new name.
|
| If an upstream project includes build scripts, then they're part
| of the project and any changes to the build scripts constitute a
| fork. Any patches to the actual code are even more clearly a
| fork.
|
| The article lists the following 5 things packagers sometimes do:
|
| 1. Build your project with slightly (or completely) different
| versions of dependencies
|
| This doesn't necessarily require a fork. If you use static
| linking of dependencies it mostly does, but things like libc are
| often dynamically linked.
|
| 2. Build your project with slightly (or completely) different
| optimization flags and other potentially ABI-breaking options
|
| If you're providing a build script, and they're changing this,
| it's a fork.
|
| 3. Distribute your project with insecure or outright broken
| defaults
|
| If they're changing the provided code or configuration defaults,
| it's a fork.
|
| 4. Disable important security features because other parts of
| their ecosystem haven't caught up
|
| It's a fork.
|
| 5. Patch your project or its build to make it "work" (read:
| compile and not crash immediately) with completely new
| dependencies, compilers, toolchains, architectures, and
| environmental constraints
|
| It's a fork.
|
| Project packagers should rename projects they fork. Project
| authors should make it easy to rename forks, preferably with a
| single location that defines the project name.
|
| Forks aren't a bad thing. But not labeling them correctly leads
| to quite a bit of confusion.
| PurpleFoxy wrote:
| Debian used to do this for Firefox. It leads to confusion with
| users. "I want to install Firefox, why is there no Firefox in
| the repos".
| Spivak wrote:
| If you've ever poked around the spec files for Fedora, CentOS,
| or RHEL you know that this wouldn't work. Everything is
| extensively patched. You might as well just call everything
| rhel-$project. Some projects genuinely have a hundred separate
| patches.
|
| Linux distros simply wouldn't work without this. Once you have
| two pieces of software that depend on different versions of
| libfoo or two different versions of glibc it's over. Nix is
| doing heroic work in this space but it's also doesn't care that
| you have 12 different versions of zlib that in an enterprise
| world still need security backports because upstreams don't do
| that.
| dralley wrote:
| This is hardly unique to Fedora / CentOS / RHEL. Debian does
| the same, even more aggressively in some ways, such as
| breaking up libraries into many smaller independent packages.
| At that point it truly is a fork.
| wbl wrote:
| Static link to files in standard places and bring the bugs
| upstream.
| genuine_smiles wrote:
| > You might as well just call everything rhel-$project.
|
| Is this a bad option?
| macksd wrote:
| This is definitely the most reasonable case I've heard made from
| this argument, and it's changed my stance on the issue.
|
| That said, I feel the cryptography project handled this poorly. I
| encountered the issue when one day, installing a Python Ansible
| module the same way I did the day before required a compiler for
| a different language that I've never used before and that is
| hardly used, if at all, in the rest of my stack. The Ansible
| project seemed to be taken by surprise, and eventually I found
| the root upstream issue, where the initial attitude was "too
| bad". Some people making those comments worked for Red Hat / IBM,
| Ansible is heavily backed by that. What company cares more about
| Linux on S390 than Red Hat / IBM? I would suggest none. So the
| fact that they had this attitude and community reaction to me
| suggests the problem is not one of expecting free work for a
| corporation on a non-free architecture. It was IMHO a combination
| of a lack of forethought, communication, and yes, perhaps a
| change that is overdue and will just be a little painful. The
| suggestion to maintain an LTS release while people figure out how
| to proceed is the right move.
| steveklabnik wrote:
| > Some people making those comments worked for Red Hat / IBM,
| Ansible is heavily backed by that.
|
| Okay, so, I don't mean to pick on you here, but I've seen this
| sentiment cropping up a few times. Not everyone can be familiar
| with everything, but I would caution you against immediately
| assuming corporate politics are at the root here.
|
| Anyone familiar with Alex Gaynor and his work over the last few
| years would know that he cares about Python, and memory safety.
| That has remained consistent regardless of his employer.
| Immediately assuming that this has something to do with company
| politics, rather than just a tireless open source contributor
| working to improve the things that he cares about, for years,
| is making a bit of a category error, in my opinion.
| macksd wrote:
| I agree - I don't feel picked on :D I guess my point is that
| it's overly dismissive to say "no free work for companies on
| niche architectures" when projects backed by the same company
| were a bit blind-sided by this and a bunch of people lost
| time chasing down what happened and why. To me that's a sign
| that this isn't just a mismatch of ideology or someone
| wanting free work: it was a failure in communication and
| mismatched expectations. If it had just been on a major
| version bump, I wonder if we'd even be having this
| discussion.
| rodgerd wrote:
| Sure, but the message - that if it's a problem for Red
| Hat's Ansible team or Red Hat's mainframe Linux team, they
| should be doing the work needed to make it not a problem.
|
| Ansible Tower subscriptions aren't exactly cheap, and
| neither is RHEL s390x. If there's not the fat to be
| uplifting the core infrastructure needed to run RHEL or
| Ansible on Red Hat's products, that's most likely a choice.
| steveklabnik wrote:
| Glad to hear it :)
|
| I think it gets really hard the larger the company you talk
| about. Before I worked at bigger places, I assumed a _lot_
| more coherence than is actually the case.
| laserharvest wrote:
| Why is it necessarily Cryptography's fault that Python Ansible
| was taken by surprise? Or any of the other affected parties
| along the chain? That Cryptography was starting to include Rust
| in the project was announced on the mailing list by Gaynor last
| summer. And that email said exactly what future (at that time)
| release would require a Rust toolchain if the project was to be
| built from source.
|
| The reply to that by the Gentoo guy (who started the Github
| issue) was that package maintainers cannot follow every single
| mailing list of every dependency. That is debatable (e.g. maybe
| you should make an exception for security applications), but
| let's take that as a given for now. In that case, what is
| Cryptography to do? Where should they announce such things in a
| way that orgs like Gentoo will see it? And also _notice_ it and
| not just mentally gloss over it as some kind of "spam"? If the
| Gentoo guy didn't see it half a year ago, would he have seen an
| announcement (or the reminders) if it was made five _years_
| ago?
| rodgerd wrote:
| > The reply to that by the Gentoo guy (who started the Github
| issue) was that package maintainers cannot follow every
| single mailing list of every dependency.
|
| It seems reasonable that, if you are the packager for
| critical packages, that you follow critical dependencies?
|
| If the problem is that the distro is supporting so many
| things that the folks working on it can't keep up - well,
| that's precisely the author's point: stop pretending that you
| can support HPPA and MIPS or whatever as well as you can
| support x86_64. But you don't get to tell a million people
| that they have to have a less secure Python because 3 people
| have a toy in a closet they want treated as a first class
| citizen.
| klyrs wrote:
| Folks with an eye towards backwards compatibility typically
| don't implement breaking changes without a year of
| deprecation warnings emitted by the software. Contrast that
| to a notice posted in a sub-basement 6 months before
| instituting a breaking change. The shock and alarm do seem
| warranted IMO.
| detaro wrote:
| How do you do a deprecation warning for a new language
| being a build dependency? What does it look like? "your
| build environment is being deprecated"?
| tom_mellior wrote:
| In C you can implement a "your build environment is being
| deprecated" message like this: #ifndef
| I_UNDERSTAND_THAT_SOON_RUST_WILL_BE_MANDATORY
| #error "Starting with version xxx this package will need
| Rust to compile. Recompile with
| -DI_UNDERSTAND_THAT_SOON_RUST_WILL_BE_MANDATORY to
| acknowledge that you understand this warning."
| #endif
|
| Anyone building from source would get notified in a way
| that's impossible to miss but easy to turn off.
| Twirrim wrote:
| > project mailing lists, Github issue tracker, project IRC
| channel, documentation (changelog, FAQ), and Twitter
| channels
|
| It's hardly a notice posted in a sub-basement 6 months
| before instituting a breaking change. They communicated via
| numerous mechanisms.
| klyrs wrote:
| It's a sub-basement from the perspective of folks for
| whom this is a second-order dependency. And it sounds
| like there are many more of them than there are folks
| that caught wind of this.
|
| Regardless. The common practice is a _year_ of emitting
| warnings from the software, that the end-user will see.
| That 's prominent and allows package maintainers enough
| time to work around the upcoming breakage. Six months
| notice on a mailing list is simply not prominent enough,
| and it's half the standard year which puts undue strain
| downstream.
| [deleted]
| macksd wrote:
| I think making a change like this warrants a major version
| bump. It won't eliminate all the surprise for everyone, and I
| do have sympathy for the people who did go out of their way
| to talk about this on the mailing list and then surprised
| everyone. But it's common to pin yourself to a minor or
| maintenance release line to automatically pick up security
| fixes, etc. I expect breakage when changing the major (or
| even minor), and that's almost always a manual upgrade. And
| that's when I _do_ read all the release notes, run tests,
| etc. before committing to the change.
| chrisoverzero wrote:
| What is a "major version bump"? Before you answer, consider
| that the library doesn't use semantic versioning. Before
| this all blew up, the versioning scheme was this[1]:
|
| > Given a version cryptography X.Y.Z,
|
| > - X.Y is a decimal number that is incremented for
| potentially-backwards-incompatible releases.
|
| > - - This increases like a standard decimal. In other
| words, 0.9 is the ninth release, and 1.0 is the tenth (not
| 0.10). The dividing decimal point can effectively be
| ignored.
|
| > - Z is an integer that is incremented for backward-
| compatible releases.
|
| The system has since changed, but it continues not to be
| semantic versioning. (It's effectively the same, in fact,
| but protects against dependents who think it is semantic.)
|
| By that scheme, it was already a "major" (signifying
| potential backwards-incompatibility) release.
|
| [1]: https://cryptography.io/en/latest/api-
| stability.html#previou...
| AaronFriel wrote:
| This reads to me like an argument for semantic
| versioning, because otherwise I need to internalize the
| rules of every package and know that some will break
| compatibility on the Y, some on the Z, ... Etc.
| chrisoverzero wrote:
| You know, I had that thought while writing it.
|
| I wouldn't want to force any particular versioning scheme
| on any particular developer, but maybe the "SemVer
| facade" versioning scheme they switched to is the best
| compromise. It has defensive value, at least.
|
| Then again, PEP 440 has nothing to say about the
| semantics of versioning, only requiring:
| [N!]N(.N)*[{a|b|rc}N][.postN][.devN]
|
| PyPA themselves describe various expected versioning
| schemes, but listing Semantic as preferred[1]. If I
| squint, I can fit `cryptography`'s previous scheme into
| "Hybrid". The biggest lesson I take from this is that if
| your version scheme isn't SemVer, work hard to make it
| look _obviously different_ from SemVer.
|
| [1]: https://packaging.python.org/guides/distributing-
| packages-us...
| detaro wrote:
| Would SemVer even strictly require a jump here? They
| didn't change what's usually thought of as the API of the
| library (i.e. if I don't compile it myself I don't really
| notice the change?), and that's what SemVer uses: "MAJOR
| version when they make incompatible API changes,"
| chrisoverzero wrote:
| Also a great point!
|
| I can't think of a clean alternative other than coming
| full-circle back to the "developer advocacy" solution,
| with its clear problems. Someone smarter than I am
| probably has it in the palm of their hand, though.
| M2Ys4U wrote:
| Even SemVer doesn't help here, as changing the _build_
| toolchain isn 't generally considered an API breakage
| (after all, the resulting binaries are API compatible)
| pornel wrote:
| The switch hasn't been sudden. It's just that many levels of
| disconnect between the project authors and downstream users
| meant it was basically impossible to communicate to all the
| affected users -- nobody looks at their deps-of-deps until they
| break.
|
| And they've released Rust as optional component you can
| disable, precisely because nobody paid attention until it
| actually shipped.
| masklinn wrote:
| > The switch hasn't been sudden. It's just that many levels
| of disconnect between the project authors and downstream
| users meant it was basically impossible to communicate to all
| the affected users -- nobody looks at their deps-of-deps
| until they break.
|
| Exactly, from the original shitstorm issue:
|
| > Rust bindings kicked off in July 2020 with #5357. Alex
| started a thread on the cryptography developer mailing list
| in December to get feedback from packagers and users. The FAQ
| also contains instrutions how you can disable Rust bindings.
|
| > Do you have constructive suggestions how to communicate
| changes additionally to project mailing lists, Github issue
| tracker, project IRC channel, documentation (changelog, FAQ),
| and Twitter channels?
|
| At one point there's sadly not much the project can do and
| still make progress.
| its-summertime wrote:
| > > Do you have constructive suggestions how to communicate
| changes
|
| That one kinda got me, when python intentionally has a
| runtime developer-to-developer communication system, per
| say:
|
| https://docs.python.org/3/library/warnings.html
| hedora wrote:
| I've noticed I can predict a lot about the quality of code an
| engineer produces by listening to their opinions on testing in
| strange environments.
|
| People interested in stamping out undefined / nondeterministic
| behavior / compiler warnings welcome bug reports from platforms
| that find new bugs.
|
| They also are more conservative about including dependencies, and
| their code is much, much more maintainable over time.
| alisonkisk wrote:
| Someone dedicated to debugging complex software on strange
| environments probably writes no code at all, since they spend
| all their time investigating bugs.
|
| Your comment is rudely dismissive of people giving you the
| product of their labor for free.
| asguy wrote:
| Do you disagree with their personal experience? Does it not
| match yours?
|
| Their experience matches mine. It's one of the reasons I
| appreciate projects like Net and OpenBSD, who actively keep
| old hardware around. They find bugs, by actually exercising
| their code base in unplanned ways (e.g. endian differences,
| alignment constraint differences).
| sgift wrote:
| It doesn't match mine since there's usually far more work
| to do than time available to do it. Most projects have to
| heavily triage bug reports and "your program fails on my
| platform that no one has built in 30 years" has a very low
| rate of "helpful to anyone else but the bug reporter".
| asguy wrote:
| Do you never stop and think "why is our code base busted
| in this person's environment?" I mean, it could just be
| that the bug filer just hasn't done their due diligence
| on upgrading (why won't win 3.11 play crysis).
|
| It also could be that your code quality sucks and you're
| writing something non-portable.
| egil wrote:
| But why would the code quality suck if features or
| maintenance of supported systems was prioritized ahead of
| arcane architectures? Time is a limited resource, so why
| should extreme portability be considered the holy grail?
| Lvl999Noob wrote:
| IMO, you both might be correct. A programmer welcoming of bug
| reports from other platforms and spending time solving them
| wouldn't write a lot of code, but the code they do write,
| would probably be very maintainable (unless the fixes were
| completely different code paths for different platforms).
___________________________________________________________________
(page generated 2021-02-28 23:00 UTC)