[HN Gopher] Supporting Linux kernel development in Rust
___________________________________________________________________
Supporting Linux kernel development in Rust
Author : dochtman
Score : 271 points
Date : 2020-08-31 18:54 UTC (4 hours ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| puzzledobserver wrote:
| One of the challenges to building an entirely new kernel is the
| vast amount of hardware support in the form of drivers and the
| features the existing kernel exports to userland in the form of
| syscalls.
|
| Would it be possible for a hypothetical new kernel, presumably
| written in Rust, to run an existing kernel such as Linux in a VM,
| just to tap into its drivers and emulate its syscalls? As more
| drivers are ported to the base kernel, the reliance on the donor
| kernel would shrink over time.
|
| Edit: This was an aside. Of course I realize that the conference
| was about adding Rust code to the Linux kernel, and not about
| rewriting any large, important, and functional codebase in
| language-of-the-day(tm). Hence my speculative language:
| "hypothetical" new kernel, "presumably" written in Rust.
| cmckn wrote:
| This conference talk considered a re-write out-of-scope; they
| were only discussing how new code could be written in Rust.
|
| Interesting idea, though!
| dmitrygr wrote:
| in some ways, this is how Service Console works in vmware ESX:
| https://en.wikipedia.org/wiki/VMware_ESXi
| wkz wrote:
| Xenomai uses this model, I believe. It is written in C, but I
| don't see why you couldn't do something similar with Rust.
| rphln wrote:
| I might be wrong, but isn't there already a precedent for
| something similar in the form of the XNU kernel? From a cursory
| glance, it is basically a BSD and a Mach kernel running side by
| side.
| ColanR wrote:
| I was thinking about something similar recently - if the
| hardware drivers could somehow be abstracted from the rest of
| the linux kernel (as if...), then suddenly a ton of
| experimental kernels could have widespread hardware
| availability.
|
| It would probably shake up the OS ecosystem to no small degree,
| but I imagine the fresh ideas that could be experimented with
| by non-experts would be hugely beneficial. Isn't that the big
| argument against so many new projects? "It looks cool, but the
| hardware support isn't there."
|
| FreeBSD might be a better target though, since that's
| specifically designed to be compatible with everything.
| alex7o wrote:
| This exists and is called rumpkerbels if I remember
| correctly.
| puzzledobserver wrote:
| rumpkerbels? Mind sharing a link? Google Search is
| curiously empty.
| steveklabnik wrote:
| They typo'd https://en.wikipedia.org/wiki/Rump_kernel
| JoshTriplett wrote:
| The point of this LPC session was how to incrementally
| introduce Rust in the existing Linux kernel, with all its
| existing drivers and syscalls and similar. A rewrite would
| indeed be a daunting and problematic proposition.
|
| There are kernels written in Rust, such as Redox, but those are
| separate projects, and that's not what this conference session
| or article are talking about.
|
| Standing reminder: the Rust project is not a fan of rabid Rust
| over-evangelism (e.g. "Rewrite It In Rust", "Rust Evangelism
| Strike Force"), and sees it as damaging and unhelpful. We
| discourage that kind of thing wherever we see it. We're much
| more in favor of a measured, cautious approach.
| aspaceman wrote:
| > the Rust project is not a fan of rabid Rust over-evangelism
| (e.g. "Rewrite It In Rust", "Rust Evangelism Strike Force"),
| and sees it as damaging and unhelpful.
|
| I was really caught off guard by this stuff recently. I was
| around the rust subreddit a lot back in 2013-2015 and never
| saw stuff like that. Stepped out while working on other stuff
| and now it seems common. Super strange.
| progval wrote:
| As an occasional rust programmer: you don't see this kind
| of behavior from the rust community. You see it when a
| random person opens an issue on a bugtracker you are
| subscribed to. Then it's usually prompty closed by a
| maintainer because the issue is very low effort, and the
| author doesn't offer to do any work themselves.
|
| Example:
| https://gitlab.com/fdroid/fdroidclient/-/issues/1049
| ATsch wrote:
| I'm not sure why you'd need a VM for this. It should be much
| easier to implement bridges for major kernel apis like
| character devices, as I understand some BSDs already do?
| chromedev wrote:
| Pretty soon they'll be arguing for inclusion of JavaScript in the
| Linux kernel
| ogoffart wrote:
| > The ubiquitous kmalloc() function, for instance, is defined as
| __always_inline, meaning that it is inlined into all of its
| callers and no kmalloc() symbol exists in the kernel symbol table
| for Rust to link against. This problem can be easily worked
| around -- one can define a kmalloc_for_rust() symbol containing
| an un-inlined version but performing these workarounds by hand
| would result in a large amount of manual work and duplicated
| code.
|
| That's why there exists crates like https://docs.rs/cpp/ which
| allow to embed C++ (and thus C) directly into the rust source
| code, simplifying the manual work and reducing the duplicated
| code.
| The_rationalist wrote:
| It's not about safety, if that was the real argument we should
| start with obvious improvements like checked C
| https://github.com/microsoft/checkedc The C++ lifetime checker or
| Cyclone
| geofft wrote:
| Checked C addresses a small number of memory safety problems -
| it addresses bounds checking / buffer overflows, but not use-
| after-frees, double frees, data races / insufficient locking,
| and several other things that fall into the general category of
| "memory safety," nor more general "safety" things like type
| confusion.
|
| The kernel is not written in C++ and is unlikely to use C++,
| for many reasons - Rust is a more natural C-but-better for the
| kernel than C is.
|
| Cyclone hasn't been maintained for over a decade; it was a
| research language, and it largely served its purpose as a
| research language in inspiring production-supported languages
| like Rust.
| boogies wrote:
| Probably a crazy unpopular opinion, but I kind of agree with the
| Hyperbola (GNU / formerly Linux-libre, in future using a fork of
| the OpenBSD kernel https://itsfoss.com/hyperbola-linux-bsd/) devs
| that Rust in Linux is not necessarily a good thing, for the
| reasons they mentioned when they announced they were switching
| kernels (https://www.hyperbola.info/news/announcing-hyperbolabsd-
| road...) (Edit: and the LLVM dependency).
| badrabbit wrote:
| Maybe a GNU Rust compiler without the trademark is in order?
| The state of KSPP is not nice as well. I wish Ada was more
| popular. From what I know, it has much of the security
| guarantees rust has and it has been battle tested.
| guerrilla wrote:
| I'm not sure, but aren't they saying that they wouldn't be
| able to call it "GNU Rust" [1] in the same way they can't
| call IceCat "GNU Firefox"?
|
| 1. https://wiki.hyperbola.info/doku.php?id=en:main:rusts_free
| do...
| homarp wrote:
| well, plenty of names to play with:)
|
| GNR: Gnr Not Rust
|
| TRust: Treated Rust
|
| ...
| gpm wrote:
| How do you pronounce GNR? It think it needs to be a
| vowel, or an R.
|
| Some attempts: GRR: Gnu Renames Rust, GAR: Gnu
| assimilates rust, GER: Gnu non Est Rust (gnu is not rust,
| but in latin to get an E), GIR: Gnu Isn't Rust, GOR: Gnu
| Oxidizes Rust, GUR: Gnu Unseats Rust.
| el_oni wrote:
| Or something like "oxide", ferrite, maGNUtite. Play with
| iron ores or compounds
| badrabbit wrote:
| They can call it something else entirely. Rusty? Brown? Who
| cares so long as it is compatible with rustc
| steveklabnik wrote:
| If it is compatible with rustc, then we would probably
| approve a request to use the trademark.
| badrabbit wrote:
| They don't want to have to ask.
| steveklabnik wrote:
| Yes, I understand that. Just trying to make it clear that
| the issue is with that itself, not on our side.
| JoshTriplett wrote:
| The vast majority of the text in that post is inaccurate and
| hyperbolic. Linux is not "forcing adoption of HDCP" (it's just
| something Linux has a driver for), Rust does not require
| internet access to use (and the trademark comments are not
| particularly accurate either), the Kernel Self Protection
| Project is alive and well and making new improvements in every
| new version of the kernel (see Kees Cook's regular blog posts
| for details), systemd is not spelled "SystemD", gettext has no
| dependency on Java (and the link _in that post_ explains that).
| And in general, decisions about what technologies to use are
| made by those who show up and do the work to build viable
| solutions, not by people who snark.
|
| It's a rant by an obscure one-developer distribution that
| _formerly_ used the Linux kernel. Some people are never happy
| no matter what you do, and if you make the mistake of treating
| their rants as useful feedback, you get a list of 30 more
| demands before they 'd consider gracing your little project
| with the honor of their all-important usage again.
| [deleted]
| ColanR wrote:
| I think you were being "inaccurate and hyperbolic".
|
| > Linux is not "forcing adoption of HDCP" (it's just
| something Linux has a driver for)
|
| What they actually said in the article:
|
| > Historically, some features began as optional ones until
| they reached total functionality. Then they became forced and
| difficult to patch out. Even if this does not happen in the
| case of HDCP, we remain cautious about such implementations.
| JoshTriplett wrote:
| My comment was entirely based on the second link. Direct
| quote from that:
|
| > Linux kernel forcing adaption of DRM, including HDCP.
|
| (Which linked to a patch adding driver support for handling
| HDCP, with absolutely no possibility of it somehow being
| forced.)
|
| Regarding the quote from that interview: sure, and there's
| also no guarantee that the Linux kernel won't drop support
| for every architecture except SPARC, except of course for
| all the people involved not putting up with it. In what
| possible universe would Linux developers decide it was a
| good idea to mandate HDCP? It's completely unfounded and
| unsupported hyperbole and fearmongering, without even a
| hint of potential truth.
|
| It's also roughly consistent with what I'd expect from
| someone who thinks that gettext daring to have support for
| Java and C# format strings is a horrific problem that
| should be ripped out by the roots.
| boogies wrote:
| I don't know about Linux, but Mozilla has mandated many
| things that I would have hoped it never would have, like
| XUL depreciation, and especially added many anti-features
| to their largest product, deciding each was not their
| hill to die on
| https://news.ycombinator.com/item?id=24124954. I left
| Firefox and have never regretted it. The Hyperbola devs
| did the same, and I will be happy to try their OS and
| support them fighting on every hill they find, no matter
| how tiny or unimportant it seems to larger organizations.
| JoshTriplett wrote:
| > like XUL depreciation
|
| There were specific reasons for that:
| https://yoric.github.io/post/why-did-mozilla-remove-xul-
| addo...
| boogies wrote:
| I was going to comment similarly. The comment could be
| described as "inaccurate and hyperbolic": if spelling
| counts, the second link does not say Linux was "forcing
| _adoption_ [emphasis added] of HDCP." Systemd stinks and I
| don't care how it's spelled as long as it's sent to ...
| nvm. And the Hyperbola devs have shown up, done the work,
| and ported pacman to their kernel, not making the mistake
| of listening to hyperbolic haters.
| Foxboron wrote:
| > And the Hyperbola devs have shown up, done the work,
| and ported pacman to their kernel, not making the mistake
| of listening to hyperbolic haters.
|
| Where? I was curious as there has been no upstream
| improvements. And there are no new code in what I assume
| is their upstream repository?
|
| https://libregit.org/Hyperbola/hyperman
| Macha wrote:
| The licensing complaints about Rust apply equally to Python,
| which they package without objection:
| https://www.hyperbola.info/packages/extra/x86_64/python/
| cylon13 wrote:
| Not exactly: [0]
|
| > Some users have correctly mentioned that many other
| software packages have trademarks, do we plan to remove them
| all? No. We are not against all trademarks, only those which
| explicitly prohibit normal use, patching, and modification.
|
| > As an example, neither Python PSF nor Perl Trademarks
| currently prohibit patching the code without prior approval.
| They do prohibit abuse of their trademarks, e.g. you cannot
| create a company called "Python", but this does not affect
| your ability to modify their free software and/or apply
| patches.
|
| > Due to the anti-modification clause, Rust is a non-
| permissive trademark that violates user freedom.
|
| [0] https://wiki.hyperbola.info/doku.php?id=en:main:rusts_fre
| edo...
| jcranmer wrote:
| I suspect whomever wrote that hasn't actually read the
| trademark policies in detail, or conferred with lawyers
| about it. Neither Python's nor Perl's agreements explicitly
| cover the case in detail, but Perl does mention this:
|
| > Licensed redistributors of Perl code are permitted by
| their licenses to use the Perl logo in connection with
| their distribution services, on product packaging, and in
| promotional materials.
|
| IANAL, but that text would especially suggest to me that
| you would have _no_ right to use the trademark even to
| distribute an unmodified binary of Perl without approval.
|
| I suspect the _real_ reason is Debian and Mozilla had a
| very public, well-known spat over licensing, and Python and
| Perl have not had very public, well-known spats with
| anybody.
| Macha wrote:
| Well, good luck calling your modified Python release
| Python: https://github.com/naftaliharris/tauthon/issues/47
| ArchOversight wrote:
| Once again the GPL camp takes and relicenses a project so that
| further improvements they make can't be used by the original
| authors.
|
| Shame really.
| boogies wrote:
| They can be used by the original authors, if the original
| authors stop intentionally licensing their code so weakly
| that anyone can make changes that no one else can even see.
| bitwize wrote:
| I'd be more concerned about the fact that the Rust Evangelism
| Strike Force is hellbent on getting everybody to depend on
| their exhausts-all-32-bit-address-space-to-build compiler and
| toolchain -- _in addition_ to whatever toolchain the project
| currently uses, at least as long as the strangler pattern is
| being applied to migrate the code base to 100% Rust. GNU
| software in particular has always been such that the core stuff
| is bootstrappable from just C.
|
| But it seems, from an RESF standpoint, that the fact that any C
| code is out there running at all is a crisis-level problem.
| JoshTriplett wrote:
| Standing reminder: the Rust project is not a fan of rabid
| Rust over-evangelism (e.g. "Rewrite It In Rust", "Rust
| Evangelism Strike Force"), and sees it as damaging and
| unhelpful. We discourage that kind of thing wherever we see
| it. We're much more in favor of a measured, cautious
| approach.
| TylerE wrote:
| I don't see how that conplaint is at all meaningful to kernel
| development.
|
| It's typical GNU/Pendantry.
|
| The kernel will not be shipping a rust compiler, patched or
| otherwise.
| boogies wrote:
| The article mentions possible ABI compatibility challenges
| and I also think it's not a good thing for a language in such
| critical applications to only have a single major
| implementation.
| eptcyka wrote:
| The linux kernel still can only be compiled with GCC. Of
| course, it's a single patch away from clang compatibility,
| but the kernel has never been compilable with much else but
| GCC.
| jcranmer wrote:
| If I'm understanding correctly, the only reason for not using
| Rust is... Rust is trademarked?
| dijit wrote:
| I'm generally in support of rust in the kernel, but in the
| interest of playing devils advocate: Would adding more and
| more dependence on rust mean that it ties the kernel to a
| single set of compilers?
|
| I can see why people would be opposed to that.
|
| I'm thinking about LLVM (and, obviously, rustc)
| Silhouette wrote:
| _Would adding more and more dependence on rust mean that it
| ties the kernel to a single set of compilers?_
|
| Like GCC?
|
| Your point is certainly valid, but I'm not sure it's a new
| problem.
| JoshTriplett wrote:
| Linux currently only compiles on GCC and Clang, and the
| latter is a recent development. rustc uses LLVM as a
| backend, just like Clang, and people are working on other
| backends. Linux maintainers didn't express any concerns
| about this being an issue.
| danudey wrote:
| The Linux kernel would only compile with GCC for...
| decades? I think? So it's not really a new thing. It would
| be really nice if the kernel did have first-class support
| for LLVM and Clang directly (rather than having to set a
| bunch of environment variables to set the compiler, the
| linker to use with that compiler, various variables, etc.
|
| That said, it would certainly be uncomfortable to move from
| "it compiles with Clang, though it's a pain, or GCC, which
| works fine" to "it compiles with GCC, if you don't use any
| of this functionality, or LLVM only, if you want any of
| these features or anything that depends on them".
| JoshTriplett wrote:
| Originally, we _thought_ that Linux maintainers were
| asking us to only support kernels compiled with Clang,
| because they didn 't want to support code built with two
| different compilers. However, in the session, Greg Kroah-
| Hartman specifically said that if it _works_ to build
| Rust code with the LLVM-based rustc and C code with GCC
| and link the two together, and there aren 't any problems
| in practice, then that's perfectly fine.
|
| So, we're currently expecting that supporting Rust will
| not place any requirements on the C compiler you use,
| unless you want to use cross-language LTO (which we'll
| eventually want to do, but it won't be a hard requirement
| for a working kernel).
| jcranmer wrote:
| > unless you want to use cross-language LTO (which we'll
| eventually want to do, but it won't be a hard requirement
| for a working kernel).
|
| Does Linux even work with LTO right now? It seems to do a
| lot of weird-linking-magic things that I wouldn't expect
| to work well with LTO.
| JoshTriplett wrote:
| At the moment, I believe the Linux kernel can do LTO with
| some additional patches, but there are still regular
| issues. It isn't nearly as well supported as Clang or LLD
| yet.
| AquaLineSpirit wrote:
| There is the mrustc compiler
| https://github.com/thepowersgang/mrustc
|
| It's not fully complete. But there is no reason why it
| couldn't be.
| AquaLineSpirit wrote:
| Also it would be kind of funny. Where the rust parts can
| only be compiled by the llvm based rust compiler, but
| clang(llvm) can't build the C parts. And gcc can only
| build the C parts but not the rust parts :)
| boogies wrote:
| Mozilla's trademark policies have been annoying enough in the
| past that Debian used to ship a fork with the branding
| stripped. Hyperbola continued this practice for a while, but
| stopped for a variety of reasons in the It's Foss interview,
| and switched to a UXP-based browser. I'm writing this from a
| UXP-based browser because I find it much better than Firefox
| and I dislike Mozilla in general for more than one reason.
|
| But no, there are other reasons outside of trademarks, some
| technical ones are in submisson's article.
| est31 wrote:
| Wait till we tell them that Linux is a trademark of Linus
| Torvalds.
| captn3m0 wrote:
| it involves the "patches-must-be-approved-by-upstream" to
| still be called Rust clause that Rust inherits from the
| Mozilla Trademark policy. Its the same policy that led to
| the creation of IceWeasel/Waterfox etc.
| steveklabnik wrote:
| The root issue with Iceweasel was the logo, in my
| understanding. It was also resolved years ago. Rust does
| not inherit these problems, and even then, Mozilla will
| not be holding these trademarks anymore, even though I do
| not expect our policy will change once ownership changes.
|
| We do also, you know, give permission.
| msla wrote:
| What kind of performance hit is rewriting part of the kernel in
| Rust going to entail when it comes to compiling the kernel? How
| many platforms will cease to be able to compile their own
| kernels? The article says there will be some, but doesn't give
| numbers.
|
| More worryingly, the article also says that only some platforms
| will be "Rusted" in this manner, splitting the codebase into
| "mainline Rust" and "obscure platform C" as far as these sections
| are concerned; this seems like it will fracture the development
| community between people who know Rust and those who don't, and
| those who can work on mainstream platforms and those who can't.
| Fewer eyes on this code seems like the wrong way to go.
| Thaxll wrote:
| Looks like there is a lot of work involved to have anything done,
| remind me the issues that the Chrome team said 2 weeks ago.
| stmw wrote:
| Can you elaborate re: issues and lots of work required? I think
| it's generally to be expected that non-C language support in
| the kernel would be hard...
| muricula wrote:
| Here's the article the parent mentioned in passing, it may
| answer your question: https://www.chromium.org/Home/chromium-
| security/memory-safet...
|
| It's very Chrome and C++ specific, but the problems will
| likely be similar in practice.
| stmw wrote:
| Ah, right, I'd read that - but it's for C++, so it's a bit
| different than the kernel situation, and I actually had a
| much more positive reaction to it ("wow, seems like
| C++<->Rust is actually pretty far along"). Guess both are
| reasonable takeaways.
| gpm wrote:
| There is no doubt a lot of work here, but C <-> Rust
| interoperability is a lot cleaner than C++ <-> Rust.
|
| And for what it's worth, for almost all langauges C <-> X
| interoperability is a lot cleaner than C++ <-> X. Since almost
| all languages (including rust) "speak C", both in terms of
| compiler support and and being able to map every important C
| concept to a concept in their language. The same isn't true
| with C++.
|
| One thing to worry about with adding any language (including
| rust) to the kernel, is that the next good looking language
| will probably interoperate well with C and not interoperate
| well with Rust. So there is a much higher cost the second time
| you try to add a language.
| JoshTriplett wrote:
| > One thing to worry about with adding any language
| (including rust) to the kernel, is that the next good looking
| language will probably interoperate well with C and not
| interoperate well with Rust. So there is a much higher cost
| the second time you try to add a language.
|
| This is something that Rust is acutely aware of. There have
| been many discussions about the idea of a higher-level "safe
| ABI", such that languages with notions like counted strings
| or bounded buffers could interoperate with each other without
| having to go by way of an unsafe C interface.
| ender341341 wrote:
| Most of the chrome teams issues stemmed from C++'s lack of an
| ABI, not wanting to deal with using unsafe around the C++
| interfaces & not wanting to have to annotate c++ to export it
| to rust.
|
| I'd say the biggest questions for Linux that may be blocking
| here relate to LLVM not being available on nearly as many
| platforms as GCC (many of which will likely never be supported
| cause they're legacy systems that are only supported in the
| sense that they aren't purposefully broken) which limits it's
| use in any core-ish kernel systems.
| mrtweetyhack wrote:
| I'd rather see a complete rewrite instead
| ghostwriter wrote:
| I won't get tired of repeating the same comment in every topic
| that suggests Rust to be a great replacement of C in _existing
| projects_ , that it isn't.
|
| The safety guarantees that Rust provides are neither unique nor
| complete, and if we discuss the amount of effort necessary for
| bringing new interfaces like the one this article mentions ("one
| can define a kmalloc_for_rust() symbol containing an un-inlined
| version"), we should compare it to other existing solutions, like
| ATS [1], that are designed to be seamlessly interoperable with C
| codebases without giving up on the safety side of the argument.
|
| Just a week ago there was a link in ATS reddit channel that
| advertised ATS Linux [2][3] initiative, that may be of great
| interest for the same people who are interested in bringing more
| memory sefety to Kernel development, without giving up on
| existing C interfaces and the toolchain[4].
|
| [1] http://www.ats-lang.org/Documents.html#INT2PROGINATS
|
| [2] https://www.reddit.com/r/ATS/comments/ibyczp/ats_linux/
|
| [3] http://git.bejocama.org
|
| [4] http://ats-
| lang.sourceforge.net/DOCUMENT/INT2PROGINATS/HTML/...
| kbenson wrote:
| > I won't get tired of repeating the same comment in every
| topic that suggests Rust to be a great replacement of C in
| existing projects, that it isn't.
|
| That's not what this is about, so even though you aren't tired
| of posting this, I'm not sure it's warranted.
|
| This is about the kernel supporting other languages in select
| places where it makes sense, such as kernel modules, where
| there's already an API in place to conform to. It's also not
| limited to Rust, it's about providing support for multiple
| other languages that could be beneficial in those locations.
|
| > I won't get tired of repeating the same comment in every
| topic that suggests Rust to be a great replacement of C in
| existing projects, that it isn't.
|
| Nobody is giving up on existing C interfaces. They are simply
| making sure those interfaces are not actively hostile to
| languages that aren't C by nature of how they are implemented
| in C.
|
| To be clear, the "kmalloc_for_rust" was mentioned as one
| possible way forward. Another would be a more advanced Rust
| bindgen tool that can determine how to deal with it
| automatically. Neither affect ATS, based on your description
| (although a "kmalloc_for_extern" might generally help any non-C
| language, which would probably be beneficial).
|
| If ATS doesn't require any changes to C to interoperate with
| it, this whole announcement should have zero impact on ATS,
| other than possibly making it easier to get an ATS module in-
| kernel since they're trying to make it easier in general for
| non-C modules.
|
| You can treat this as a zero-sum gain, where Rust or Go's gain
| is ATS' loss, in which case you might as well pack up your bags
| now, since the writing is on the wall based on publicity, or
| you can treat it as a rising tide raises all boats type
| situation, where more general acceptance of alternatives to C
| is still beneficial to ATS actually being allowed in-kernel,
| even if it doesn't play exactly to ATS' strengths. I know which
| one I would put my money behind as being a better strategy in
| the end.
| ghostwriter wrote:
| > It's also not limited to Rust, it's about providing support
| for multiple other languages that could be beneficial in
| those locations
|
| But what are these benefits that other languages bring on the
| table? From the article it appears that proponents of the new
| toolchain mention memory-safety as the primary concern:
|
| > They focused on security concerns, citing work showing that
| around two-thirds of the kernel vulnerabilities that were
| assigned CVEs in both Android and Ubuntu stem from memory-
| safety issues. Rust, in principle, can completely avoid this
| error class via safer APIs enabled by its type system and
| borrow checker.
|
| So the natural follow-up question to that concern, it seems
| to me, would be whether there's something in the existing
| C/GCC toolchain (or around it) that can resolve the raised
| concerns without adopting a whole new toolchain (Rust/LLVM)
| and without having to extend the existing interface with
| language-specific suffixes like "kmalloc_for_rust".
| bsder wrote:
| > So the natural follow-up question to that concern, it
| seems to me, would be whether there's something in the
| existing C/GCC toolchain (or around it) that can resolve
| the raised concerns without adopting a whole new toolchain
| (Rust/LLVM)
|
| The answer to this question appears to be: no.
|
| Memory errors sit somewhere around 2/3 to 3/4 of errors
| irrespective of codebase when C is involved (this has been
| consistent in things like the Linux kernel as well as the
| Windows codebase). C++ hasn't seemed to have changed that,
| either.
|
| Remember, Rust _IS_ Mozilla 's answer to this question.
|
| > without having to extend the existing interface with
| language-specific suffixes like "kmalloc_for_rust".
|
| The requirement to do that is pointing out the that kernel
| API is poor and is too intimately tied to the build chain.
| We have seen this before--when Linux got ported to Alpha,
| for example.
|
| The point is to fix the API so that more languages than
| just Rust can be used.
|
| I suspect that more "system programming" languages may
| appear once Rust does the difficult groundwork of breaking
| this kind of intimate toolchain dependence. As things
| currently stand, why develop a system programming language
| given that no one will ever use it?
| kazinator wrote:
| > _They are simply making sure those interfaces are not
| actively hostile to languages that aren 't C ..._
|
| I.e. not actively small and fast, like an OS should be.
| ekidd wrote:
| Yes, I've seen this comment a number of times now.
|
| ATS is a very interesting language! But there are a wide range
| of options for formal verification of systems programs,
| including the techniques used for the formally verified seL4
| kernel, ATS, recent Ada work, TLA+, etc. Some of these tools
| have been very successful for certain projects.
|
| But the more powerful proof systems involve tradeoffs.
| Frequently, you need to invest significant amounts of developer
| effort in exchange for very rigorously proven code. I've played
| with a number of these systems over the years, and they're
| great. But the costs would be hard to justify for many
| projects.
|
| Rust occupies a slightly different niche. It focuses on
| preventing several classes of memory errors and data races.
| It's certainly harder to learn than Python, but probably easier
| than some common subsets of C++.
|
| So the right tool for the job depends on what you're trying to
| do. If you want to formally verify your kernel's security
| model, or your distributed protocol, Rust would be a poor
| choice. If you want a language that makes it easy to work
| within sight of the metal, and that takes pains to make
| expensive operations visible, Rust can be a reasonable choice.
| [deleted]
| ghostwriter wrote:
| I agree with your comment, and I think we are mentioning the
| same question here, that is essential for the kind of
| discussion that this article raises - whether or not a
| suggested amount of effort to bring Rust into existing and
| well-established C codebase is a great idea and is justified
| on a technical merit basis, compared to the effort of brining
| more sophisticated formal proof assistants that aim at non-
| intrusive inetrfacing with C while being productive for the
| cause of promoting safe and formally-verified programming.
|
| Python's gradual typing with MyPy demonstrated a great
| utility of static type checking that could be brought to
| existing project through many small and non-desruptive
| iterations. From my observation an learning of ATS, I expect
| even greater utility of gradual formal proofs that ATS
| enables for C. There are tons of successful C codebases, they
| are used in production around the world and contain a great
| amount of battle-tested knowledge. They just miss formal
| proofs that now could be gradually written for them.
| simias wrote:
| The Linux kernel is fairly modular, I could definitely imagine
| rewriting subsystems by subsystems with fairly good ergonomics
| and safety. Like rewrite serial_core.c in Rust and then let
| people write serial drivers in Rust. Then the SPI code. Then
| the I2C. Then USB etc...
|
| Easier said than done of course, and definitely a huge effort
| especially for such a huge project as the Linux kernel, but
| there's also a significant amount of resources that could be
| allocated to that task by several large institutions.
|
| I don't know if it's a good idea, or if ATS or something else
| could be better, but I think you're overly pessimistic.
|
| Also "the safety guarantees that Rust provides are neither
| unique nor complete", while technically true, is also
| borderline FUD-y. At least you should expand on what you have
| in mind here, because I doubt that there exists a programing
| language that deals with low level hardware configuration that
| could ever provide "complete guarantees" about anything.
|
| And beyond that, keep in mind that Rust provides a lot more
| than just safety guarantees, its syntax is lot more advanced
| that good old C in particular for anything dealing with types
| (including destructuring, matching, inference etc...). Even if
| you don't get "full" safety right away it can still be a much
| nicer language to code in (and I say that as somebody who knows
| C well and quite like it overall).
| gameswithgo wrote:
| >The safety guarantees that Rust provides are neither unique
| nor complete,
|
| Well no language can ever have "complete" safety guarantees.
| But I would like to know what other languages offer the same
| set of guarantees without GC overhead. This would include:
|
| * the usual memory safety * no nulls * no undefined behavior *
| no data races
|
| Does ATS guarantee those things?
| amelius wrote:
| The problem is that Rust still keeps you thinking about
| memory management. From a security-perspective this makes it
| worse than GC'd languages.
| Athas wrote:
| Yes, ATS can guarantee all those things and more. In
| particular, ATS allows you to write "unsafe" code, which you
| then prove safe with a mechanically verified proof. This is
| in contrast to unsafe Rust, which depends on programmer
| discipline, and has no (intra-language) mechanism for
| verification beyond the basic type system. You could say that
| Rust is all-or-nothing, while ATS allows gradual
| verification, as well as proving safe much more complex code
| than you can in Rust. I'm not going to claim that ATS is
| overall better than Rust - it's not - but it's a far more
| powerful language than Rust for safe systems programming.
| It's just that most people are not willing or able to pay the
| very high price in ergonomics. I recommend watching this
| presentation for a good idea of what ATS programming is like:
| https://www.youtube.com/watch?v=zt0OQb1DBko In short, it's
| about as user-friendly as an overheating nuclear reactor.
| [deleted]
| Ericson2314 wrote:
| To me the benefit of Rust is less safety, than better
| abstractions and modularity. Linux has been growing in
| complexity without much of a breaking point, and the discipline
| and separation of concerns high level languages could help with
| is a huge long-term benefit.
|
| (For example, say we could one day make a bunch of basic Linux
| drivers somewhat kernel agnostic? That would be amazing.)
|
| Now, ATS is I hear a fine language, but I think the ability to
| write good, and especially abstract, ATS is sadly too far
| outside of most kernel dev's skill set. Even among PL
| researchers, there is a tendency to write monolithic Coq---to
| wit, is there a Hackage or crates.io for Coq?---because
| abstraction are hard and the academic paper economy doesn't
| really reward it.
|
| Rust is just easy enough, and with enough crates.io momentum,
| that I hope the relative cost of reusable vs unreusable code
| will be less.
|
| And finally, it takes a lot of political will to introduce a
| new language in any existing project, let alone one as big and
| storied as Linux. Like it or not, but general popularity of the
| language absolutely does help with that.
| darksaints wrote:
| Reading through your various replies to comments, it seems your
| argument boils down to the FFI issue, and I find that a little
| bizarre. Sure, ATS can inline library calls due to the lack of
| an FFI, and maybe it has a cleaner interface to those
| libraries, but now you've introduced a completely different set
| of problems: now you have to tell the compiler how to prove
| your code.
|
| In my experience, learning the fundamental concepts behind
| theorem proving is a lot harder of a problem than writing an
| FFI interface to a library. In fact, that problem is so easily
| solved that there are tools that do 99% of the work for you.
|
| And sure, maybe rust will never be able to inline C code, but
| inlining function calls is such a trivially small performance
| gap that most applications would never know the difference.
|
| I'm not convinced that there are better alternatives for
| contributing to or incrementally rewriting an existing code
| base. FFI is for most purposes a trivial impediment compared to
| the alternatives.
| gpm wrote:
| Your evidence (that there are potentially other alternatives)
| doesn't support your claim (that Rust isn't a great C
| replacement).
|
| There can be multiple great C replacements.
|
| Not being "perfect" in terms of the guarantees you provide
| doesn't mean you aren't a great improvement. Nor would not
| providing any guarantees at all mean that a language is
| necessarily not a great improvement.
|
| That said, I would be very interested in seeing a comparison
| between C, Rust, and ATS in practical code. This is the first
| I'm hearing of ATS and it sounds interesting.
| ghostwriter wrote:
| > Your evidence (that there are potentially other
| alternatives) doesn't support your claim (that Rust isn't a
| great C replacement).
|
| I didn't claim that Rust isn't a great replacement overall, I
| was specifically mentioning that it's not great for
| _existing_ C codebases, by the fact that it requires
| significant work to replace internal interfaces that may be
| important for performance-, backward-compatibility- and
| conventional tooling reasons. It doesn 't interoperate with C
| as well as other alternatives are capable of, whilst not
| being objectively better at safety guarantees either.
| [deleted]
| johncolanduoni wrote:
| How does ATS provide safety guarantees without additional
| semantic information about the C code it's linking to? If
| it does require that, then you're in the same boat as Rust:
| your choices are trivial interoperation with no safety at
| the interfaces or encoding the semantics of the C interface
| in the new language. The only difference I can think of is
| built in ability to inline C from headers (based on your
| comment about kmalloc), is that the case for ATS?
| ghostwriter wrote:
| It's not only inclusion of the headers. ATS compiles to
| C, and it can inline C without FFI. The benefit that you
| receive is the ability to introduce gradual automated
| proofs around unchanged stable interfaces that are known
| to be safe. You start with inlining everything but the
| small core that is proven to be safe. Then you can
| iterate indefinitely at the desired pace to bring new
| layers of formally verified API in places of previously
| unverified C calls without breaking backward
| compatibility and expected runtime profiles (safe
| pointer-based memory access [1] and stack-allocated
| closures [2] as an example)
|
| [1] http://www.ats-lang.org/MYDATA/SPPSV-padl05.pdf
|
| [2] http://ats-
| lang.github.io/DOCUMENT/ATS2TUTORIAL/HTML/c1267.h...
| johncolanduoni wrote:
| This may need fewer code changes (if the code in question
| is correct and the interfaces can possible be implemented
| in a safe way, that is) but it still requires developers
| to write proofs in a dependent type theory. I've only
| looked at ATS briefly, but I'm familiar with a lot of the
| other languages in the space (Idris, Fstar, etc.). Are
| ATS's proof tactics drastically more productive/brief
| than those (F* in particular, which has great aliasing
| reasoning)? If not I think you're radically
| overestimating the approachability of this route compared
| to Rust.
| fluffy87 wrote:
| I've used ATS, Idris and Rust, and I find ATS way more
| cumbersome to use for writing proofs than idris.
|
| I've never used ATS to do what the GP is suggesting of
| incrementally wrapping C with proofs, but I can't imagine
| this being simpler than just rewriting the C in Rust. You
| wouldn't get the same proofs, but you would get memory
| and thread safety, which for many apps would be an
| incremental improvement.
|
| I haven't taught ATS, but I found it harder to learn than
| Idris, and I can't imagine C programmers which have a
| hard time with Rust learning it quicker than they would
| learn Idris or Rust.
| geofft wrote:
| > _it can inline C without FFI_
|
| I'm not sure what you mean by "FFI" (I would apply the
| term "FFI" to the way ATS interoperates with C, too), but
| you can certainly get cross-language LTO between C and
| Rust, since they both compile to LLVM IR.
|
| > _you can iterate indefinitely at the desired pace to
| bring new layers of formally verified API in places of
| previously unverified C calls without breaking backward
| compatibility and expected runtime profiles_
|
| You can do this in Rust, too.
| joshuawarner32 wrote:
| I'm glad that there other efforts for memory-safety going on
| concurrently, and we as a software community should absolutely
| continue to invest in those. I also agree that rust is not
| suited for all circumstances that people try to use it in - but
| I would contend that the set of situations where rust is
| absolutely the right choice is non-empty. I say this as part of
| a ~20 person team, rewriting business-critical software in
| Rust, where most of the engineers I talk with are glad they're
| writing rust.
| JoshTriplett wrote:
| It's certainly not just about safety. It's about the
| combination of safety, expressiveness, productivity, and
| various other factors.
| ghostwriter wrote:
| > but I would contend that the set of situations where rust
| is absolutely the right choice is non-empty
|
| Maybe I should've highlighted it in my initial comment,
| because I definitely agree that this list is non-empty for
| Rust. I was mentioning specifically _existing_ C codebases
| that are heavily used in production around the world (last
| time a similar topic was related to QUEMU project).
| scythe wrote:
| > I won't get tired of repeating the same comment in every
| topic that suggests Rust [...] we should compare it to other
| existing solutions, like ATS [1]
|
| One of the problems with repeating comments is that the
| previous replies are not incorporated into the discussion; for
| example, mine:
|
| >Back when it was on the PL shootout, ATS was consistently in
| the top five for performance. However, it was also consistently
| the worst in program size
|
| I looked into learning ATS some years ago before Rust existed,
| and shied away due to the complexity. It's been around much
| longer than Rust, and I think there's a reason it hasn't caught
| on: it's just too onerous. ATS might be appropriate for
| formally verified systems, but language ergonomics _does_
| matter, especially for a project with as many contributors as
| the Linux kernel.
| stmw wrote:
| ATS certainly looks interestinb, but it's an academic language
| (of which there are many) that most people probably haven't
| heard of... At some point, the momentum of a new programming
| language is just as important -- in practical terms -- as its
| formal attributes and qualities.
| rumanator wrote:
| Your centering your criticism of ATS on popularity. Does an
| investment on PR trumps technical merit?
| dbaupp wrote:
| Popularity and momentum translate into (and are proxies
| for) important things like: library availability, long-term
| maintenance and support, more edge cases are explored (so
| less "research", breaking new ground and bugs when going
| off the beaten track), tooling and even availability of
| teaching material like documentation and tutorials.
| ghostwriter wrote:
| In case of ATS, any C library can be treated as a
| "unsafe-marked" ATS library, so there's no problem with
| library support, the problem is with their formal
| verification. And Rust libraries regularly suffer from
| the same lack of formally-verified and proven-to-be-safe
| APIs.
| dbaupp wrote:
| That helps with the "library availability" sub-point
| specifically, yes. However, there's a more than that when
| evaluating a language, as suggested by my comment above.
|
| In any case, if one is discussing modern and safe
| languages, there's a dramatic difference between using a
| C library and using a library native to the language.
| Even ignoring safety, the ergonomics/developer experience
| is dramatically different and the impedance mismatch can
| be very frustrating (for a "best case" example of this,
| Swift puts a lot of effort into exposing Objective-C
| interfaces as "swiftier" APIs automatically, but this
| benefits significantly from a relatively opinionated set
| of idioms in the source language, something arbitrary C
| libraries do not have).
|
| You're right that being able to import a C library
| directly is a very nice feature.
|
| _> And Rust libraries regularly suffer from the same
| lack of formally-verified and proven-to-be-safe APIs._
|
| Right... but having a larger community means there's a
| much higher chance of a safe or easier-to-use library
| existing for any particular task. In particular, I think
| it means there's almost certainly more libraries in
| total, with more safe libraries and more unsafe libraries
| overall, so simply comparing number (or proportion) of
| low-safety libraries is misleading.
|
| (Formally verified is shifting the goal posts here: the
| bar is just "has a native library".)
| rumanator wrote:
| > In any case, if one is discussing modern and safe
| languages, there's a dramatic difference between using a
| C library and using a library native to the language.
|
| The dramatic nature of that difference is whether that
| library exists or not. Odds are, if exists then it's
| written in C. Thus this point is mute with regards to
| Rust because at best it's relegated to a nice-to-have, in
| the sense you can enjoy the same features that are
| already available in C but with language-specific
| assertions.
| Twirrim wrote:
| Yes, it really does if you're talking about success. Just
| like with Betamax vs VHS, and many, many other examples
| over the years. Technical merit is only one small part of a
| success story, and it's not the most important one.
| chinigo wrote:
| Popularity != PR.
|
| Choosing an obscure language with little community support
| imposes real-world development costs: it's harder to find
| or ramp up new developers, there are fewer eyes identifying
| bugs in the implementation, tooling support can be subpar,
| documentation and blog posts are harder to find, etc.
|
| (btw I know nothing about ATS so I'm not saying this is a
| good description of that language in particular.)
| ghostwriter wrote:
| I agree with your points regarding tooling and dev
| support in general. ATS, in that regard, benefits from
| being a "frontend compiler" to GCC - all the tooling for
| GCC can be used equally well for ATS codebases.
| Developers need to know C well (not an issue with all the
| learning material available), and at least one language
| from ML family (OCaml, Standard ML, F#, Haskell to some
| extent) to quickly pick up the syntax and common
| recursive constructs. There are three great books [1]
| available online, and a collection of common system
| programming examples that demonstrate how data types and
| algorithms can leverage safe language features [2]
|
| [1] http://www.ats-lang.org/Documents.html
|
| [2] http://ats-lang.sourceforge.net/EXAMPLE/EFFECTIVATS/
| outworlder wrote:
| > Does an investment on PR trumps technical merit?
|
| If PR == popularity, yes.
|
| Otherwise we would all be writing our stuff in Haskell,
| Lisp, Elm, F# and similar, and Algol-derived languages
| would be a footnote at this point.
| outworlder wrote:
| > The safety guarantees that Rust provides are neither unique
| nor complete
|
| That sounds like "LEDs shouldn't be used for traffic lights
| because they don't melt snow", while ignoring all the benefits.
| Just add a heater where it is required.
|
| Rust does add benefits. Are these worth the extra effort? It
| seems that some people think it's worth it.
| geofft wrote:
| As one of the presenters of this session, I'd like to reply by
| saying, this is cool! I'm hopeful it shows some good results!
| I'm not wedded to Rust in particular - it just seems to have
| the right balance of solving the particular problems the kernel
| wants to solve (there are plenty of other great languages like
| Go and Java and Python and Haskell and C++ that aren't
| suitable), easy to learn for people who have been working with
| C and have picked up an intuition for things like lifetimes and
| ownership that matter in C but aren't concrete, and popular
| (which is important because it means there are good learning
| resources, communities building things, etc.).
|
| It looks like the "ATS Linux" work is happening on a Git fork
| at http://git.bejocama.org/ats-linux-5.7.2 . However, I don't
| see any commits there - it looks like it's just the v5.7.2 tag
| from upstream. Do you know if there's any sample code that's
| been written demonstrating kernel code in ATS?
|
| I think that, empirically, we have built something that works.
| You can check it out (https://github.com/fishinabarrel/linux-
| kernel-module-rust - see tests/* in particular). I'd be
| delighted to see folks who are excited about other languages
| build something in their language, too.
|
| I'd like to correct two specific things about your comment:
|
| > _if we discuss the amount of effort necessary for bringing
| new interfaces like the one this article mentions ( "one can
| define a kmalloc_for_rust() symbol containing an un-inlined
| version"), we should compare it to other existing solutions,
| like ATS_
|
| Here is the amount of effort necessary to work around that:
|
| https://github.com/fishinabarrel/linux-kernel-module-rust/bl...
|
| I took a semester-long MIT class on a proof assistant (Coq) and
| I thought it was great but I'm still not fully comfortable with
| it. I realize that Coq and ATS are different languages, but I
| would humbly submit that if you compare the amount of effort
| spent calling krealloc instead of kmalloc vs. learning ATS, the
| former is probably lower.
|
| > _without giving up on existing C interfaces and the
| toolchain_
|
| Looking at Chapter 8 "Interaction with C" from the ATS book
| http://ats-lang.github.io/DOCUMENT/INT2PROGINATS/HTML/c2016....
| , it seems like ATS's interoperability with C is pretty
| comparable to Rust's https://doc.rust-
| lang.org/book/ch19-01-unsafe-rust.html#usin... - I don't think
| either of them require giving up on existing C interfaces or
| the C toolchain. One of the specific reasons to use Rust in
| this context instead of Go (which is a great language too) is
| that Rust is designed to fit in very closely with the C
| toolchain. The first paragraph of that chapter, which discusses
| how ATS's data structures can be zero-overhead bridged with C
| and you can view ATS as a better-typed frontend for C, seems to
| equally well apply to Rust.
|
| (In fact, as mentioned in the article, we'd love to
| _automatically_ generate bindings from existing C interfaces.
| It turns out that most interfaces lack documentation -
| certainly machine-parseable docs, but often human-readable docs
| too - that indicates their ownership properties and locking
| constraints and so forth, but definitely the best approach is
| to get this sort of information into C annotations instead of
| having manual bindings in Rust. Then, if it turns out that
| another language like ATS or Sing# is a better fit than Rust,
| the information is right there.)
| Waterluvian wrote:
| It honestly doesn't matter a whole lot what makes people
| excited about contributing to Linux. I think it's worth
| embracing whatever does as long as it's at worst harmless.
|
| Edit: I think what I'm trying to say is, let's embrace what
| excites people.
| cbmuser wrote:
| If we get the Rust backend for gcc finalized and merged first, it
| would be much easier to get Rust code into the kernel:
|
| > https://github.com/philberty/gccrs/
| gok wrote:
| That's a front end?
| Spivak wrote:
| Compile C to Rust! Easiest language migration ever.
___________________________________________________________________
(page generated 2020-08-31 23:00 UTC)