[HN Gopher] Thoughts on what a next Rust compiler would do
___________________________________________________________________
Thoughts on what a next Rust compiler would do
Author : arunc
Score : 153 points
Date : 2023-01-26 20:06 UTC (2 hours ago)
(HTM) web link (matklad.github.io)
(TXT) w3m dump (matklad.github.io)
| thagsimmons wrote:
| This sounds great, but I don't think Rust has the complement of
| highly skilled core developers needed to tackle something this
| ambitious. I'm a close observer of the Rust project, and my
| impression is that Rust's core development team has been hollowed
| out over the past few years - there's been a string of quiet
| departures, which sadly included some of the most powerful
| contributors. The Rust project is quite secretive and opaque
| about its internal politics, so I don't know if there's any
| unified reason for the attrition, and I don't think it's useful
| to try to read the tea leaves. At this point, I'm just keen to be
| reassured that Rust has the momentum to complete things like
| async and shore up holes in the type system on a reasonable
| timescale.
| pcwalton wrote:
| People come and go from every project. I haven't noticed any
| particular drop in momentum for the Rust compiler. (Also, the
| Rust compiler dev community is hardly "secretive and opaque"--
| just look at the amount of drama that regularly spills out into
| the open. I can't think of any compiler that's _more_ openly
| developed than Rust, in fact.)
|
| A brand-new Rust compiler may well never happen, but if it
| doesn't happen it's because the business value for a complete
| rewrite isn't there, not because of some mass departure of
| compiler developers.
| spoiler wrote:
| > I'm a close observer of the Rust project, and my impression
| is that Rust's core development team has been hollowed out over
| the past few years - there's been a string of quiet departures,
| which sadly included some of the most powerful contributors.
|
| Can you give more specifics on this?
| thagsimmons wrote:
| I don't want to list specific contributors here. We often
| have no information on why people left the project - there
| might be all sorts of personal factors. I will say that my
| broader sentiment - that Rust's momentum has slowed, that
| it's not delivering on its commitments in a timely way, that
| there are concerns about it's ability to deliver in future -
| has been expressed publicly by high profile past core
| contributors:
|
| https://github.com/rust-
| lang/rust/pull/96709#issuecomment-11...
|
| I don't think Rust should tackle any ambitious project to
| rewrite the compiler while these basic concerns remain.
| TylerE wrote:
| I say this with great respect, but the thing keeping me from Rust
| isn't the compiler, it's the language.
| alphanullmeric wrote:
| It would actually be such an amazing language if it wasn't so
| verbose. There are so many places where more concise syntax
| wouldn't even hurt safety and they forgo it anyways. But here
| we are, where macros are required for a hello world.
| spoiler wrote:
| You don't technically need the macro to print hello world.
| However, the macro is useful due to format strings, which are
| also type-checked (eg to checks if the args implement
| `Display` or `Debug`, etc).
|
| The good thing about the default hello world example is that
| it introduces you to both language features without being
| overwhelming (IMO).
|
| You can read more about format strings here:
| https://doc.rust-lang.org/std/fmt/
| JoshTriplett wrote:
| > There are so many places where more concise syntax wouldn't
| even hurt safety and they forgo it anyways.
|
| Can you give some examples of this? If there are ways we
| could simplify the language without hurting existing use
| cases, we should consider doing so.
| WinstonSmith84 wrote:
| Once you're done fighting with the compiler (Rust Analyzer),
| it's actually very enjoyable and one can be _relatively_
| productive. The language allows so much flexibility that it's
| very easy to produce well organized code...
|
| But the compiler is really slow. Even an incremental build on a
| mid size project is never below 10s, whereas on a similar size
| project, Golang will take me less than 1 second to build
| incrementally. Also, I find cross-compilation (from/to major
| platforms) really challenging on Rust, as opposed to Golang
| where it's for my use cases super straightforward
| stefncb wrote:
| My personal issue with Rust isn't the borrow checker. I
| actually find it quite intuitive. What I don't like is the
| extreme complexity of everything. There's just so much...
| stuff.
|
| What I want is a Rust, but with almost everything stripped
| away. Complexity similar to Go.
| the_mitsuhiko wrote:
| I would be quite impressed if there was a way to have Rust
| with "less stuff". It's not like there is a lot stuff that
| can be removed without making the language quite useless.
| acedTrex wrote:
| so you want Go then?
| jug wrote:
| I just use Go (or Nim) for native. It's not comparable in
| goals and what Rust tries to do at all. But Go compiles
| super fast, is very fast, solves the glaring C safety
| issues, very keen on helping you with threads, and has a
| very fast garbage collector. If I run into performance
| problems, it's not because of Go. But yeah Rust has its
| place in certain scenarios. I still hesitate to call it
| general purpose though -- I think it's best suited for
| stuff that underpins applications/usermode. And I don't
| think it can be made much less obnoxious.
| TylerE wrote:
| nim comes the closest of anything I've found, the
| infrastructure just isn't quite there.
|
| ocaml has potential but I strongly dislike the numeric
| operators, and I'm not a huge fan of the syntax in general.
|
| OCaml with python like syntax would be damn near perfect. I
| could even overlook the operator thing.
| oconnor663 wrote:
| It's worth pointing out that if you say "Go except with
| Rust's guarantees for thread safety", a _lot_ of the
| complexity comes back. You need to pull in lifetimes and
| move semantics and the no-mutable-aliasing rule. You need
| the Send and Sync traits. You probably want Deref-based
| smart pointers too. (And you'd need to make extensive use
| of the new generics features that are already in Go.)
| tcmart14 wrote:
| The slow compile times I look at as a trade off. Rust does a
| lot more during it's compile phase than most other compilers
| and there is a cost. Do I do all this static analysis and eat
| time here to hopefully reduce runtime bugs and subsequently
| the time to debug runtime bugs? It takes me a whole lot
| longer to debug something with a debugger than if the
| compiler can catch it.
|
| I do 100% agree on your commends on cross compiling though. I
| also don't like how the target for linux has "unknown" in it.
| agumonkey wrote:
| I don't have much experience but after writing some tiny rust
| scripts and similar sized things in kotlin (trying kotlinc
| v1.7 that was supposed to be a lot faster than previous
| versions), rust managed to stay ahead.
|
| it's true golang is smoother, but the language is a lot
| messier (idioms and typechecking).
|
| Can't conclude anything yet, but it makes me think back of
| rust more often than others.
| LAC-Tech wrote:
| It's the standard library & tooling for me. To be perfectly
| honest for must stuff I use rust for I'd prefer to have garbage
| collection. But the standard library and tooling I just too far
| ahead of the competition (diplomatically avoiding naming
| languages here)
| loeg wrote:
| That's fine. It doesn't have to be everything to everyone. We
| probably don't need this kind of comment on every single
| article about Rust, though. (Or similar generic "I don't like
| this language" comments on any article related to any
| programming language.)
| TylerE wrote:
| Then we equally don't need pro rust comments on literally
| every other thread
|
| Since a major thrust of the linked article is barriers to
| rust adoption it seemed relevant.
| jackmott wrote:
| [dead]
| loeg wrote:
| > Then we equally don't need pro rust comments on literally
| every other thread
|
| Correct.
| echelon wrote:
| What about the language do you dislike? It's a strongly typed
| Ruby with generics.
|
| Edit: trait-based OO is a breath of fresh air compared to tree-
| style class inheritance. Super flexible without having to
| overthink.
|
| Immutable by default is reassuring, Option/Result are fantastic
| null/exception replacements.
|
| Enums and match blocks are powerful and gracefully help ensure
| handling of all cases, have nice syntax, and work well for a
| systems language.
| nonethewiser wrote:
| Not OP but for me it's the syntax. I'm not even saying it's
| bad or could even be improved... I just don't find it
| intuitive. I've only sat down and walked through some of the
| learn rust book and tried to hack out a few small things.
|
| I'm not sure what it is - there just always seem to be random
| symbols that dont seem to follow conventions of other
| languages. Guess I am use to C-style (including C++, C#, JS)
| syntax and python. Its possible that there are just new
| concepts that aren't really represented in the languages I
| use too.
|
| Again, I don't necessarily think its bad. Just what I don't
| find intuitive.
| nicoburns wrote:
| Interestingly very little of Rust's syntax is novel (the
| exception being lifetimes which of course don't exist in
| other languages). The closest language syntacticly is
| probably TypeScript (you'd be surprised how many programs
| are actually valid in both Rust and Typescript!). And where
| syntax doesn't match TypeScript it usually matches
| something else common. For example the `self` parameter in
| methods comes from Pythong, and the closure syntax comes
| from Ruby
| matklad wrote:
| Lifetime _syntax_ is lifted verbatim from OCaml.
| spoiler wrote:
| Not quite used for the same thing, but Haskell also
| allows using 'x (for symbol names)
| tcmart14 wrote:
| For me, its the lifetime syntax I don't like the most. It
| doesn't feel "ergonomic" to type something like <`a>. Its
| mostly the ` I wish were different.
| JoshTriplett wrote:
| (Aside: it's 'a , not `a.)
|
| One of the hopes is that it's _usually_ not necessary to
| write those lifetimes explicitly in most programs, unless
| you 're doing something unusual.
|
| If you could go back and change it, what would you have
| used for the lifetime syntax?
| mlindner wrote:
| New syntax is pretty easy to pick up. It's just window
| dressing.
|
| What you're actually seeing is a trap a lot of programmers
| fall into where if something is unfamiliar then it's
| treated as if it's incorrect.
| nonethewiser wrote:
| > What you're actually seeing is a trap a lot of
| programmers fall into where if something is unfamiliar
| then it's treated as if it's incorrect.
|
| I said that I don't think it's bad or wrong.
| afr0ck wrote:
| In real world software, you need lots of freedom in order to
| design high performance scalable systems. This includes using
| non-trivial data structures, and being able to access and
| organise memory in subtle ways. The Linux kernel is an
| example of this. Look at the reclaim subsystem in the kernel,
| for instance, and how it interacts in subtle and very complex
| ways with various other subsystems, the page cache, the
| generic block layer, the memory control groups, the VFS and
| the filesystems. All of them are multi-threaded and run on
| complex high performance CPUs performing memory re-ordering,
| speculative execution, branch perdiction and parallel
| execution of both processes and instructions. You really need
| extremely smart smart data structures and organisation and
| very smart algorithms to manage the concurrency. C is a
| language where you have all that freedom to do whatever you
| want leading to high performance and low latency. In Rust,
| you can't even safely implement a doubly linked list. Now, if
| you have to use unsafe every time you need high performance
| or scalable data structures, then Rust is as unsafe as C and
| it's just making your life harder for no reason, except hype.
| alkonaut wrote:
| > You really need extremely smart smart data structures and
| organisation and very smart algorithms to manage the
| concurrency. C is a language where you have all that
| freedom to do whatever you want leading to high performance
| and low latency.
|
| But it leaves that work to the developer, thousands of
| hours of testing and reviewing to ensure no little corner
| case is missed.
|
| I think the argument that unsafe is used making the
| language have "holes" is somewhat misdirected. When I see a
| rust implementation of a doubly linked list with unsafe I
| know exactly where I'm on my own (a few lines) and where
| the compiler does the job for me (the rest of it). It's not
| as if that means safety or flexibility is out the window.
| It means "do the manual safety review on these two lines".
| jolux wrote:
| > Now, if you have to use unsafe every time you need high
| performance or scalable data structures, then Rust is as
| unsafe as C
|
| That's not true, because you can wrap unsafe code in safe
| interfaces and use it from safe code.
| vore wrote:
| As someone who has done kernel development work (though not
| Linux), not every part of a kernel is as filled with
| dragons as you make it out to be. Why would my e.g.
| filesystem driver need to punch a lot of deep unsafe holes
| down to the internals of memory management? Why can't I
| make use of a safe abstraction implemented using unsafe
| code to free myself from caring about those details?
|
| With C, you can only do these kinds of safety guarantees
| with a lot of discipline. With Rust, you can offload much
| of your discipline to the compiler.
| josephg wrote:
| > Rust is as unsafe as C and it's just making your life
| harder for no reason, except hype.
|
| I hear what you're saying, but that hasn't been my
| experience with rust. I love C, but I've been writing rust
| for the last couple of years. As much as I love C, rust has
| stolen my heart.
|
| You're right that unsafe rust is a bit less ergonomic than
| just writing C. Sometimes I miss void pointers. I
| definitely miss how fast C compiles. But most rust isn't
| unsafe rust. Even deep in my custom in-memory btree
| implementation I think more than half of my methods are
| safe. And safe rust is a fabulous language when you can use
| it. There's all these bugs you just can't write.
|
| Early on with rust I had this magical experience. My
| program segfaulted in one of my tests because of memory
| corruption. It was "spooky action at a distance" where the
| segfault happened well after the buggy code was executed.
| Bugs like this are awful to track down in C because the bug
| could be literally anywhere. But when I looked at the
| trace, I realised there was only one unsafe function which
| could be causing the problem. (Since I could rule out all
| my safe code). Sure enough, 10 minutes later I had a fix.
|
| I have a skip list for large strings that I ported from C
| to rust. I still have no idea why, but the rust code ran
| about 20% faster out of the box than the original optimized
| C code. And the code is significantly smaller and easier to
| work with. I've added a few more optimizations since then -
| it's about 10x faster than the C code now from tweaks I
| made that I never got around to adding in C for fear of
| breaking something.
|
| And then there's the things rust does well that aren't in C
| at all. Cargo is incredible. Monomorphization is the right
| tool for a lot of problems, like custom collections.
| Parametric enums & match statements are so much better to
| work with than C enums and unions. Rust's std is fantastic.
| I use Option, Result, Vec, BtreeMap, PriorityQueue, the OS-
| agnostic filesystem API, and so on daily.
|
| So yeah, I hear you about C being lovely. But I think rust
| is even better. It's awful to learn but I think it's
| totally worth it.
| zozbot234 wrote:
| The problem with "subtle and very complex" interactions is
| that they don't really scale to larger systems. They're
| inherently anti-modular, since they depend on the state of
| the program as a whole. This is exactly what's avoided in
| Safe Rust.
| lalaithion wrote:
| The key is that Rust lets you define safe interfaces, so
| you can separate your data races and union juggling, which
| need to be unsafe, from your business logic, which
| basically never needs to be unsafe.
| nyanpasu64 wrote:
| "Object graph" architectures are common in C++ and
| sometimes necessary in Rust for _business logic_ , when
| building GUI applications or emulators. But Rust doesn't
| allow mutating through a &/Rc, and throws a compile error
| if you create multiple &mut, and the workarounds are
| unreasonably boilerplate-heavy and RefCell carries
| runtime overhead, whereas C++ doesn't get in the way of
| making your code work.
|
| I've put together a playground at https://play.rust-
| lang.org/?version=stable&mode=debug&editio.... It's not
| just dereferencing invalid */& that's illegal in Rust,
| but constructing and dereferencing valid &/&mut in ways
| that don't respect tree-shaped mutability. Rust's pointer
| aliasing rules invalidate otherwise-correct code, placing
| roadblocks in the way of writing correct code. There's so
| much creation of &mut (which invalidates aliasing
| pointers for the duration of the &mut, and invalidates
| sibling &mut and all pointers constructed from them),
| that's so implicit I don't know what's legal and what's
| not by auditing code. (Box<T> used to also invalidate
| aliasing pointers, but this may be changed. The current
| plan for enabling self-reference is Pin<&mut T>, but the
| exact semantics for _how and when_ putting a &mut T in a
| wrapper struct makes it not invalidate self-reference and
| incoming pointers, is still not specified.)
|
| (I've elaborated further at
| https://news.ycombinator.com/item?id=33658253.)
| [deleted]
| packetlost wrote:
| Huh? Rust shares very little with Ruby??? Are you thinking
| Crystal?
| woodruffw wrote:
| Crystal is directly inspired by Ruby, so I don't think is
| too far of a bridge!
|
| I programmed Ruby for years before learning Rust, and
| aspects of Rust's library and language inspiration from
| Ruby are obvious to me: the closure syntax closely
| resembles Ruby's blocks, for example, and much of the core
| iterator APIs match their Ruby counterparts. It's a _very_
| different language overall, of course!
|
| (Then of course there's the part where many early Rust
| contributors came from the Ruby community.)
| TylerE wrote:
| The borrow checker. The syntax. It isn't a good fit for my
| needs. It'd be like using a forklift to move a few books. I'm
| not generally a fan of strong explicit typing/high ceremony
| in general.
| MuffinFlavored wrote:
| Curious, are you a fan of Typescript?
| TylerE wrote:
| Never used it. I don't even have the need to touch
| vanilla javascript that often.
| smt88 wrote:
| > _strong explicit typing /high ceremony_
|
| If you think strong typing is "ceremony" then you probably
| spend most of your time writing (and documenting) code and
| not reading, refactoring, or collaborating on it.
|
| The time people spend writing unnecessary, buggy unit tests
| (that static analysis can do in better languages) is far
| greater than the time to just use the type system.
|
| Most languages don't even force you to be explicit anymore.
| They infer the types, so you don't even write extra code.
| You just get better errors and speed.
| sedatk wrote:
| It's probably not the language syntax per se, but satisfying
| the compiler (borrow checker) can be tedious for beginners.
| vore wrote:
| I felt this way about Rust for a really long time, but after
| sitting down and doing a few projects with Rust I've really
| come to appreciate how much the language stops me from being a
| dumbass.
| baq wrote:
| How much faster can the compiler be? This is the top issue with
| rust IMHO - compile times are real bad.
| thagsimmons wrote:
| This is absolutely a pain point for us, working on a very large
| Rust project. In particular, incremental compile times are
| absolutely critical for developer comfort, and by far the most
| common complaint working on our codebase is that developer
| tooling, IDEs and running unit tests is slow. We've done
| everything that can reasonably be done - splitting the project
| into crates, using mold as a linker (the single biggest
| improvement), etc. Without a really big improvement to
| incremental compile, we will have a continuing drag on our
| development momentum.
| WinstonSmith84 wrote:
| Any reasons why it could not be as fast as the Go compiler?
| Maybe GC and the borrow checker could slow down a bit, but
| apart from that?
| loeg wrote:
| The borrow checker is not an insignificant cost. Maybe you
| could imagine an incremental compilation mode that just
| didn't evaluate the borrow checker for faster development.
|
| As other commenters have noted, the Go backend is extremely
| simplistic ("barely an optimizing compiler"). Rustc is also a
| lot faster in -O0 mode than any optimizing mode. But -O0 is
| somewhat dumber than Go's compiler. You could imagine an -O1
| (or -O0.5) with compilation time vs performance tradeoff
| similar to Go, but it isn't there today. There's also an
| attempt to adopt a more simplistic backend (cranelift) into
| Rustc to improve codegen performance, but so far it hasn't
| provided much speedup.
| alkonaut wrote:
| It could, if it stopped doing a lot of things it does which
| the Go compiler doesn't.
|
| Or of course if the Go compiler started doing them so it
| became a lot slower.
|
| One example is "full"monomorphization of generics. But there
| are lots of such examples.
|
| Usually though I find the Rust compilation speed a non issue,
| so long as "cargo check" and the IDE analysis is great, I
| just rarely compile to begin with.
| bonzini wrote:
| The Go compiler is barely an optimizing compiler. Rust needs
| massive amounts of inlining to avoid the performance cost of
| abstractions, and after inlining the code needs to be cleaned
| up. In this respect it isn't unlike C++.
| sk0g wrote:
| Does anyone particularly care if development builds inline
| much? If you can get Go-like compile speeds for development
| iteration, and the current ones for release, it would be
| the best of both worlds for me.
| loeg wrote:
| In fact, that's exactly why cargo's default for dev
| builds is -O0: https://doc.rust-
| lang.org/book/ch14-01-release-profiles.html...
| cxr wrote:
| The reference compiler needs a lot of time to make the
| resulting binary fast, okay fine. But where's the rust-
| lang.org group's fork of the golang.org group's `go build`
| tool modified to accept source files written in the Rust
| language as input and that (quickly) outputs a binary with
| a performance profile no faster or slower an equivalent
| program written in Golang? ~15 years of compiler research
| is too long and comparisons between the two languages
| (however fair or un-) are too numerous for there to be any
| excuse for this not to exist, and yet
| [deleted]
| the_mitsuhiko wrote:
| If you consider how much LLVM instructions are emitted and how
| much time Rust spends linking, the potential for speedups seems
| very significant.
| choeger wrote:
| I agree that the C-model is keeping Rust back. But it's
| ubiquitous and extremely successful. Dynamic linking also
| practically gives us an ABI (a very simplistic one) for free,
| which is great for FFI and debugging.
|
| So if I could choose, I'd invest money into a new ABI/linker that
| reconciles polymorphic languages with separate compilation
| without demanding a uniform object model. Not only Rust would
| benefit from such a linker.
|
| I have no complete solution for how such a thing might work, but
| I think it's possible because the actual specializations that
| stem from monomorphization are very few. It might simply be
| possible to create one function for every possible
| monomorphization ahead of time. Even if this turns out to be
| futile, the actual specialization only needs to change very few
| things in the code, so monomorphization could effectively be
| handled by a dynamic linker.
| tomsmeding wrote:
| > Even if this turns out to be futile, the actual
| specialization only needs to change very few things in the
| code, so monomorphization could effectively be handled by a
| dynamic linker.
|
| This is true in only the simplest case. Already if you have a
| polymorphic function that sums the elements of an array, you'll
| be doing function calls to string concatenation if those
| elements are strings but would really like the vectoriser to
| produce nice SIMD code when specialised to floats.
|
| Specialisation is something that should happen _before_ the
| optimiser.
| choeger wrote:
| > Specialisation is something that should happen before the
| optimiser.
|
| That's true, but your example shows a very important point:
| These cases are limited and (in current languages) known by
| the compiler developers. I don't know a language that would
| allow a user-defined datatype to come with such specific
| optimizations. So for today's languages one could probably
| enumerate the special optimizations before specialization and
| then later only dispatch to the optimized code.
|
| But even if we say that optimization must happen after
| specialization, I still think that these optimizations are
| general enough to be put into a dynamic linker. That's what
| all JIT engines do, after all.
| Animats wrote:
| There are already at least two Rust compilers, now that there's a
| GCC version. That's good; it means the language becomes stronger
| than the implementation.
|
| Right now, the weak points are mostly library side. Too many 0.x
| version crates where the API is still in flux.
| echelon wrote:
| All of this sounds fantastic.
|
| I can't donate my time, but I'd be happy to donate money to
| anyone taking on this effort.
|
| On that note, I'm looking for more Rust crates and maintainers to
| donate to. I just started paying Bevy, and I'm looking for more
| areas of the Rust ecosystem that need it.
|
| This language brings me immense business and personal value, and
| I'm happy to give back.
| elabajaba wrote:
| Bjorn3 for all their work on cranelift and working towards
| getting it to be a better debug compiler for rust.[1]
|
| cwfitzgerald for wgpu. He took over as the head maintainer
| after kvark left Mozilla to go to Tesla.
|
| winit maintainers, they're a core ecosystem crate but don't get
| that much support because windowing isn't flashy.
|
| [1] https://github.com/bjorn3/rustc_codegen_cranelift
| karmakurtisaani wrote:
| Web3 or something useful?
| [deleted]
___________________________________________________________________
(page generated 2023-01-26 23:00 UTC)