[HN Gopher] DockYard R&D: FireFly Optimizes Your Elixir Compilation
___________________________________________________________________
DockYard R&D: FireFly Optimizes Your Elixir Compilation
Author : freedomben
Score : 78 points
Date : 2022-09-03 16:25 UTC (6 hours ago)
(HTM) web link (dockyard.com)
(TXT) w3m dump (dockyard.com)
| pdimitar wrote:
| I would appreciate if this post included just a few shell script
| snippets showing how to replace current Erlang compilation
| toolchain so Erlang/Elixir/other-BEAM-langs can try it, i.e. a
| drop-in replacement for your day-to-day dev workflow.
|
| I don't mind compiling `firefly` from source (as the GitHub page
| instructs). That's all fine and an accepted reality when an
| experimental tool is starting to take off.
|
| But not providing much of a context + actionable list of items on
| how to try it in your day-to-day work is to me a PR piece doomed
| to fall on deaf ears.
|
| This is not an investor pitch to business people. You're
| advertising to programmers. Please craft your articles
| accordingly.
|
| Don't get me wrong, I am glad this exists and will follow it. I
| just wish we had something to copy-paste in our `~/.bashrc` file
| and and be able to try the tool today.
| tr1ll10nb1ll wrote:
| Dockyard seems to be making strides with their projects which in
| turn makes working with Elixir lucrative (assuming the projects
| do take off).
|
| Yesterday, they posted LiveView Native and now-- this.
| JediLuke wrote:
| yewenjie wrote:
| I think this project used to be called Lumen until pretty
| recently - https://github.com/GetFirefly/firefly
| jononomo wrote:
| It makes no sense to me to use Elixir without running on the
| BEAM. Absolutely zero. You lose OTP, for one thing, which means
| that the entire programming paradigm goes out the window.
| bitwalker wrote:
| You don't lose OTP, because OTP is a library, written almost
| entirely in Erlang (not counting the set of NIFs/BIFs which
| provide intrinsic functionality), which we absolutely aim to
| compile with Firefly just like any other Erlang (or other BEAM
| language) sources.
|
| The BEAM also provides a runtime, but that runtime can be
| implemented using other strategies. It essentially provides a
| M:N green threading abstraction (processes), with a specific
| set of semantics around how those communicate (messages) and
| how failure is handled (links/monitors/etc). Firefly provides a
| runtime that aims to be equivalent to that of the BEAM from the
| perspective of the developer, the only difference is in how
| that is done behind the scenes, what is produced by the
| compiler, and what restrictions we impose that the BEAM doesn't
| (namely no hot code loading, at least for the forseeable
| future).
|
| I'm not sure where you got the idea that Firefly throws away
| OTP, or tries to implement Erlang with different semantics,
| because that is explicitly _not_ the goal.
| hosh wrote:
| The people writing Firefly are trying to support OTP on WASM.
| Because the WASM runtime have different characteristics and
| guarantees from BEAM, and they don't want to run BEAM inside
| WASM, they created an alternate runtime. There was at least one
| instance where Dockyard proposed and implemented changes to the
| WASM spec itself in order to support Elixir/Erlang/OTP in WASM.
| sergiomattei wrote:
| I don't think Firefly gets rid of the OTP.
| jononomo wrote:
| OTP is a programming library/paradigm that doesn't work
| without the ability to create extremely lightweight threads,
| which is what the BEAM provides.
| peoplefromibiza wrote:
| Apparently it has feature parity with the BEAM, but lacks
| support for NIFs
|
| https://github.com/GetFirefly/firefly#runtime
| bitwalker wrote:
| It doesn't currently, but you are correct that the goal
| is to maintain feature parity with the BEAM (with
| explicit caveats to that, namely hot code loading). There
| actually is support for NIFs, just not via the erl_nif
| interface that NIFs use today, support for that will
| arrive eventually.
| freedomben wrote:
| I don't understand, isn't the BEAM just a software runtime
| that provides a lightweight thread implementation that OTP
| can use? Why couldn't they implement an alternative that
| runs on WASI?
|
| I agree that Elixir without OTP becomes much less useful,
| but there could be some changes to the language to enable
| in-process state changes so that you wouldn't be limited to
| a single process and the immutability restrictions that
| would make that very difficult to do anything useful with.
| I'm sure they thought of this problem and have a solution
| in some form.
| bitwalker wrote:
| Correct, the BEAM is just one implementation of the
| runtime, there can (and have) been others. Firefly aims
| to be as close to the BEAM semantics as possible, but
| there will naturally be some differences as we're taking
| a different approach with compilation, with some benefits
| as a result, but naturally there are also tradeoffs.
| pessimizer wrote:
| BEAM isn't the only runtime possible with green threads.
| OTP is just the libraries.
|
| Lots of people have written alternatives to BEAM. The only
| problem they run into is that BEAM is very good, and would
| be tough to beat. I was an admirer of Erlang on Xen:
| https://github.com/cloudozer/ling
| derefr wrote:
| As someone who's the exact target audience of this post
| (deploying Elixir apps in production), I have no idea what this
| PR piece is trying to say.
|
| > Unlike BEAM it compiles applications ahead of time, allowing
| Lumen to perform optimizations BEAM can't.
|
| What does this mean? Erlang/Elixir _are_ AOT-compiled languages.
| They 're compiled into bytecode. (Also, "BEAM" is not a compiler,
| but a bytecode VM which executes the bytecode emitted _by_ the
| compiler.)
|
| Do they mean that they further compile the BEAM bytecode to
| native host-ISA object code? Or did they write alternative Erlang
| and Elixir compilers, to compile these HLLs directly to host-ISA
| object code, skipping both the language runtime and BEAM bytecode
| entirely? Do they mean "compiles applications" literally, in the
| sense of Whole-Program Optimization?
|
| In short -- is this something like GraalVM's Native Images; or
| something like an AOT version of BEAM's HiPE extension; or
| something else?
|
| > FireFly is able to compile Elixir applications without having
| to run through the Erlang Virtual Machine.
|
| So is the key benefit here that the _compilation_ is faster
| because it 's not running on the BEAM (good for e.g. CI); or is
| the key benefit that the _resulting executable_ is faster because
| it 's not running on the BEAM?
|
| Also, there are some questions I have that this post didn't even
| try to answer. E.g.:
|
| It is my understanding that the the bytecode-ness of the Erlang
| VM is crucial to the lightweight bounded-runtime cooperative-
| scheduling mechanism that allows Erlang/Elixir code to be high-
| concurrency + soft-realtime. (Effectively, every ISA op
| implicitly decrements a per-actor reduction-counter as part of
| its implementation; and the yield-point checks are _also_ inside
| the impls for certain BEAM ISA ops, rather than being their own
| explicit BEAM ISA instructions.) How does a non-bytecode version
| of Erlang /Elixir abstract-machine semantics, achieve these same
| guarantees? Is there an explicit reduction-counter being carried
| around in the emitted native code?
|
| And _also_ , there is no mention of disadvantages/constraints of
| using this system. It's pretty clear that you wouldn't be able to
| do hot reloading or dynamic trace-point insertion without the
| BEAM there to intermediate it. That's fine for some use-cases,
| but they should explicitly mention the trade-offs and target
| audience.
| freedomben wrote:
| While I'd be fascinated to hear the answers to all your
| questions, I would guess maybe 2% of people would even
| understand them. I wouldn't expect a post like this to get
| anywhere near that technical.
| e3bc54b2 wrote:
| Indeed all the DockYard pieces announced are nice, but the
| posts are puff pieces at best.
|
| From what I understood, Elixir/OTP are there, but instead of
| compiling to BEAM bytecode, they compile for WASI (targeting
| whatever can run WebAssembly). It is not HiPE, nor JIT, there
| will not be any bytecode. Only AOT compiled native code, except
| targeting WASI.
|
| Generally BEAM is understood to be slower of the big runtimes
| (compared against Java, CLR, Go and often V8/Javascript).
| FireFly claims to be faster, and smaller in compiled form than
| current interation compiled against BEAM.
|
| How that plays with actor model and preemptively switched
| lightweight processes is a mystery to me too.
|
| I'd really appreciate something with more technical sauce but
| human explanation. Currently all we have is PR puffs like this
| and source code (at least for the LiveView native, haven't
| tried looking for this).
|
| All that said, this does indicate there is good activity and
| desires to get Elixir working on stuff other than its current
| strong suit. This usually means the language and ecosystem is
| growing. It could be Baader-Meinhof phenomenon, but I started
| learning Elixir couple of weeks ago, starting with LiveView
| tomorrow, and there is nice stream of posts coming for it.
| Exciting times :)
| freedomben wrote:
| Agreed, these posts from Dockyard have been really low
| quality. Dockyard makes some awesome stuff generally, so I'm
| surprised at how shallow and marketing-speak these have been.
|
| That said I'm really excited for this and the other projects.
| Looking forward to some more technical dives.
| hosh wrote:
| They make up for it in their ElixirConf keynotes and
| presentations.
| hosh wrote:
| If you are looking for more substantial technical
| information, they are found in the ElixirConf '19 keynote
| speech and demo when Lumen was announced.
|
| https://youtu.be/uMgTIlgYB-U
___________________________________________________________________
(page generated 2022-09-03 23:00 UTC)