[HN Gopher] Scaling Erlang Developer Experience at WhatsApp [pdf]
___________________________________________________________________
Scaling Erlang Developer Experience at WhatsApp [pdf]
Author : anuragsoni
Score : 223 points
Date : 2020-09-11 14:20 UTC (8 hours ago)
(HTM) web link (codesync.global)
(TXT) w3m dump (codesync.global)
| fbn79 wrote:
| Working in a small Company always let me astonished learning
| things about unicorn apps. But really an application like
| Whatsapp need 1000+ developers!? I can think about teams about
| android, ios and web app. X2 if you think about teams for
| business oriented services. But 1000+ developers for what?
| jamestimmins wrote:
| Just some of the needs at that scale: Fraud
| Security Integrations Infrastructure Data
| Accessibility Growth I18n API
|
| In addition, there won't be individual teams for android, ios,
| etc. Those domains will be broken across many task-specific
| teams.
| MiroF wrote:
| > But 1000+ developers for what?
|
| Managerial prestige
| gt565k wrote:
| Whatsapp had a total of 55 employees when it was acquired by
| facebook for $16B.
| ramraj07 wrote:
| Even whatsapp the app is deceptively simple. It works
| seamlessly in all the dialects around the world, on a swathe of
| devices quite possibly matched by no other app; and it manages
| to handle every actually useful feature anyone wants in a
| messaging app with amazing aplomb. In India for the majority of
| the people whatsapp is not just their communication platform,
| it's also their photo album, dropbox, task manager, social
| networking platform and contacts manager. All within the
| simplest UIs I've ever seen.
| tiffanyh wrote:
| > "We are working on a prototype, open-sourcing in November"
|
| I'm curious to see how Elixir integration with such static type
| system would work.
| seanclayton wrote:
| I absolutely love Elixir and use it all the time, I really just
| wish it had a type system (that's not the slow behemoth
| Dialyzer). I understand the difficulty of typing processes and
| I don't envy the task, but as a user of the language it would
| make it _that_ much better to work with! Just imagine working
| with a language like Elixir that has the type system of F#!
| rch wrote:
| I think it's more natural to gradually implement components as
| Erlang NIFs in Rust via https://github.com/rusterlium/rustler.
| an_opabinia wrote:
| Erlang is the Actors-model version of Greenspun's tenth rule of
| programming:
|
| "Any sufficiently complicated server backend contains an ad hoc,
| informally-specified, bug-ridden, slow implementation of half of
| Erlang."
|
| Is it really the end-all-be-all model of server architecture?
| What are microservices, if not Actors Clunkified? "Serverless"
| functions, if not the simplest actors in disguise? Docker, actors
| except over IP addresses and clunky APIs?
|
| The real takeaway: Valuable stuff is stateful. You know you're
| doing valuable work for a real human end-user if you are dealing
| with tricky state problems. Extracting money out of that is a
| bunch of developer-productivity-limited business concerns. It
| seems actors are a good model to solve that.
| davidw wrote:
| That quote regarding Erlang is attributed to Robert Virding.
| amgreg wrote:
| What I like most about some experienced engineers who have been
| at companies from the beginning is that they realize the needs of
| a small-team, cash-constrained, rapidly growing company are
| different from that of a dozens-of-teams, complex, but also
| rapidly scaling enterprise. The devil is in understanding when
| and how to transition your tooling, architecture, and -- dare I
| say it -- attitudes, from the one to the other.
|
| I am curious to hear more from these engineers; how decisions
| were made about transitioning/scaling; and what decisions were
| made.
| jacquesm wrote:
| This comment is gold. Yes, that's exactly it. Trying to use the
| tooling and processes that are a bad match for your situation
| is _the_ way to waste a fortune or to end up being overtaken on
| the right by a competitor that made smarter choices.
| ConradKilroy wrote:
| @amgreg, fantastic observation! "scalable attitudes"
| thdrdt wrote:
| There are things that could (should?) be done from the start.
| Most has to do with automation. For example automated build
| tools. Maybe you don't want automated unit tests in the
| beginning but those parts can be added later without changing
| the company's culture and stucture too much.
|
| Or automated invoices. It is much easier to change this than
| changing a co-worker who manually created all the invoices.
| outworlder wrote:
| > Most has to do with automation.
|
| This.
|
| Don't ever put anything into production that's not automated.
| Ever. "Oh it's a simple HTTP server that's going to be used
| by 5 people". Cool. Automate it. "Oh but it's going to take
| time!". I thought you said it was simple...
|
| When things are not automated, we treat them differently. If
| a server is not automated, now it becomes time-consuming to
| setup another, which means it's now a pet, not cattle. But if
| it is automated and is misbehaving, we can easily replace it
| (and save the current for troubleshooting later if
| necessary). Or we can spin up 100 of them.
|
| Same goes for development. If builds are not automated, they
| become more difficult to do, more error prone, which means
| people will avoid making contributions unless they are forced
| to do so (by some ticket). When contributing to a code base
| that has no guardrails becomes a lose/neutral proposition,
| people won't want to.
|
| And now it's been 5 years and it's difficult to automate
| things because there's a 100 different workflows that people
| use and you are going to break them.
| tangjurine wrote:
| What exactly do you mean by automate? Like the http server
| for example?
| [deleted]
| spinningslate wrote:
| > Maybe you don't want automated unit tests
|
| Interested why you'd say this, can you elaborate? Do you mean
| tests in general or unit tests specifically?
|
| I can't imagine automating a CD pipeline without having some
| automated testing as an integral step. If you don't have
| automated tests, how do you remain confident that you haven't
| introduced bugs/regressions?
|
| Even in a small codebase that's easy to do - especially when
| you're evolving quickly.
|
| Like I say: not tacit criticism, genuinely interested.
| jayd16 wrote:
| You can have automated builds and manual testing and
| deploy. It goes a long long way, especially in a small team
| that's still prototyping rapidly.
| AshamedCaptain wrote:
| I find it funny they say Whatsapp "chose" Erlang. They "chose"
| Erlang because ejabberd was written on it, and they chose
| ejabberd because of a Jabber mailing list suggestion. That's
| about it...
| btbuildem wrote:
| > some {T} | undefined
|
| Can we please avoid TS pitfalls?
| phowat wrote:
| Would you mind expanding a little bit on why this is bad ? I'm
| fairly new to TS and find myself doing it a lot.
| angio wrote:
| The advantage of the Option type is that you don't need to
| check for nullability, you simply pattern match on Some(_) or
| None. A type of `some{T} | undefined` means that the
| expression can have values `Some(_)` or `None` or
| `undefined`, which defeats the point of the Option type!
| mbo wrote:
| > A type of `some{T} | undefined` means that the expression
| can have values `Some(_)` or `None` or `undefined`
|
| Can you please explain this further? Why is `None` an
| inhabitant of this union?
| masklinn wrote:
| Except Erlang doesn't have a none. Nor an undefined for
| that matter. So `some {T} | undefined` means exactly that,
| there is no underlying null pointer which could sneak up on
| you.
|
| `undefined` here is not built-in type or value of the
| language, it's a standard atom like any other.
|
| So are null, nil, value or false (although false does have
| a special status in that it's the customary output of
| boolean operations alongside true).
| angio wrote:
| I'm aware of that, I was explaining why `some{T} |
| undefined` is problematic in Typescript. If you have a
| signature with both option type and undefined it's a
| signal of a code smell.
| btbuildem wrote:
| It can lead to cascading checks for undefined; best case
| scenario is a lot of boilerplate type code.
| masklinn wrote:
| > It can lead to cascading checks for undefined
|
| Do you know anything about Erlang? Because unlike
| typescript, `undefined` in erlang is not a special value,
| it's just an atom like any other.
|
| > best case scenario is a lot of boilerplate type code.
|
| Do you know anything about Erlang(2)? Because unlike
| typescript erlang has pattern matching, both faillible and
| not.
|
| I'm not talking "it can kinda unpack objects", erlang has
| actual pattern matching (though not unification despite the
| prolog-inspired syntax), and a simple-looking assignment is
| a pattern match (and a potential assertion).
| airstrike wrote:
| Couple things: personal attacks are not cool, and "please
| respond to the strongest plausible interpretation of what
| someone says, not a weaker one that's easier to
| criticize. Assume good faith."
|
| https://news.ycombinator.com/newsguidelines.html
| masklinn wrote:
| > Couple things: personal attacks are not cool, and
| "please respond to the strongest plausible interpretation
| of what someone says, not a weaker one that's easier to
| criticize. Assume good faith."
|
| Couple things: asking somebody if they know anything
| about a subject is not a personal attack, and that GP
| doesn't know anything about Erlang and completely
| miunderstands the snippet _is_ the strongest plausible
| interpretation of their comments which assumes good
| faith.
|
| The alternative interpretations which do not are that
| they're actively lying, or that they can't be trusted
| with any device more complicated than a rock or a small
| stick.
| dang wrote:
| It sounds like you didn't intend it as a personal attack,
| but inserting rhetorical questions like "Do you know
| anything about $foo" in an argument about $foo definitely
| comes across that way on the internet, and isn't allowed
| here. The fact that you repeated it twice makes it much
| worse--this is an attempt to expose the other commenter
| as "someone who doesn't know anything", which is
| needlessly personal.
|
| If you know more than someone else, that's great, but
| please just share some of what you know so the rest of us
| can learn. Putting others down (even implicitly) doesn't
| serve learning, it's just distracting and builds up
| toxins in the ecosystem. Consider how much better your
| comment would read without those bits. The best way to
| react to bad information is respectfully with better
| information.
| AzzieElbab wrote:
| I am curios why they need 1K people instead of 10? What changed
| on the server side of WhatsApp?
| user5994461 wrote:
| No global service can run with 10 people. Just to have a good
| coverage 24/7 you're looking at a 50 people company.
| oblio wrote:
| Yeah, just ops would be 3 timezones x 8 hours x 2 (ideally 3)
| people per timezone. So that's 6-9 people just for a
| barebones ops team.
| [deleted]
| abrolhos wrote:
| I understand that the 1K+ engineer number comes from Facebook
| as a whole, when mentioning the use of Hack as a typed PHP.
|
| More like to give a perspective on the transition on how to
| come from a small company into a huge one.
| [deleted]
| user5994461 wrote:
| WhatApps got audio and video calls after the Facebook
| acquisition.
|
| Usage increased by a few orders of magnitude too.
| ramraj07 wrote:
| Scaling, reliability, support, dialects and localisation,
| accessibility, edge cases, device support, security, moderation
| (not just moderators but moderation tools) infrastructure,
| research and exploration, monetization, commercialization.
| Basically everything no one in a startup thinks about for years
| if not more. And to that add the simple fact that as the team
| grows the per engineer productivity just has to drop due to
| communication overhead among other things.
| stevencorona wrote:
| I absolutely love the BEAM and Elixir, but one of my common
| complaints and source of errors is the lack of compile-time type
| sanity checking, so I'm excited to see movement in this
| direction. Dialyzer is sorta okay, but the speed, cryptic errors,
| and syntax have kept me from fully embracing it.
| findjashua wrote:
| Gleam may be what you're looking for:
|
| https://gleam.run/
| faitswulff wrote:
| I'd be curious how WhatsApp's internal statically typed
| Erlang solution looks compared to Gleam. If I had to guess
| based on current trends, I'd guess that it's a gradually
| typed, opt-in toolchain a la Sorbet or Typescript, in order
| to maintain backwards compatibility and gradually update
| their legacy codebase as opposed to a whole different
| language.
| callamdelaney wrote:
| This isn't ready for prime time yet iirc.
| sodapopcan wrote:
| It's not but is in constant development. The author is very
| active on elixirforum.com
| derefr wrote:
| Last I heard (years ago), I though that WhatsApp's backend was
| being rewritten in some other language, because Facebook didn't
| think maintaining an Erlang codebase was tenable. Did that
| change?
| [deleted]
| ramchip wrote:
| Perhaps you're thinking of Facebook chat?
|
| https://www.quora.com/Why-was-Erlang-chosen-for-use-in-Faceb...
|
| https://www.quora.com/When-did-Facebook-switch-away-from-usi...
| kornish wrote:
| One of my favorite talks is by Rick Reed about scaling Erlang at
| WhatsApp. What an absolute savage. He flies through an articulate
| and in-depth curriculum on system performance and bottleneck
| mitigation.
|
| The talk is called "That's 'Billion' with a 'B'" and makes for a
| great lunchtime watch:
| https://www.youtube.com/watch?v=c12cYAUTXXs
| staticassertion wrote:
| Yeah, not quite related to the linked slides, which are about
| scaling in terms of productivity, but still a great talk.
|
| I really found that "isolation" is just a key optimization and
| one of the most useful properties of a system, which they call
| out in that talk. I wrote about it a bit more in depth here:
| https://www.graplsecurity.com/post/architecting-for-performa...
|
| I've written our data processing layer and our event
| orchestration layer as basically an actor oriented system, with
| push/pull systems in a couple of key places where it makes
| sense. It's incredible what you can do with strong isolated
| infrastructure in terms of performance, security, reliability,
| and quality.
| secondcoming wrote:
| I know nothing about Erlang, but I do know high-volume systems.
| Needing 550 beefy servers to handle all that load is - in my
| mind - not that impressive.
| strmpnk wrote:
| I remember that year. I was giving a talk at EF during the same
| time slot but the schedule originally had me in the large room
| and they had a much smaller one.
|
| When the news of the acquisition hit, everyone wanted to see
| the WhatsApp talk. The organizers knew this so we swapped
| rooms. So, I started my talk by asking if anyone in the room
| was here for the WhatsApp talk and told them they could quietly
| leave and I wouldn't mind and a bunch of people got up.
|
| Heheh. I don't blame them. I didn't really like my talk and
| Rick Reed is very good at what he does and the talk is no
| exception.
| gautamcgoel wrote:
| Can someone explain what Erlang offers that Go does not? Both
| have great support for concurrency, but Go has type checking and
| a more familiar syntax.
| [deleted]
| [deleted]
| emit_time wrote:
| Doesn't erlang have a REPL, allows you to do live deployments.
| Individual processes crashing doesn't bring everything down,
| you can tell it how to handle those. Dynamic typing. Functional
| programming. Concise syntax, with "let it fail" mentality.
| epidemian wrote:
| There's this talk, The Sould of Erlang and Elixir, that opened
| my eyes to what makes this platform great:
|
| https://youtu.be/JvBT4XBdoUE
|
| It's not so much about what Erlang, as a language, "offers that
| Go does not". It's about the BEAM VM that runs it that makes
| all the difference.
|
| I very much recommend watching the talk if you're interested in
| the topic :)
| coder543 wrote:
| I watched that talk just now, and it was basically a
| retelling of everything I really like about Go. Except, Go is
| statically typed, and a lot faster than BEAM in benchmarks
| that I've seen. Go is also a very "normal" language, so it's
| usually easy to onboard developers into it, no matter what
| their background is. JetBrains GoLand makes the IDE
| experience essentially impeccable. I can refactor code with a
| few clicks, and GoLand precisely updates all references
| thanks to the static type system.
|
| Go can run massive numbers of Goroutines on a single system,
| and it preemptively timeslices individual Goroutines, so I'm
| never worried about any single task preventing other tasks
| from making progress, which was like 75% of that talk.
|
| Go has a _fantastic_ production runtime debuggability story,
| which was the other 25% of that talk.
|
| Go's pprof support feels magical:
| https://golang.org/pkg/net/http/pprof/
|
| You can connect to running systems and see flamegraphs of
| where CPU time is being spent, you can take traces of the
| goroutines to understand why the system is idle when you
| think it shouldn't be, you can see how much memory different
| parts of the system are using, and so much more. It's not
| mentioned as prominently as I think it should be, but `go
| tool pprof` has this `-http=:` option that will open a web
| browser to let you just see everything graphically and click
| around, and it's an amazing experience.
|
| There were really only three things in there that Go doesn't
| offer:
|
| 1. Forcefully terminating processes
|
| Go doesn't allow anyone to forcefully terminate a Goroutine
| except the Goroutine itself. Personally, if I identified a
| buggy Goroutine, I would just focus on writing a patch and
| deploying it. Manually terminating a process is cool, but
| then the user types "-1" again and again, and you're just
| sitting there playing whack-a-mole instead of fixing the
| problem and deploying the fix.
|
| 2. Patching the running system without restarting it
|
| The advantages of this are difficult to enumerate. It sounds
| nice and clean conceptually, but I would worry about existing
| processes / goroutines being in unexpected states because of
| erroneous code that they had previously run. Having a way to
| gracefully shut down a system and restart it without abruptly
| terminating responsive client connections is much more
| predictable to me.
|
| 3. Seamlessly distributed execution
|
| He briefly talks about distributed Erlang towards the end of
| the talk. but mentions that it has serious problems and that
| a number of people don't recommend using it in production.
| Personally, even if it were nearly perfect, I think the
| benefits of it are limited. It's not difficult to have nodes
| communicate with each other using any number of other things,
| including proper RPC libraries all the way down to just raw
| TCP connections. The hard thing is service discovery, and I
| don't think distributed erlang meaningfully solves the
| problem different than other solutions. If everything is
| running homogeneous code, why bother your nodes with knowing
| about the existence of other nodes? Just use a load balancer,
| and then do everything internally on each node. If your
| systems are heterogeneous, you need a way to send the right
| messages to the right nodes, and that's where a named load
| balancer for each type of node can do just fine.
| tokenrove wrote:
| I've worked a bunch in both, and Go really doesn't compare;
| you can't judge based on watching a talk. I find it much,
| much easier to write safe, robust code in Erlang, and
| operationally, you have so much more power to inspect the
| state of the BEAM. Another key BEAM thing is the process
| heap, which amounts to region allocation when used
| carefully. The reduction mechanism used for scheduling
| processes, though crude, is also still much lighter-weight
| than Go's approach.
|
| On the other hand, Erlang is a weird language and a weird
| environment, that takes a lot of time to really understand,
| so I understand why it's never going to "win", and golang
| tools and the runtime will continue to improve. YMMV.
| coder543 wrote:
| > The reduction mechanism used for scheduling processes,
| though crude, is also still much lighter-weight than Go's
| approach.
|
| I'm not sure how it could be lighter weight than what Go
| has implemented natively at the machine level, instead of
| in what amounts to a bytecode interpreter. If you have
| any links, I would be very interested to read more.
|
| > I've worked a bunch in both, and Go really doesn't
| compare
|
| I respect your experience, so it is probably a failure of
| imagination on my part that I simply can't imagine how
| the BEAM tools could be that much better. I haven't had a
| problem with a rogue task (or really anything else) in
| production with Go where I was unable to figure out what
| was going on extremely quickly. The tooling has been
| amazing for me, unlike basically every other language
| I've worked seriously with. The YouTube talk certainly
| didn't do the BEAM tools justice if they are that much
| better.
|
| With Go, I actually get really nice GUIs to look around
| the running Go process and analyze the situation, instead
| of just an interactive CLI session where I have to craft
| my own commands to find the top tasks and such. The Go
| pprof tools also work as an interactive shell (but for
| analysis, not for remote arbitrary code execution), but I
| would rather just have the flamegraph in front of me 99%
| of the time. I fully admit that I've never had a chance
| to use Erlang/Elixir/BEAM in any meaningful way, but I
| have tried to understand what they offer, and I haven't
| seen the compelling magic that some people talk about.
|
| Now, if someone is running a Go service _without_ the
| HTTP pprof server running on some port that they can
| access, then yes... it wouldn 't even come close to
| comparing to what BEAM offers when you have the option to
| connect to a running BEAM instance.
| tokenrove wrote:
| And, note, I'm not saying Go is bad, here, just that
| there is a lot that is underrated and misunderstood about
| Erlang.
|
| On the Erlang side, check out the BEAM Book chapter on
| scheduling: https://blog.stenmans.org/theBeamBook/#CH-
| Scheduling and the core scheduling loop in the BEAM: http
| s://github.com/erlang/otp/blob/master/erts/emulator/beam.
| ..
|
| On the Go side, check out https://github.com/golang/go/tr
| ee/master/src/runtime/proc.go and the asm*
| implementations.
|
| It's been a little while since I looked at it, but I
| recall that much less state had to be saved in an Erlang
| process switch in the usual case; I seem to recall it can
| be done in a handful of instructions in many cases. Go of
| course has to save a bunch of registers much as you'd
| have to do in any native context switch.
|
| Edited to add: it can be useful to look at that part of
| the BEAM disassembled in objdump or gdb, to appreciate
| it, since it's hard to tell how much work is happening
| with all those macros.
| sodapopcan wrote:
| I only started with Elixir in recent months and it's the
| first language that has ever made me comfortable about
| writing concurrent code. I didn't spend a lot of time
| with Go, but the idea of just calling a function that is
| now running in parallel but was always disconcerting to
| me. Of course, I could have spent more time with it and
| gotten more comfortable learning the ins-and-outs, but
| Erlang/Elixir's addressable processes running with their
| own stack/heap/gc and passing messages between each other
| is something that clicked very quickly with me. It's such
| an incredibly simple idea. For being a "weird" language,
| I think there is a lot of power the simplicity of its
| design, especially around learning. You just have to get
| over the weird syntax (which is a hot topic).
|
| For transparency, I've never written any production code
| in either Elixir or Go.
| rightbyte wrote:
| The funny thing is that Erlang is the Ericsson language.
| Go is the Google language.
|
| Ericsson never had the Google "street cred" or K& not R.
| Ericsson is boring phone interchanges and boring radio.
| Google was hipster latte with software freedom before it
| turned out to be worse than MS, IBM and Oracle combined.
|
| One might wonder what could have been without all the
| smoke and mirrors.
| [deleted]
| juhatl wrote:
| One major thing would probably be OTP, which would be very
| difficult to implement as-is on top of Go. A recent comment
| thread [1] over at Lobste.rs explored this in more depth.
|
| Sasa Juric made an absolutely wonderful hands-on presentation
| [2] of what (oftentimes rather unique) value propositions
| Erlang/Elixir/BEAM bring to the table. It's a very tightly-
| packed presentation, but I strongly recommend having a look if
| you're curious.
|
| 1.
| https://lobste.rs/s/ntati1/even_go_concurrency_is_still_not_...
|
| 2. https://www.youtube.com/watch?v=JvBT4XBdoUE
| Jtsummers wrote:
| Erlang is a system and language for distributed (not just
| concurrent) programming. Go is a language for concurrent (not
| distributed) programming. Unless things have changed
| substantially since I last investigated Go, the language does
| not sit within a system designed for distributed computing.
| There are libraries you can use, but the language itself does
| not attempt to solve the same problems as Erlang.
|
| If you use Erlang within one node, it's roughly equivalent to
| Go. But Erlang is designed so that the same concurrency
| mechanisms you use on one node also work across a cluster of
| nodes. There are more differences than this, but this is
| probably the most critical one.
| ericlewis wrote:
| So you're saying erlang can automatically distribute async
| tasks across nodes? Probably too much to comment, but how or
| where can I learn more about this way of thinking of it?
| Jtsummers wrote:
| Not automatically, but if you connect multiple nodes into a
| cluster, when you spawn a process you can select which node
| it will be spawned on. Additionally, once the nodes and
| processes are running you can send messages across nodes to
| different processes using the same method you use for
| processes on a single node. [0] Shows an example of
| communication across nodes, as well as some preliminary
| material that would be helpful if you don't know Erlang.
|
| I mostly learned Erlang from Joe Armstrong's book [1],
| though the first edition. I'm confident the second edition
| is as good as the first. _Learn You Some Erlang_ [2] is a
| great free (also available for purchase, but free online)
| book.
|
| [0] https://erlang.org/doc/getting_started/conc_prog.html
|
| [1] https://pragprog.com/titles/jaerlang2/programming-
| erlang-2nd...
|
| [2] https://learnyousomeerlang.com/
| [deleted]
| masklinn wrote:
| * Erlang builds _reliability_ into the language and runtime
| through concepts of liking, supervision, ...
|
| * Erlang is a much simpler language
|
| * Despite lacking a static type system, Erlang's functional
| bend, immutability, SSA and pattern matching makes for a rather
| safe language
|
| * Erlang has always been fully preempted, it didn't have to
| wait until 2020 for that to happen
|
| * Erlang also builds distribution in, though it's "trusted
| distribution" so probably of fairly limited use at large
|
| * Erlang uses shared-nothing concurrency, greatly limiting
| side-effects across processes, which again helps with
| reliability: there is no action at a distance in sequential
| erlang, no third-party can come in and mess with your state
|
| * when dealing with binary data, binary patterns are absolutely
| amazing (and rather unique)
| coder543 wrote:
| > * Erlang is a much simpler language
|
| Go is widely considered to be a very simple language, often
| too simple, so I find this claim interesting.
|
| > * Despite lacking a static type system, Erlang's functional
| bend, immutability, SSA and pattern matching makes for a
| rather safe language
|
| This is fine stuff, but I would rather have a type system, so
| this is just making up for a deficiency with some other
| stuff, so it's really just a sidegrade, not so much an
| upgrade or a downgrade.
|
| > * Erlang has always been fully preempted, it didn't have to
| wait until 2020 for that to happen
|
| With extremely rare exceptions, Go has _always_ been fully
| preempted from a developer experience point of view. The fact
| that the underlying implementation was using function calls
| as yield points simply didn 't matter. The fact that it is
| fully preempted now means it's kind of weird to still hold it
| against Go that they are doing the same thing as BEAM now.
|
| It is _done_ , this is no longer a bullet point.
|
| > * Erlang also builds distribution in, though it's "trusted
| distribution" so probably of fairly limited use at large
|
| Go has _always_ "built distribution in" by offering to build
| single static binaries. If anything, BEAM languages are
| _harder_ to deploy.
|
| > * Erlang uses shared-nothing concurrency, greatly limiting
| side-effects across processes, which again helps with
| reliability: there is no action at a distance in sequential
| erlang, no third-party can come in and mess with your state
|
| I agree that immutability is an advantage. In general, third
| parties can't come in and unexpectedly mess with your state
| in Go either, as long as you're either maintaining your state
| local to a function or keeping your global variables private
| to each package.
|
| Keep in mind that Go's package imports must form a tree with
| no loops. Third party packages literally cannot see your code
| or any of the exported global variables, since you are
| importing them, and not the other way around.
|
| Go does not offer any way for packages to reflect over the
| whole program's state in memory, so there's no way for third
| party packages to go hunting around in memory and playing
| with your values, which would be stupid, unless they use
| "unsafe" to start poking at random memory addresses, and I'm
| certain you can do something equally dumb in Erlang with a
| sufficient amount of effort. It's just as crazy to think of
| doing something useful that way in Go as it is in Erlang.
|
| > * Erlang builds reliability into the language and runtime
| through concepts of liking, supervision, ...
|
| This is literally the only compelling talking point I've seen
| for Erlang/BEAM vs Go.
|
| However, I've written large Go applications with distinct
| services running within the same binary, and it is possible
| to make extremely reliable systems. Go's context package
| allows you to build trees of processes that you want to be
| able to cancel, including cancelling specific sub-trees
| without affecting adjacent trees or parent trees.
|
| The single biggest footgun with building reliable systems in
| Go is that it _prefers_ to crash early. So, every Goroutine
| that you launch, you must make sure that the first thing it
| does is catch any panics to avoid taking down the entire
| program, including unrelated Goroutines. When correctly
| written, a human will be immediately alerted to the bug that
| caused this panic (which shouldn 't ever happen anyways), and
| the program will continue operating normally in the mean time
| until a human gets a chance to push out an update to fix that
| bug.
|
| In practice, enforcing just one rule (Goroutines must catch
| panics) in code reviews is not that hard, and it isn't common
| in my experience for most code to arbitrarily launch
| Goroutines -- Goroutines should be managed.
|
| In return, Go offers much better performance than BEAM, as
| well as most of the practical benefits of BEAM that I've seen
| discussed online.
|
| I think Rust is a better language than either Go or any BEAM
| language, but the absence of a preemptive concurrency model
| like Go and Erlang offer just makes it hard for me to enjoy
| the prospect of using Rust for general purpose network
| services.
|
| Once Go adds generics in the next year-ish, that should solve
| my biggest real annoyances with the language. I really look
| forward to being able to package up a number of the
| concurrent patterns I write into reusable packages, which
| will largely eliminate once and for all the need for me
| manually spin up a new Goroutine in the code that I'm writing
| and potentially forget to catch any panics.
| Jtsummers wrote:
| > > * Erlang also builds distribution in, though it's
| "trusted distribution" so probably of fairly limited use at
| large
|
| > Go has always "built distribution in" by offering to
| build single static binaries. If anything, BEAM languages
| are harder to deploy.
|
| I suspect GP meant distribution in the distributed
| computing sense, not the deployment sense. What, baked into
| Go not via a library, handles multiple Go _programs_
| communicating within the same physical node or across
| multiple physical nodes? Erlang gives you the _same_
| communication method whether distributed or within a single
| node, Go does not (that is, Go channels are internal to a
| specific program, they cannot be referenced from different
| programs or computers).
|
| If they did mean it in the deployment sense, I'd still
| argue it's a point in favor of Erlang. Two nodes
| communicating can execute code on the other.
| coder543 wrote:
| Yes, I definitely misunderstood the point being made
| there.
|
| > What, baked into Go not via a library, handles multiple
| Go programs communicating within the same physical node
| or across multiple physical nodes?
|
| Go does have an RPC package built into the standard
| library, for what it's worth, but I know it's not the
| same thing as the runtime transparently stitching
| distributed machines together under the hood.
| masklinn wrote:
| > This is fine stuff, but I would rather have a type
| system, so this is just making up for a deficiency with
| some other stuff, so it's really just a sidegrade, not so
| much an upgrade or a downgrade.
|
| Sure, I'm just saying the lack of a type system in erlang
| is not quite as bad as, say, in javascript.
|
| > Go has always "built distribution in" by offering to
| build single static binaries. If anything, BEAM languages
| are harder to deploy.
|
| Distribution in the sense of distributed computing, not in
| the sense of giving somebody else binaries to run. _That_
| is not something erlang is good at. But you can rather
| easily join nodes together and not just have them
| communicate but have them leverage one another e.g. spawn
| processes on an other node in the cluster.
|
| > I agree that immutability is an advantage.
|
| Immutability and isolation are different things, though
| Erlang has both.
|
| > In general, third parties can't come in and unexpectedly
| mess with your state in Go either, as long as you're either
| maintaining your state local to a function or keeping your
| global variables private to each package.
|
| Go is a shared-memory concurrency language, with no
| restrictions on the sharing.
| coder543 wrote:
| > Distribution in the sense of distributed computing, not
| in the sense of giving somebody else binaries to run.
| That is not something erlang is good at. But you can
| rather easily join nodes together and not just have them
| communicate but have them leverage one another e.g. spawn
| processes on an other node in the cluster.
|
| Ah, I misunderstood, sorry. Personally, I see limited
| benefits to making things fully distributed at the
| runtime level. I'm sure it's nice sometimes.
|
| > Go is a shared-memory concurrency language, with no
| restrictions on the sharing.
|
| Sharing is the critical word there. You don't _have_ to
| give access to that shared memory to anything, but the
| point I was making is that nothing can _take_ shared
| access to your stuff without you handing it to them. It
| 's not the wild west of interpreted languages where any
| piece of code can access all of your code and variables
| and edit them just for fun without you realizing.
|
| Go also supports channels. How you design your system is
| up to you, and doesn't BEAM also support shared memory
| concurrency? A little thing called ETS? I would guess
| that there are other escape hatches as well, but I'm not
| familiar enough with Erlang or the BEAM to say for sure.
|
| If you want to make a poorly designed system, I'm sure
| that Erlang won't stop you, but it will obviously try to
| guide you into certain accepted patterns by making other
| things harder.
| toast0 wrote:
| > Personally, I see limited benefits to making things
| fully distributed at the runtime level. I'm sure it's
| nice sometimes.
|
| I think it's nice because when you write code as
| processes that receive requests and send responses, they
| can handle requests from the same node or different nodes
| with no change in code, you only change where you start
| that process (and make sure you register the process
| somewhere either as a named registered process or pg2/pg
| or whatever). Certainly there can be some scaling
| difficulties (see Rick Reed's talk linked elsewhere), but
| it generally works. And, you can move the processes while
| the system is running to address changing needs. Building
| on top of asynchronous messaging makes it easy to do
| things like normally wait for a reply after sending for a
| synchronous feel, but occassionally doing multiple sends
| and then waiting for parallelism, or just sending without
| waiting for a reply when that's appropriate.
|
| > BEAM also support shared memory concurrency? A little
| thing called ETS? I would guess that there are other
| escape hatches as well, but I'm not familiar enough with
| Erlang or the BEAM to say for sure.
|
| Ets is built with shared memory, yes, but interaction
| with your code is as if you were messaging a table
| process; the data you get back from an ets read won't
| change while you have it, but it might not be what you
| wrote, because another process may have written something
| else; same as if another process messaged a storage
| process in between your write and read. ETS is less of an
| escape hatch and more of a performance optimisation --
| acknowledging shared memory concurency is helpful for
| data storage and retrieval, but it's tricky to get right
| and should be written once, so BEAM doesn't give the
| tools to do it and it needs to be done in C.
|
| You can also do fun things with erlang:process_info or
| tracing to snoop on some data from other processes (but I
| don't think modify?). And, BEAM allows native code (NIFs)
| which is the ultimate escape hatch --- you can do lots of
| fun things there, of course.
| coder543 wrote:
| > Ets is built with shared memory, yes, but interaction
| with your code is as if you were messaging a table
| process
|
| To me, this is a distinction without difference.
|
| Shared, mutable memory access without some form of
| synchronization is a bug and is always invalid.
|
| You either hold a lock the entire time you're using a
| shared value, or you use a lock to take a copy of the
| value and then go about your business. With the singular
| exception of atomic values, the value should _never_
| change while you have it without you taking an explicit
| action to update it, and atomics are rarely used outside
| of either metric tracking or very specialized code.
| Besides atomics, everything you described about ETS is
| the same as any shared memory system.
| toast0 wrote:
| > Shared, mutable memory access without some form of
| synchronization is a bug and is always invalid.
|
| That doesn't prevent it from being really easy to write,
| and sometimes very hard to notice in a lot of languages.
| In BEAM languages, you can, of course, set up similar
| bugs through the use of other processes (or ETS), but you
| have to work harder to do it wrong than to do it right.
___________________________________________________________________
(page generated 2020-09-11 23:00 UTC)