[HN Gopher] Bootstrapping a multiplayer server with Elixir
___________________________________________________________________
Bootstrapping a multiplayer server with Elixir
Author : thibaut_barrere
Score : 148 points
Date : 2021-07-29 16:31 UTC (6 hours ago)
(HTM) web link (elixir-lang.org)
(TXT) w3m dump (elixir-lang.org)
| s3cur3 wrote:
| Hey folks, I'm Tyler! I'm the guy who did the original work on
| this. I'm happy to answer any questions!
|
| You might also be interested in the discussion happening on the
| associated Twitter thread:
| https://twitter.com/elixirlang/status/1420782381825503240
| Zababa wrote:
| Thanks for the article, that was a nice read! How much
| concurrent player do you think you could handle at most?
| s3cur3 wrote:
| The limitation is more about the number of players we can
| have in the same area--if you had a bunch of players equally
| spaced around the globe, I'm pretty sure we'd saturate the
| Ethernet link before running out of CPU on a big (say,
| 32-core) VM. The CPU cost (and outbound network bandwidth) of
| a player scales with the _square_ of the number of other
| players who can potentially "see" them.
|
| In my scalability testing, we could handle upwards of 1000
| players in the same 50 nm by 50 nm square on my 8 core dev
| workstation. That's pretty far beyond the limit of what would
| actually be _fun_ to fly the sim with. It becomes pandemonium
| even with 100 planes in the same area. We could easily do a
| dozen pockets of 100 planes in the same area even on our
| current 8 core VM.
| Zababa wrote:
| Thank you for the detailed answer. 50 nm here is 50
| nautical miles I assume?
| s3cur3 wrote:
| Correct! Sorry about that. :)
| bullen wrote:
| Do those 1000 send to all 1000 or are those 50nm squared
| partitioned so only players in a certain range see each
| other?
| s3cur3 wrote:
| In general, we send each player all the planes that their
| in-game map could possibly show (which is usually
| something like 140 statute miles by 140 statute miles).
|
| We could one day be smarter about this and only send you
| everything if your map is actually open, and otherwise
| limit it based on your field of view. That hasn't been a
| priority yet, though.
| Bayart wrote:
| >In my scalability testing, we could handle upwards of 1000
| players in the same 50 nm by 50 nm square on my 8 core dev
| workstation. That's pretty far beyond the limit of what
| would actually be _fun_ to fly the sim with.
|
| But it would make a great fit for an MMO. AFAIK there
| aren't any of those now levering Erlang / Elixir. If
| anything they're doing the opposite and trying to contain
| people in smaller and smaller instances.
| Simonflare wrote:
| Who else puts Elixir in their Tinder bio?
| cpursley wrote:
| Of course, it's the best way to get lots of concurrent
| connections!
| conradfr wrote:
| And hopefully a lot of pattern matching?
| Thaxll wrote:
| Building a gameserver in Elixir is not great imo, performance is
| subpar. I work in online games and no one would want to use that
| language.
|
| I guess they have low constrains since they only need to support
| 10000 players at a time and they want a single instance for that.
|
| - Erlang runtime is slow
|
| - Memory consumption is high
|
| - GC is not very good ( vs Java / C# / Go )
|
| - high cost in term of performance because of immutability
|
| It's not the kind of things you want for a game.
|
| To give you some perspective on popular FPS games, they run
| between 30hz and 120hz with up to 64 players and usually only use
| a single CPU core. And they're doing gameplay simulation / state
| replication / physics sometimes AI etc ...
|
| Also it must be pretty painful to not re-use their C++ client
| code on the server side, they have to re-implement twice? ( this
| concern is actually higher than the performance one )
|
| Overall I think it's an impressive achievement to be able to do
| that for a single person in 6 month!
| andrewmcwatters wrote:
| Only on HN can you provide specific industry advice and be
| downvoted for your insight because your opinion wasn't
| agreeable.
|
| I upvoted your comment to bring it out of 0 or less because
| these concerns are valid and critically make or break important
| when building multiplayer servers.
|
| My relevant background here is in building multiplayer server
| software where a minimum of 1000 concurrent players is a
| required goal metric.
| Zababa wrote:
| I don't think the downvoting is linked to opinions, it's
| linked to facts. Why should "specific industry advice" be
| preferred over working code? That sounds dogmatic to me.
|
| > My relevant background here is in building multiplayer
| server software where a minimum of 1000 concurrent players is
| a required goal metric.
|
| They are handling 10k concurrent players with low resources
| usage.
| andrewmcwatters wrote:
| That's really great, but as soon as you understand their
| specific requirements it's simple to go from feasible to do
| 100000 concurrent clients to impossible with compute budget
| constraints.
|
| You can build a "game" with a million "concurrent" clients
| that do absolutely nothing, to dealing with thousands where
| you have tick rates that make that impossible.[1]
|
| [1]: https://www.planimeter.org/grid-
| sdk/api/Tick_rate_and_bandwi...
| Zababa wrote:
| I don't really understand your point. Are you saying that
| the resources used will scale exponentially with the
| number of players? If that's the case, I can see how this
| could make things difficult in the long run but caring
| about it now sounds a bit like premature optimization to
| me. Maybe the players don't even want to be all on the
| same world?
| You-Are-Right wrote:
| You do not understand his point.
| andrewmcwatters wrote:
| Not interested in explaining rudimentary concepts to
| someone who thinks my work isn't based in reality. Do the
| math yourself.
| Zababa wrote:
| Sorry if I offended you.
| s3cur3 wrote:
| YMMV, but:
|
| - Memory consumption in production peaks at like 300 MB for us
|
| - Scalability (i.e., being able to go wide on, say, a 64-core
| CPU) was more important to us than raw, synchronous speed
|
| - Garbage collection is not something I've ever had to think
| about with Elixir (which hasn't been my experience with Python
| and Java)
|
| - Because it's a sim and not something like a FPS, we don't
| have to worry about cheating (what would cheating even mean?),
| so we don't have to do any physics on the server side
|
| - Despite being way more familiar with C++, I can't imagine
| going from zero to production in 6 months (or even 2 years) if
| we'd tried to do this in C++
| andrewmcwatters wrote:
| > what would cheating even mean?
|
| This is such a rudimentary question in game development that
| I don't know how you can't answer it. Mainstream server
| authoritative design has been a thing since 1999, and
| probably even longer when you look beyond Quakeworld.
|
| All it takes is a few malformed packets from a script kiddie
| and then you realize you have to do distance checking and
| other anti-cheat functionality.
|
| "Cheating" in gamedev doesn't specifically mean there is a
| win scenario and you need to make sure people don't shortcut
| that and play unfairly.
|
| It means in this case that someone can't connect to your
| server and transport their plane to another nearby player and
| wave it around in their flight path to annoy others while
| they spam chat to grief people for fun and to produce a
| YouTube video.
|
| And those checks are pretty simple.
| s3cur3 wrote:
| Sorry, to clarify, our thinking on this point was: our
| players aren't particularly motivated to lie to the server
| since there isn't a "goal" to a flight simulator. Given the
| choice between shipping (much, much) faster and completely
| locking down the experience against the vague, yet-to-be-
| realized threat of people sending fake packets to the
| server, we opted for the former.
| networked wrote:
| Have you had any reports of multiplayer griefing so far?
| I wonder if griefing happens in serious flight simulators
| like _X-Plane_ and what form it takes if it does. (If
| there are griefers, then they might conceivably find a
| way to exploit this. But I am asking mostly because I am
| curious about what the player community is like.)
| s3cur3 wrote:
| Griefing, yes. The audience for our mobile app skews
| young and, shall we say, less "serious aviators" than the
| desktop sim, so to some degree people _like_ that you can
| buzz a 747 on short final in an F-22. :D
|
| Long-term we've talked about things like a reputation
| system, where flying responsibly would earn you points to
| get into a separate "world" filled with people who also
| want to do more serious flying.
| nerdponx wrote:
| I would also wonder about vectors for causing vandalism
| (botting your client to write racist words in the sky
| with a fleet of stunt planes emitting smoke trails), or
| other general forms of malice (trying to DoS or otherwise
| crash the server).
|
| Presumably X-Plane is small enough and "boring" enough
| that your only malicious interactions would be only lazy
| drive-by attempts, but those kinds of situations would be
| more concerning to me than someone modding the client to
| make their plane fly at Mach 7.
| andy_ppp wrote:
| You're misunderstanding, there isn't the concept of winning
| in a simulator like this. It's not goal driven it's
| practicing modelling the real world not showing how many
| points you get when you land a plane.
| andrewmcwatters wrote:
| I'm not misunderstanding, it's just clear that they have
| virtually no requirements. As soon as you have any
| meaningful requirements, this entire architecture
| probably falls apart, let alone the fact that you can't
| hire for it, as no one is going to want to work on your
| obscure game server.
|
| The criteria are also so brittle at those concurrent
| numbers that if the game decided to do any more than what
| they do today, you would face engineering challenges more
| difficult than other game server software.
|
| As you deal with more and more concurrent clients there
| are calculable limits to how you can facilitate more
| features and how bandwidth and server frame time budget
| limits you.
| Zababa wrote:
| That's just a "No true Scotsman". Your definition of
| "virtually no requirements" also seems to be circular
| with this architecture failing or not. People have
| created a chatroom with 2 millions people connected at
| the same time with Elixir on a single box
| https://www.phoenixframework.org/blog/the-road-
| to-2-million-.... Your ideas about performance are not
| based in reality.
| andrewmcwatters wrote:
| As soon as you have to broadcast deltas on object
| properties to visible sectors, you immediately hit
| precisely measurable metrics on what is possible on
| various types of connections.
|
| If you broadcasted 8 bytes of deltas to 2 million
| "players" concurrently 120 times per second, you're going
| to have problems.
|
| Experience shows there's just no reason to talk to people
| about these specifics because even though they've never
| implemented it before or even done the math, you
| definitely know better than I do.
|
| @rehm: I mean, you can call me an armchair specialist
| because you're angry, but I've shipped this stuff and
| it's still used today. I employ people who can do that
| sort of basic math. Your truisms aren't helpful in
| engineering discussions, it's just noise. Yes of course
| products' requirements vary: that's not the discussion.
|
| @Zababa: It's not a "fantasy," unless you're not doing
| anything besides recording vec3 and transmitting it on a
| regular rate, most games have these concerns today. It is
| a specific function of major game engines to deal with
| this type of processing and more bespoke than not to not
| deal with it at all.
|
| Show me a real-time game that runs at even 20 tick that
| doesn't deal with these constraints.
|
| I'd love to know what fantasy world you live in where
| these things are not engineering problems that people
| haven't worked on for decades.
|
| I wish people who didn't have any experience whatsoever
| with these systems would stop trying to hold discussions
| with people who do on HN because it just makes the
| experience pander to the lowest common denominator.
| remh wrote:
| You are being disagreeable. Requirements are
| requirements, sometimes they are easier, sometimes they
| are more complex. But you dismissing an engineer who
| successfully shipped something on production that met
| their requirements because you think you have "industry
| advice" is pure arrogance.
|
| Every piece of software is written with the actual
| requirements in mind, defined (usually) by the people who
| know the context they are working with the best. Not by
| armchair specialists.
| Zababa wrote:
| I don't understand your point, you're building a strawman
| and then attacking it. This architecture works for them,
| and will probably keep doing so as you can see on the
| other response.
|
| > Experience shows there's just no reason to talk to
| people about these specifics because even though they've
| never implemented it before or even done the math, you
| definitely know better than I do.
|
| But they DID implement it and it works perfectly well for
| them. I don't understand why you're splitting hairs about
| fantasy scenarios. If they had a different problem, they
| would solve it differently. If you've faced something
| like that, feel free to share you story, what were the
| constraints and how you solved them.
| elteto wrote:
| No need to get aggressive. It's a high fidelity simulator.
| There is no goal, no missions, no points, no wins or
| losses. You just fly.
|
| There is probably very little the server is actually doing.
| Player state as seen from the server is probably just x,y,z
| coords, plane type, etc. Just the bare minimum to be able
| to draw another player in the game.
|
| For that use case Elixir (or Go, or maybe even Python with
| more computational resources) is a very good fit.
| jay_kyburz wrote:
| Yes but lets acknowledge that it's not really a game, and
| that nobody should do this for any kind of public game
| with winners and losers.
|
| A multiplayer server needs to run the simulation for
| every player and be the authority, the clients should
| simply collect input from the player and run a "lite"
| simulation to hide latency.
| JoeyJoJoJr wrote:
| No, not every multiplayer game needs an authoritive
| server. It depends largely where your priorities are. For
| myself, having a base of players with some cheaters would
| be a great problem to have.
| Zababa wrote:
| They did implement their game server in Elixir, and it performs
| great for them in reality. From what I understand, they are
| supporting 10 000 players with a low CPU and memory usage, so
| if this scales linearly they should be able to support 100k
| players. You're also ignoring the reliability aspect, which is
| why they chose Elixir (and the BEAM) in the first place.
| Performance is also less of a problem since they're on the
| server, and thus can add more hardware as needed.
|
| Could you share a bit more about your experience in the
| industry? I think you and they have different use cases, which
| makes you reach different conclusions.
| ravi-delia wrote:
| I don't disagree with you on running games on the BEAM
| (although slow is a little too far imo), but in this case what
| they're doing is using Elixir to connect clients with the
| server. The actual physics simulation and such isn't running on
| Elixir.
|
| edit: I just read through the article, and it doesn't seem to
| say one way or the other. 99% sure they aren't running any kind
| of physics engine on the BEAM, but there isn't anything
| specific
| Thaxll wrote:
| The article is no clear on what the server is actually doing,
| for example just replicating state between players through
| that hub or it's doing simulation.
|
| Either way they had to re-implement Racknet which is annoying
| because from now on everytime they add / change something in
| the network layer they won't be able to re-use the client
| code in the server.
|
| I'm curious on why they did not go with C++.
| Zababa wrote:
| Answered here:
| https://news.ycombinator.com/item?id=28000050
|
| "Despite being way more familiar with C++, I can't imagine
| going from zero to production in 6 months (or even 2 years)
| if we'd tried to do this in C++"
| s3cur3 wrote:
| This post on our dev blog gives some detail on the language
| decision here:
|
| https://developer.x-plane.com/2021/01/have-you-heard-the-
| goo...
| tudelo wrote:
| So, since you seem to have some knowledge here, what would you
| suggest a single engineer trying to make a somewhat scalable
| multiplayer game? Either instanced with <64 players or a more
| larger user count.
|
| I did some experimenting with rolling my own networking layer
| in c++ and it was a lot of work, and tried to do it through
| unity but it seems to require quite a large investment in time
| to understand how networking fits in the overall picture of the
| game. But maybe I'm looking for magic where it does not or can
| not exist :)
| Thaxll wrote:
| The way to go is usually to re-use your game client and strip
| the rendering part.
| andy_ppp wrote:
| Using the right tool for the right job is important, here they
| chose Elixir and it's working well for them. There's no need to
| be this disgruntled with your ill informed opinions about the
| Erlang VM (gc not very good - no it's just optimised for
| different things, latency being one). The Elixir core team have
| recently released nx-elixir which allows much faster maths and
| allows mutability as the calculations happen outside of the
| runtime, so you might want to look at this before you make
| assumptions.
| anonymousDan wrote:
| To claim GC of the JVM as an advantage Vs elixir is farcical,
| the lack of shared state/independent heaps is one of the main
| advantages of Erlang Vs stop-the-world GC with the JVM.
| thibaut_barrere wrote:
| Favorite quote:
|
| "this article explores why they chose Elixir and how a team of
| one developer - without prior language experience - learned the
| language and deployed a well-received multiplayer experience in 6
| months."
| sergiotapia wrote:
| I've been writing Elixir since 2016 and I don't see myself
| switching anytime soon. It's a beautiful language to use, just
| nice and lovely. The functional aspects mean every function has
| no side effect. It's very very liberating.
| thibaut_barrere wrote:
| It is true that _most_ functions do not have side effect and
| that this is very liberating (makes it easier to refactor and
| maintain). Also, most of the time when there is a side effect,
| it is clearly visible (due to using a stateful abstraction or
| library).
|
| So all in all while stating "every function has no side effect"
| is incorrect, I still share your overall feeling.
| dudul wrote:
| > The functional aspects mean every function has no side
| effect.
|
| This is not accurate. Any function can print, can call a DB or
| a web service.
|
| "Purity" (as in referentially transparent) is tangential to
| function programming. It can be achieved with other paradigms.
| That being said, functional programming does encourage it.
| c4wrd wrote:
| Is there any open source implementations of game servers/highly
| concurrent production servers like this for reference (or at
| least blog posts that contain a good amount of code)? I can find
| a lot of toy examples, but I'd love to see some insight onto an
| actual implementation of a product that is open-source as I'm
| learning Elixir.
| s3cur3 wrote:
| The core of the X-Plane server is our RakNet UDP protocol,
| which is open sourced under the MIT license here:
|
| https://github.com/X-Plane/elixir-raknet
|
| It's not a full game server, but the "Usage" section of the
| README provides a sketch of what the rest of the server (the
| part that implements the business logic) looks like.
| z3t4 wrote:
| Elexir might be a nice language, but you could probably also
| implement the server using socat... That said, scaling to tens of
| thousands of concurrent players is like the holy grail,
| especially if you want to shoot stuff and there need to be hit
| detection, and safe from client modifications/hacks.
| Vgoose wrote:
| It used to be around every few weeks I would read something cool
| about Elixir. This past two weeks it's down to every few days.
|
| I think someone's trying to tell me something.
| auraham wrote:
| I am learning Elixir and I'd love to see a tiny game made in
| Elixir to play with and learn from the code. Any examples?
| ravi-delia wrote:
| This is the kind of thing that I love about Elixir. As great as
| Phoenix is, realistically you are going to be equally productive
| in any web framework. Elixir is my favorite language because it
| gives you primitives that perfectly suit your needs for just
| about anything running on a socket, especially once you add in
| Plug.
|
| Throwing together a prototype is instant, but the awesome part is
| how quickly you can turn that prototype into something useful.
| More concurrency? Done. Need to sock away some state somewhere
| you can easily find and deal with it? Easy. It's easy to write,
| easy to reason about, and gives you fantastic composable tools to
| put something great together. It doesn't feel like a codebase so
| much as an incredibly well managed and cohesive OS.
___________________________________________________________________
(page generated 2021-07-29 23:00 UTC)