[HN Gopher] How we built an auto-scalable Minecraft server for 1...
___________________________________________________________________
How we built an auto-scalable Minecraft server for 1000+ players
Author : hopfog
Score : 96 points
Date : 2021-09-03 07:06 UTC (1 days ago)
(HTM) web link (www.worldql.com)
(TXT) w3m dump (www.worldql.com)
| Jaxkr wrote:
| Hi, I made this! Was gonna wait until it was downloadable and the
| documentation was more complete before posting it on HN. Thanks
| for sharing it though.
|
| The Minecraft setup will be available and useable by anyone
| Wednesday of next week. More documentation and a roadmap will
| follow shortly after.
| gravypod wrote:
| This is very impressive work! Do you plan to make WorldQL
| highly available? It seems like if it died you'd loose the
| server.
| Jaxkr wrote:
| Thank you for your kind words!
|
| WorldQL uses Postgres under-the-hood to store permanent
| information. I was inspired by companies like TimescaleDB
| which build new functionality on top of Postgres's rock-solid
| base.
|
| Any high-availability solution for Postgres will also be
| available for WQL.
| gravypod wrote:
| Do the clients of WorldQL (the workers) know how to
| gracefully fail over when WorldQL fails?
| tehbeard wrote:
| Curious how you plan to handle Redstone and hostile mobs.
| Jaxkr wrote:
| Thanks for your question! Redstone is still WIP but will
| follow:
|
| 1. An optimistic execution strategy (the servers can run
| their own redstone)
|
| 2. A locking system allowing only one server to have redstone
| current in a given chunk at once.
|
| 3. A rollback-based system to repair race conditions caused
| by (1)'s optimism.
|
| Hostile mobs are ALSO still WIP and:
|
| 1. Are only synced if two players are near each-other on two
| different servers.
|
| 2. Have their aggression entirely managed by a WorldQL script
| which sends messages to the appropriate servers instructing
| them to call LivingEntity.setTarget on the correct player.
|
| Was hoping nail those both down before I shared it here, so
| please forgive me! Thanks for your interest and stay tuned.
| Fiahil wrote:
| Is there a specific reason why Minecraft hasn't been "adapted"
| from the original game to a robust design that could scale to
| larger worlds and player populations before your project ?
| forrestthewoods wrote:
| Pretty much no video game engine scales with cores. There are
| too many dependent systems. There's no game engine that I'm
| aware of that effectively uses more than a couple of cores
| for gameplay simulation.
|
| Distributed physics is a well understood problem with no
| "solution". Just different trade-offs. This solution is cool,
| but it's not novel or anything. Minecraft is particularly
| challenging because the entire world is highly mutable.
|
| There are no general solutions to any of this. Just a bunch
| of custom one-offs. SpatialOS is trying. My opinion on it is
| extremely, extremely negative. And I'll leave it at that.
|
| Unity and Unreal both have primarily single-threaded
| gameplay. Writing multi-threaded gameplay code is
| extraordinarily difficult. Unity has been working on their
| DOTS/ECS system for years, but it does not appear to be close
| to ready for the mainstream.
|
| So I think the short answer is "it's really really hard.
| Like, radically harder than you are imagining. Even if you
| think it's hard. The value of that work is likely not worth
| the cost. If most players WANT to play on small servers with
| close friends then the mountain range of work to effectively
| support a 1000 player server is not worth it".
| 8note wrote:
| Couldn't Minecraft chunks run fairly independently, with
| some message passing when something needs to move between
| chunks?
| cdev_gl wrote:
| I rent a small Linode to run a minecraft server for my extended
| family. Sometimes it struggles to keep up with just a half
| dozen of us, and when I tried swapping from a spigot based
| system to a forge one so the kids could have mods it was
| basically unplayable. I can't imagine scaling out to thousands
| of users, this was some amazing work.
|
| While developing this did you come up with any best practice
| recommendations for individual nodes?
| Jaxkr wrote:
| Your single-core performance is the most important thing for
| a Minecraft server, especially for Forge mods.
|
| For vanilla, I'd try Paper (a high performance spigot fork)
| and see if you still have problems. If you're lagging with 6
| people while running Paper, you simply need a better CPU.
| ianhawes wrote:
| What were the specs of the server and which Spigot fork did
| you run?
|
| A month ago I ran a Paper instance from a Digital Ocean
| instance with about 4 GB of RAM and OpenJ9 JVM and it never
| dipped below 20 TPS even with a larger render distance. This
| was on Vanilla 1.17.
| e12e wrote:
| It might be too pricy, but if you go to about 25usd/month,
| you can get a dedicated server with an i7 from hetzner server
| auction. These cheap boxes don't have ecc ram etc - but
| should work well for things like this.
|
| https://www.hetzner.com/sb?country=us
| fire wrote:
| Do you have any articles or information available that explain
| why a spacial database is useful for and helps solve problems
| like this?
|
| This is super interesting and I'm excited that someone has
| finally made real progress on distributed game world software
| denvaar wrote:
| Love the font on this site. Looks like you can find it here:
| https://brailleinstitute.org/freefont
| ZoomerCretin wrote:
| I'm curious to know how or if this solved any of the issues
| inherent with multithreading. Moving the threads to new processes
| can't fix race conditions.
| Jaxkr wrote:
| It utilizes a rollback-based system to mitigate races between
| Minecraft servers.
|
| I'm writing up some formal documentation on it now, I really
| wanted to have that done before this project was exposed to the
| scrutinizing Hacker News community, but I can't control what
| people share! :)
|
| Stay tuned.
| winrid wrote:
| Using rollback is surprisingly popular in games. For example,
| CSGO servers will rollback if a client fires a shot at a
| player ducking around a corner, and on the shooter's screen
| the shot lands, but on the other players screen they had
| dodged the bullet - the server will "rewind" and say the shot
| hit.
| crazy_horse wrote:
| How did MMOs like Asheron's Call and Wow handle this?
| mysterydip wrote:
| For WoW, at a very high level, instancing and sharding. Players
| are spread across realms.
|
| The zones are so large that by nature players will be spread
| out leading to less interactions. Raids are instanced to just
| your party. In areas with natural congestion (such as auction
| houses), things could lag at times.
|
| While combat and movement is realtime, it's mostly waiting for
| timers, so the latency and bandwidth requirements are reduced
| compared to a first person shooter for example.
| jonathanlydall wrote:
| With WoW they would obviously run each continent as an
| independent "instance", but for Wrath, they clearly had
| multiple servers handling Northrend. There were places where
| mobs would not path over which was where the two "instances"
| would overlap.
|
| WoW later introduced a kind of dynamic overlay of the same
| open world area from two realms where it was not very
| populated in the area so that players would be more likely to
| actually see other players.
|
| There was also a feature where you could party up with a
| friend on a different realm and you would transparently be
| playing on their server in the open world (with restrictions
| like not being able to trade with them).
|
| I haven't played WoW since Mysts, so I don't know how it's
| changed since, but WoW was definitely more complicated than
| just sharding and instancing.
| mysterydip wrote:
| You're correct, I was referring to vanilla WoW which is all
| I played. It has definitely evolved since then, thanks for
| the info!
| ScaleneTriangle wrote:
| If you look at the developer blogs for EVE Online, you'll have
| endless reading about massive scale multiplayer servers.
| Possibly the most technically impressive massively multiplayer
| experience
| Thaxll wrote:
| EvE is one of the worst architecture for an MMO, there is
| almost nothing to do and yet they need to slow down the
| simulation loop when there are too many players. I mean when
| you gameserver is based on Python what else can you do?
| bruce343434 wrote:
| They also call servers not being able to keep up and thus
| the gameloop slowing down "tidi" (time dilation). Like
| yeah, that's just what happens when a gameserver can't keep
| up.
| 63 wrote:
| Impressive work! There's a lot of potential here, but probably
| within a pretty limited market. Outside of 2b2t and maybe
| Hypixel, is there really anyone who wants 1000 players on the
| same world? Is the small market the only reason why a solution
| like this hasn't been created before, or are there bigger
| complications that aren't listed or haven't come up yet, or maybe
| the technology wasn't available? Surely someone like Hausemaster
| who stands to gain substantial amounts of money from this has
| looked into it before, right?
| chews wrote:
| This is nice magical software. Well done.
| Jaxkr wrote:
| Thank you! It's very early, but I hope it'll be useful.
| jonathanlydall wrote:
| Interesting timing.
|
| I play with a few friends on a modded 1.7.10 server and we've
| decided to restart with 1.16 due to the lag having become
| untenable.
|
| We run it on an i7 machine at my house dedicated to it, so it's
| not a hardware issue.
|
| Like clockwork the server would freeze for about 5s about every
| 30s. Using opus our best guess is that it's unloading chunks and
| the GC is happening.
|
| We're going to run 1.16 now which I hope has some performance
| enhancements and so that we can use more modern Java 16 runtime
| with its nicer GC systems.
|
| I'm also hoping that Minecraft has at least since moved chunk
| generation off the main thread since there was no good reason for
| world exploration slowing down things like it did.
|
| I also built a JavaScript redstone simulator website and I'm
| curious how you will handle that and other block updates at
| server boundaries.
| saulrh wrote:
| Is there any chance of this supporting modded servers?
| Performance is universally the limiting factor on modded
| minecraft; if you're not careful you can start having performance
| problems only thirty hours into a single-player world, and I've
| never seen a modded server support more than three or four
| players without severe lag setting in quickly.
| Jaxkr wrote:
| I've gotten this question a lot. At the moment it's not a
| priority because it's frankly too challenging right now.
|
| However, once the documentation is complete, someone could
| hypothetically implement it using
| https://github.com/magmafoundation/Magma-1.16.x
| blackcat333 wrote:
| amazing work. i have been designing my own back end for an mmo in
| my free time off work. this will help me alot. much thanks for
| this.
| mintplant wrote:
| Just to clarify, WorldQL is hosted-only software, right? No on-
| premise? If I tie my game to WorldQL, do I have any options if
| WorldQL goes under (other than re-engineering everything)? Not to
| diminish the impressive work on display here - I've just been
| wondering this since I saw your r/gamedev post.
| Jaxkr wrote:
| It'll be self-hostable too. I need to be clear about that on
| the homepage. Thanks for your question.
| [deleted]
| binkHN wrote:
| FWIW: WorldQL is free for your first $50k in gross revenue.
___________________________________________________________________
(page generated 2021-09-04 23:00 UTC)