[HN Gopher] The hunt for a cluster-killer Erlang bug ___________________________________________________________________ The hunt for a cluster-killer Erlang bug Author : eproxus Score : 53 points Date : 2022-06-14 21:05 UTC (1 hours ago) (HTM) web link (engineering.klarna.com) (TXT) w3m dump (engineering.klarna.com) | tiffanyh wrote: | Fantastic detailed write up. Wish there was more of these style | of articles on HN. | banashark wrote: | Very interesting writeup. Distributed systems problem solving is | always a very interesting process. It very frequently uncovers | areas ripe for instrumentation improvement. | | The Erlang Ecosystem seemed very mature and iterated. It almost | seemed like the "rails of distributed system" with things like | Mnesia. | | The one downside to that seemed to be that while I was working on | grokking the system, the limits and observability of some of | these built-in solutions was not so clear. What happens when a | mailbox exceeds it's limit? Does the data get dropped? Or, how to | recover from a network segmentation? These proved somewhat | challenging to reproduce and troubleshoot (as distributed | problems can be). | | There are answers for all of these interesting scenarios, but in | some cases it almost would have been simpler to use an external | technology (redis/etc) with established | scalability/observability. | | I do say this knowing that there was plenty I did not get time to | learn about the ecosystem in the depth that I desired, but was | curious how more experienced Erlang engineers viewed the problem. | davidw wrote: | > So our initial 1 GB binary data pretty printed as a string will | take about 1 GB x 3.57 characters/byte x 2 words/character x 8 | bytes/word = 57.12 GB memory. | | Yeah, I saw that one in an Erlang system too. It was pretty ugly. ___________________________________________________________________ (page generated 2022-06-14 23:00 UTC)