[HN Gopher] Lichess gets a big upgrade. It doesn't go as planned ___________________________________________________________________ Lichess gets a big upgrade. It doesn't go as planned Author : _fizz_buzz_ Score : 259 points Date : 2022-12-15 17:07 UTC (5 hours ago) (HTM) web link (lichess.org) (TXT) w3m dump (lichess.org) | TulliusCicero wrote: | > Improved type inference | | > We want types, not boilerplate. Sometimes it's best to let the | compiler figure out what things are by itself. | | At first I thought type inference was cool too, but more and more | I don't like it. I prefer being able to see a variable's type | easily, even when skimming over code. Type inference makes code | less readable to me. | Barrin92 wrote: | that's more an issue of editor integration. F# in Vscode for | example overlays what types it has inferred automatically. | TulliusCicero wrote: | And what happens in every other tool that displays source | code? | TylerE wrote: | Who cares? That's the point of an integrated editor... | blt wrote: | yeah, to be fair I only know it through the C++ `auto` keyword, | but I dislike its overuse. It's great for weird templated | iterator types but I prefer to see the concrete type if it's a | reasonable length. | beached_whale wrote: | With C++'s implicit conversions, it makes that less of an | issue though. Like a function returning long and assigning to | an int "just works" and can silently trunctate. With auto, | both are long | jandrese wrote: | Or the question you should always ask: What if the compiler | gets it wrong? Is there a way for you to tell? | MaxBarraclough wrote: | Compiler/runtime bugs are always painful, I don't see that | type inference is a special case there. | | Fortunately such bugs are pretty uncommon. | jandrese wrote: | Is it a bug if the compiler thinks that maybe the value | could grow too large to fit in an integer so it promotes it | to a floating point type. But then you compare it to a | value that is converted out of a string of ASCII digits and | the comparison is off by 1 because 4 != 3.99999999996 and | this causes your code to iterate one additional loop which | doesn't cause an immediate problem except about 1 in every | 256 times it causes a boundary condition where the next | packet will have a corrupt first byte? | | Technically the compiler technically did nothing wrong, but | you are still left with a head scratcher because all of the | logic seems sound and yet it produces occasional garbage. | adgjlsfhk1 wrote: | type inference isn't magic. It would absolutely be a bug | for the compiler to change the type of a variable from an | int to a float because it thinks there will be overflow. | ben-schaaf wrote: | > Technically the compiler technically did nothing wrong | | I don't know a single language where integers are | "promoted" to floats when they get large; certainly no | statically typed language. You could argue JavaScript | does this, but it's neither compiled nor does it actually | have integers. So yes, if a compiler did that it would be | a bug. | duped wrote: | What compiler does that? What you're describing sounds | like a weirdly designed type system, not a compiler that | implements it. | | That said, this _is_ an example of the compiler being | wrong. | | If a value is changed to a data representation with | different comparison semantics, and later compared to a | different value of a different representation, it means | the original optimization (hand wave over the definition | of that) was unsound in the first place. AOT compilers | would classify this as a bug that needs to be fixed | imminently, and JIT compilers that do this kind of thing | use something called dynamic deoptimization to remove the | change once its been invalidated. | treis wrote: | The problem is when you get things like: | | var someLongThing | <application.someHierarchy.someLongAssPackage.someLongThing> = | new | application.someHierarchy.someLongAssPackage.someLongThing() | hinkley wrote: | class class1<T extends Comparable<T> > implements | SmallestLargest<T> T[] values; | | This is a spot where inference doesn't work, but has the same | problem as the relatively tame example you used. | | The solution here is not to eliminate the type signatures. | The solution is not to wait until compile time to figure out | the type of a thing. Nominal typing, as in named types, is | the solution here. If you have a 'thing', which also happens | to be an | application.someHierarchy.someLongAssPackage.someLongThing | implements | application.someHierarchy.someLongAssPackage.someInterface, | anotherLongAssDefinitionForTheReturnTypes but that's an | implementation detail, not a type description. | | It's an Address, or a customer impression, or a similar | product, or a fuel tank. Just fucking call it what it is | instead of pretending like you're working on your PhD thesis. | Your coworkers aren't robots, or academics, they're human | beings who've got real work to do and kissing your | architectural astronaut butt/complementing your farts are not | on the list. Name things like a human. A wise human, but a | human. | rubyist5eva wrote: | My IDE can show me inferred types as an overlay in my editor | (even in languages with dynamic types like Ruby, jetbrains | inference is pretty good when the code is simple), perhaps look | for an extension for your preferred editor? | winter_blue wrote: | If GitHub could dd support for it, then that would be | amazing. | | As it stands, type inference has a tendency to make code | reviews more difficult. | rubyist5eva wrote: | Why Github? I'm assuming you mean their code review | interface, which is so incredibly lacking I don't even | bother with it anymore and I just pull the branch locally | and use my IDE. | TulliusCicero wrote: | Then you need that support in every related place, as the | other comment about GitHub points out. | vkou wrote: | Type inference everywhere can make code less readable, but type | inference when it's clearly the code author's intent that the | type of the variable should not be of much interest to anyone | improves readability. | | A common example of this is a variable that's initialized in | one line, fed into another function in the very next line, and | then never used again. [1] If the function call that produces | it, and the function call that consumes it is named well, the | variable's type is often, but not always superfluous. | | It's like naming for loop variables. Sometimes, you want a | specific, detailed name for them. Other times, you just call it | 'i', because what it is... Is not very important. It's a | counter, counter goes up until we hit the end of a list, don't | think about it too much. | | [1] The two function calls could have been mashed together, but | that would make the code less readable. The full variable type | could have also been included, but its not very relevant to the | intent of the code - which is simply[2] feeding the output of | one function as an input into the other. | | [2] Obviously, when contextual information (like real-world | side effects of creating a particular variable - say, a thread, | or a record...) that complicates the "Create -> Use" case is | relevant, or if the APIs in question have a lot of function | overloading, or poorly-named interfaces, or if non-trivial | casting is going to take place, it may still be beneficial to | explicitly provide the variable type. | vlovich123 wrote: | Type inference w/ overlays is usually enough. The only thing | I'm missing with VSCode rust overlays is they sometimes don't | show the lifetime on an inferred type which makes it a little | more annoying. | dkarl wrote: | Explicit types are useful in some places. Type inference | doesn't mean you can't use them, just that you don't have to | spell them out everywhere even when they aren't useful. | spullara wrote: | I try to avoid using it where the type isn't obvious from the | context. But when it is, I like it a lot. | matsemann wrote: | Type inference locally is great. Doing `val something = | MyClass()` or `val something = someFunc()` . No ambiguity, just | less typing and clutter. At bounds however (function | input/output for instance) I prefer explicit types, less chance | for mistakes and the signature acts as documentation to what | the function does. | 1337shadow wrote: | Static typing removes freedom I am addicted to with duck | typing, however, I only ship TDD code which also serve as | coding documentation. Both are protective but if you're not | writing tests then by all means use static typing. | yetanotherloser wrote: | I'm finding this hard to relate to. Our circumstances and | experiences must be pretty different. But I'm afraid to me | that sentence reads a bit like "I have a lathe so I don't | need a bicycle". You or I might not do bicycle jobs, or | lathe jobs, but it's a bit of a stretch to imply one | renders the other irrelevant. | | I find it interesting that you characterise static typing | as about safety. I think it's more about communication. | (for me this includes "communication to myself when I wrote | this seven years ago and didn't think about it since", | which I appreciate may be a bit of an edge case). Tests are | also a great medium of communication, but a different one - | in which case the metaphor becomes swap a sonnet for a | sonata? I'd like both please, but maybe not always at the | same time :-D | dmitriid wrote: | Neither tests nor types are documentation. | 63 wrote: | One of the things that's made me the most frustrated learning | rust is all the type coercion and inferencing. Sometimes code | will just work when I have the wrong mental model because types | are coerced but then it ends up biting me later when suddenly | everything breaks because it can't coerce anymore and I find | out nothing was the type I thought it was. I wish there was a | way to turn it off while I'm learning just so I can understand | what's really happening. | jgilias wrote: | Nobody forces you to rely on type inference. It's perfectly | valid Rust where you type everything if you so wish. This | might actually be a good way to learn the ins and outs of the | type system. | | EDIT: Also, what do you mean by type 'coercion'? As far as I | know the types in Rust are never silently coerced to | anything. They just are what they are. | seritools wrote: | > As far as I know the types in Rust are never silently | coerced to anything. | | There are some specific coercion sites and kinds: | https://doc.rust-lang.org/reference/type-coercions.html | kibwen wrote: | True, although coercion in Rust is quite limited (it | doesn't even coerce smaller ints into larger ones), and | the coercions that exist are pretty harmless IMO (like | letting you call a function that accepts a `&foo` if all | you have is a `&mut foo`). I suspect they're referring to | something else, as I've never seen coercion highlighted | as a pain point before. | vlovich123 wrote: | They mean inference. Generics get resolved to different | types (and lifetimes) depending on surrounding code. | yetanotherloser wrote: | wow, I never contemplated before whether that kind of | language design could be evil, but you make me think it | might be. Maybe not in the ordinary course of sensible | use, but in some horrible "I didn't think of that". Then | I feel a bit hypocritical because I'm sure I've relied on | a hundred more problematic abstractions without worrying | about it. | kibwen wrote: | Far from evil, it's amazingly useful, and is why things | like `let x: Vec<_> = foo.collect();` work (the return | type of the `collect` method is generic). Rust stops type | inference at function boundaries, so the precise type of | something is never far away if you need it. | efficax wrote: | types are not coerced in rust? you have to implement explict | `From` traits and call `into`. | tomca32 wrote: | I think they meant to say "inferred" | williamscales wrote: | I've never written rust, so maybe this is way off. But could | the code editor display the inferred type right next to the | variable declaration? | c0balt wrote: | Most editors (or plugins for them) that support a LSP, like | rust analyzer[0], can do it. This includes afaik (neo)vim | (via coc.nvim), vscode (via rust-analyzer extension), emacs | and Clion. | | [0]: https://rust-analyzer.github.io/ | Jorengarenar wrote: | >(neo)vim (via coc.nvim) | | I think _vim-lsp_ and _Neovim 's native LSP_ both have it | too | toast0 wrote: | It could, but then you put off grumpy people like me with | old editors that don't do that kind of thing. Plus you need | fancy tools to look at patches. And none of it is going to | work inside tokio's select macro (or other useful macros), | which could be a lot of your code, depending on what you're | writing. | | I'd rather work in a language where nobody has type | information. It's a much more even playing field. ;p | bryanlarsen wrote: | What are you using for an editor? Both emacs and vim can | do that kind of thing. | toast0 wrote: | I use joe. I've been using it for 25 years at this point, | and it works enough, and I'm not interested in changing. | About 15 years ago, I upgraded to a version of joe with | syntax highlighting which was pretty cool, but I'm not | much interested in more than that. I do occasionally poke | around with things in fancy IDEs (it's easier to fiddle | with Arduino projects with an IDE than otherwise, because | nobody writes about how to do the pieces individually... | but maybe I can drive platform.io from the command line | in the future), but I'm much happier with a simple as | dirt editor, and another terminal to run Make in (or | whatever, using Rust for work now, so running cargo | instead of make). | moonchrome wrote: | IMO a decent editor is a must for statically typed | languages - it's such a productivity boost to navigate by | type information, type safe refactoring, etc. | | Main benefit of static typing is enabling tooling to | reason about your code - not using IDEs is throwing a | huge chunk of it away. Historically it was easy to get | projects that would be too large for real-time tooling - | but these days the IDEs got better and you can get 32 or | even 64 GB ram into a workstation trivially - I haven't | seen a scenario like that in years. | | I've also noticed GitHub supports type navigation in some | languages, but yeah for nontrivial reviews I'll usually | do a checkout anyway. | twicetwice wrote: | Yes, I believe rust-analyzer language server + vscode | extension is the setup I have that does this for me. | cauthon wrote: | Yes, highly recommend this setup. Displaying inferred | type + compiler errors has been amazing while learning | Rust. | Kukumber wrote: | The power of open source! | | Congrats to lichess and everyone involved who helped! | d2049 wrote: | > To be able to tell if Scala 3 itself is faster, we would have | to rollback to Scala 2 and try it with the proper JVM tuning. I'm | not willing to do that, sorry! Once you've tried Scala 3, there's | no going back. | | Pardon my ignorance. Why can't you set up a performance test in | both environments and measure the speed? | ARandumGuy wrote: | I think it's a matter of effort. Many performance bottlenecks | only really show up when you scale up to massive levels. While | it's possible to replicate that on a test environment, it takes | time and effort to set that up. While doing that is worth it | for some applications, I can see why the Lichess developers | didn't want to bother. | hocuspocus wrote: | Time, effort and spare resources that Lichess had better | allocate somewhere else. | | If Thibault enjoys Scala 3 and feels more productive, while | the overall performance on the same hardware is approximately | equal, it's already a win for the project. | agilob wrote: | Setting up reliable and reproducible performance environments | is as much work as running good quality prod systems. It | requires monitoring thing rarely available to monitor, | extending prod metrics, creating snapshots, running repeatable | tests on different scales. This isn't work for a single person. | I'm in a team of 7 working to acquire next customer who will be | on average 5x bigger than peaks of our current biggest | customer. We've been designing, configuring and creating | metrics on EKS since September, alongside training devs and | developing performance engineering framework in k6. Lichess | doesn't need that much effort and accuracy, but it's still lots | of work. Did you know Fargate will schedule you Intel Skylake | or Broadwell and there's even 30% difference in performance | between them on certain tasks? | typingmonkey wrote: | That would only confirm that the new lichess version is faster, | but not that scala3 is faster then scala2. | cpleppert wrote: | The JVM only runs out of codecache (usually) after a period of | time running with a production load. | toast0 wrote: | > Pardon my ignorance. Why can't you set up a performance test | in both environments and measure the speed? | | Everyone has a performance test environment, not everyone has | one that's separate from production. In this case, generating | realistic test patterns is probably very difficult, so you get | one environment for everything. | tiffanyh wrote: | 64 comments a week ago on similar topic ("Lichess on scala3 - | help needed") | | https://news.ycombinator.com/item?id=33865932 | hn_acc_2 wrote: | Looks like the top comment was spot on. Were HN users the | "Avengers" mentioned in OP? | jakub_g wrote: | For those short on time: "It doesn't go as planned" is explained | in the middle of the article: | | > When everything compiled, I shipped it. And to everyone's | surprise, apart from a few bugs I had created while rewriting | thousands of lines of code... it worked. It just did. No | explosions, no obscure bugs, no memory leak, no performance | degradation. That was rather unexpected. | | ...followed by | | > Then we saw the JVM CPU usage rise to alarming heights, with | unusual patterns. And no obvious culprit in the thread dumps... | | > it was just the JVM that needed some tuning. | | Details: https://lichess.org/@/thibault/blog/lichess-on- | scala3-help-n... | bravetraveler wrote: | 'if it builds it ships' in action, love to see it | lamontcg wrote: | > "It doesn't go as planned" | | bit of an editorialized clickbait title really. | zython wrote: | INNOCENT dev migrates to Scala 3, what happened next will | SHOCK you. | lamontcg wrote: | Spend a week of running the JVM in production without | tuning it challenge [IMPOSSIBLE] | smcin wrote: | Yes it's clickbait, so I didn't upvote it; an accurate title | would have been "Lessons learned upgrading Lichess from Scala | 2 -> 3 and the need to tune the JVM". A drama-free title like | that would have been fine, and I would have upvoted. | jonathanyc wrote: | I agree. Clickbait titles are not universalizable. Kant | would not be pleased. | kzrdude wrote: | The actual title seems to be Lichess & Scala 3 and the | subhead is "Lichess gets a big upgrade. It doesn't go as | planned." | AuthorizedCust wrote: | JVM tuning is frustrating. The need for that is generally not | an issue on Microsoft .NET. | warent wrote: | moffkalast wrote: | As an anecdotal case, a company a friend of mine works at has | a backend workforce of primarily Java-only devs, so that's | the only thing they can really maintain. There's a lot of | sunk cost and inertia on the enterprise side when it comes to | Java. | | But yeah for starting something new, with an open language | choice? No way anyone sane would go for it these days. | Especially after Oracle basically closed sourced it, making | JRE distribution illegal. | warent wrote: | I take the copious downvotes as confirmation! | phoronixrly wrote: | As opposed to what? | [deleted] | vlovich123 wrote: | Machine code. Assembly if you're weak willed and need some | hand holding. | [deleted] | danielheath wrote: | Written with a magnet, and a steady hand. Keyboards are a | crutch. | hinkley wrote: | Shit the ROMs on the Apollo mission were hand-woven wire | looms. 4K wires weaving between and around magnets. Why | hold a magnet when you can use a marlin spike? | poulpy123 wrote: | I personally use cosmic focused on a disk by the | turbulences created by a butterfly to program, but it's | not a competition | toast0 wrote: | Because nobody has ever expanded their machine code | beyond the L1 cache size and experienced a huge | difference in performance. | renewiltord wrote: | Pathetic. If you can't fit your code into L1, are you | even a real programmer? No one has time for your Electron | shit. Get real, bruh, what's your deadlift? | nurettin wrote: | Back in my day we didn't have L1, so our code just ran on | 16 registers. | tux3 wrote: | Of course _real programmers_ write forged microcode to | define just the machine code instructions they need, and | they flash the ucode update live in prod to deploy their | app. | | But if you're weak willed and need some hand holding, the | official ISA's machine code is tolerated too. | [deleted] | toast0 wrote: | I mean, I'm not a JVM fan, but this sort of thing happens in | lots of languages, and even operating systems. Upgrade from | version X to X + 1, especially with lots of changes in | everything else, and you're bound to hit different limits or | explore the interface with limits you were hitting and didn't | realize. | | The other option is to never update anything; which is valid, | but painful. | amackera wrote: | JVM is easily one of the best runtimes out there by any | objective measure. | | (JVM != Java, remember) | smeagull wrote: | Hubris. I've seen shops build servless applications with Java | because "we have java developers". | | I left that contract very quickly. | winrid wrote: | Is this really an issue with graalvm? | WhiteBlueSkies wrote: | I wonder how node.js would have performed at that scale. | jbm wrote: | I truly wish I could get Opaque Types (Newtypes?) in Typescript. | Great writeup! | stareatgoats wrote: | Got sucked into reading the article because of the "didn't go as | planned" but it left my thirst for failure stories unquenched. | Should have known. | | That aside, that thing with opaque types in Scala sounds | interesting - but how would the compiler know that the parameter | was a UserId and not just a random string? | lmm wrote: | You have to explicitly call something like UserId(...), just | like with any other type. (You can actually do this technique | in C by using a 1-element struct, although it's a lot more | cumbersome than Scala since the type system is so much more | limited). | exabrial wrote: | -1 for Type inference and `var`. It literally save you no time, | increases cognitive load, and is an exposure point for future | bugs. | | Just use Explicit types. They aren't hard, time consuming, or | bad. They are your friends. | syastrov wrote: | > That's where things got a bit hairy. Lila is built on Play | Framework which is not yet ported to Scala 3. | | > So I forked it and butchered it to remove everything we don't | need - which is actually most of the framework. | | I guess there is hope that Play framework itself will be migrated | to Scala 3 and that the dependency on the fork can be removed, | but this is taking on a risk - what if there are security updates | to the upstream in the mean time? | 12345hn6789 wrote: | Isn't light bend more or less falling apart? I thought they | announced they would no longer publish changes to play | tasuki wrote: | I think the plan might be to get rid of the play framework | altogether and rather use a couple small, independent libraries | to achieve the same. | | The likelihood of getting security fixes in the future is about | the same as getting new security holes created by whatever | updates. Butchering away everything they didn't need certainly | didn't harm security. | eppp wrote: | But what was the JVM tuning? Thats the most interesting part! | lukhas wrote: | Currently it's "-Xms30g -Xmx30g -XX:+UseG1GC | -XX:+PrintCodeCache -XX:ProfiledCodeHeapSize=500m | -XX:NonProfiledCodeHeapSize=500m -XX:NonNMethodCodeHeapSize=24m | -XX:ReservedCodeCacheSize=1024m -XX:InitialCodeCacheSize=1024m | -XX:ParallelGCThreads=24" | | Everything after G1GC was suggested by various helpful experts | on HN, Discord, by email and other media. | csunbird wrote: | Are you running the whole lichess on one machine, or is this | one shard only? 30 GB RAM for one instance of application | seems very high. (sorry did not have time to read the whole | article yet) | Buttons840 wrote: | Is there any other runtime that needs as much tuning as the | JVM? | | It seems like a failure of the JVM that someone capable of | porting thousands of lines of code with little trouble then | fails to tune the JVM. It's like, "we wrote all the code, now | the hard part starts, tuning the JVM". | lmm wrote: | I took just the opposite lesson from this: the JVM offers | enough turing knobs that in the rare, extreme cases where | the defaults don't work for you (I suspect few single- | server workloads in the world have the combination of | complexity and request rate that Lichess does), there are | still ways to do something about it. If he'd been using, | say, Go, he'd probably have had to give up and roll back, | or patch the runtime code. | hellcow wrote: | I've been using Go for 7'ish years across a large | codebase, comprising multiple services interacting with | millions of people. | | Every release Go gets better and requires 0 changes to | the code. I have never needed to fine-tune the GC. I have | never needed to spend a month rewriting my code to work | with Go 2.0. I have never been nervous to update to a new | Go version. I write the code once, and it runs great in | prod for years. I love and value these things. I also | suspect that Lichess's use case would perform extremely | well in Go out-of-the-box, since it's just a web app. | | I certainly hope the JVM team would like Lichess and | other web apps to run well without needing arcane | configuration knowledge gathered over years of experience | and battle scars in production. | kinjba11 wrote: | > or patch the runtime code | | A huge spike in CPU usage would be treated as a | regression in other language runtimes and would likely be | addressed. The fact that this is solvable with advanced | JVM knobs is both good and bad. Good in the way you say. | Bad because the complexity of maintaining all those knobs | makes it difficult if not impossible to improve the | runtime defaults, and every runtime problem becomes a | tuning problem. | thomashabets2 wrote: | Java is a language built up of failed experiments in | language design. | | My full rant about it is | https://blog.habets.se/2022/08/Java-a-fractal-of-bad- | experim... | | It's partially my subjective opinion, but it seems that | almost every single language design decision that Java made | was, in retrospect, a bad one. | | Not that I could have done better. It's just that none of | it panned out. | jdlshore wrote: | Eh... your hate is blinding you. | | Java was the first mainstream garbage-collected language. | Not the first GC language, but the first one to get | serious traction. It started the post-C++ era. | | That was a pretty bold design decision for the time, and | one that worked out. The VM was another big one, and it | also worked out. | adgjlsfhk1 wrote: | That's not quite fair. They made 1 good design decision | (killing pointers). | esaym wrote: | Perhaps it is different now, but I've always hated how to | figure out how much memory a java app needs. You can | certainly give it 30GB of ram and it will happily use it | all up and then start making garbage collection calls. But | does it _really_ need all that ram? I think the best | practice of the time was to continually lower your max heap | amounts until you started getting allocation errors, then | bump your number up by 20%-50% (or something like that). | stardenburden wrote: | The linked blog post has some details: | https://lichess.org/@/thibault/blog/lichess-on-scala3-help-n... | csense wrote: | Also see previous HN discussion (top comment is the | solution): | | https://news.ycombinator.com/item?id=33865932 | cpleppert wrote: | Yeah, literally the first thing you do when running into | JVM issues. It was my second initial suggestion in Discord | as well. | xyst wrote: | Would be interesting to see if this application would benefit | to switching the JVM from openjdk to Azuls "high performance" | JVM. | | Rather than use a big bang deployment. Setup an A/B test, and | gather a sampling of data. | | I haven't been able to truly test azul's claims yet. Java | apps that I manage/deploy were on a small scale in regards to | requests/sec and I personally didn't see much difference. | philipwhiuk wrote: | > Significant indentation and optional braces | | And this is why it's annoying to use Scala. They just decided to | rewrite the syntax of the language | tasuki wrote: | In Scala, eg 2.13 is a major version. Scala 3 is like a super- | major version and still it's backwards compatible with 2.13: | the new syntax is fully optional. | | Which languages don't make any changes to syntax over major | versions? | spiderice wrote: | I'm not against syntax changes, but significant indentation | seems like an insane choice to make. That's been a thorn in | the side of Python for decades. Even if some people do like | the look of it, it doesn't seem worth the tradeoffs. | civopsec wrote: | It's not a problem for Haskell and Scala is closer to | Haskell. | hocuspocus wrote: | Like many seasoned Scala developers I was skeptical at first | but it's actually pretty enjoyable despite a few | inconsistencies. Jumping between Scala 2 and 3 codebases isn't | much of a problem. Granted, it makes the life of people working | on tooling significantly more complicated but I'm not one of | them. | | The decision was directly motivated by Odersky's experience | teaching Scala to students for 15+ years, it wasn't made for | the sake of fueling drama in the community. ___________________________________________________________________ (page generated 2022-12-15 23:00 UTC)