[HN Gopher] NilAway: Practical nil panic detection for Go
___________________________________________________________________
NilAway: Practical nil panic detection for Go
Author : advanderveer
Score : 150 points
Date : 2023-11-17 06:59 UTC (1 days ago)
(HTM) web link (www.uber.com)
(TXT) w3m dump (www.uber.com)
| pluto_modadic wrote:
| cool... what does this mean the best linter / correctness
| checking is at the moment?
|
| I have some code that eventually core dumps and honestly I don't
| know what I'm doing wrong, and neither do any golang tools I've
| tried :(
|
| maaaaaybe there's something that'll check that your code never
| closes a channel or always blocks after a specific order of
| events happens...
| mseepgood wrote:
| I don't think a pure Go program can core dump, unless you use
| Cgo (wrongly) or unsafe. It can only panic.
| yencabulator wrote:
| Races between goroutines can corrupt memory. E.g. manipulate
| a map from two goroutines and you can wreck its internal
| state.
| mutatio wrote:
| Can this actually manifest? Even without the -race flag I
| think maps are a special case which will panic with a
| concurrent mutation error if access isn't synchronized.
| yencabulator wrote:
| Another example: thread A toggles an interface variable
| between two types, thread B calls a method on it. You can
| get the method of type X called with a receiver of type
| Y.
| tgv wrote:
| I've had that, and it did panic.
| masklinn wrote:
| > Can this actually manifest?
|
| Yes. Per rsc (https://research.swtch.com/gorace)
|
| > In the current Go implementations, though, there are
| two ways to break through these safety mechanisms. The
| first and more direct way is to use package unsafe,
| specifically unsafe.Pointer. The second, less direct way
| is to use a data race in a multithreaded program.
|
| That races undermine memory safety in go has been used in
| CTFs: https://github.com/netanel01/ctf-
| writeups/blob/master/google...
|
| These are not idle fancies, there are lots of ways to
| unwittingly get data races in go:
| https://www.uber.com/blog/data-race-patterns-in-go.
| tialaramex wrote:
| It's interesting that the 2010 article you linked
| suggests they might consider improving this but nope, Go
| 1.0 and the Go people use today just basically takes the
| same attitude as C and C++ albeit with a small nuance.
|
| In C and C++ SC/DRF (Sequentially Consistent if Data Race
| Free) turns into "All data races are Undefined Behaviour,
| game over, you lose". In Go SC/DRF turns into "All data
| races _on complex types_ are Undefined Behaviour, game
| over, you lose ". If you race e.g. a simple integer
| counter, it's _damaged_ and you ought not to use it
| because it might be anything now, but Go isn 't reduced
| to Undefined Behaviour immediately for this seemingly
| trivial mishap (whereas C and C++ are)
| yencabulator wrote:
| Go doesn't go out of its way to make weird things happen
| on UB like C compilers these days tend to, but once you
| corrupt the map data structure, weird things can happen.
| Trying to contain that explosion isn't necessarily
| "better", as it would make maps slower / take up more
| memory / etc.
| vore wrote:
| Kind of ironic the raison d'etre of Go is a memory safe
| language for concurrent programming but you can easily
| footgun yourself into doing something memory unsafe using
| concurrency...
| 65a wrote:
| Go generaly doesn't used shared memory and concurrency,
| or at least it's been considered an anti-pattern:
| https://go.dev/blog/codelab-share
| vore wrote:
| Yes, but at the same time shared memory concurrency is
| not considered an unsafe usage of Go either...
| adonovan wrote:
| There are ways a Go program can fatal: by running out of
| heap, or stack, by corrupting variables by racing writes, by
| deadlocking, by misuse of reflect or unsafe, and so on.
| insanitybit wrote:
| > Nil panics are found to be an especially pervasive form of
| runtime errors in Go programs. Uber's Go monorepo is no exception
| to this, and has witnessed several runtime errors in production
| because of nil panics, with effects ranging from incorrect
| program behavior to app outages, affecting Uber customers.
|
| Insane that Go had decades of programming mistakes to learn from
| but it chose this path.
|
| Anyway, at least Uber is out there putting out solid bandaids.
| Their equivalent for Java is definitely a must-have for any
| project.
| MichaelNolan wrote:
| The go version NilAway isn't as good as the java version
| NullAway yet. But the team working on it is very responsive and
| eager to improve.
|
| For java projects I think NullAway has gotten so good that it
| really takes the steam out of the Kotlin proponents. Hopefully
| NilAway will get there too.
| tuetuopay wrote:
| > Insane that Go had decades of programming mistakes to learn
| from but it chose this path.
|
| Yup, every time I write some Go I feel like it's been made in a
| vaccum, ignoring decades of programming language. null/nil is a
| solved problem by languages with sum types like haskell and
| rust, or with quasi-sums like zig. It always feels like a
| regression when switching from rust to go.
|
| Kudos to Uber for the tool, it looks amazing!
| IshKebab wrote:
| Dart made the same nullable mistake but actually managed to
| fix it, which is quite impressive.
|
| Go is just obstinately living in the 90s. I guess that's not
| really a surprise. It's pretty much C but with great tooling.
| yoyojojofosho wrote:
| Dart has a great write up on how they fixed the null
| problem by adding non-nullable types:
| https://dart.dev/null-safety/understanding-null-safety
| masklinn wrote:
| Several language have "fixed the null problem" after the
| fact, though usually as an opt-in e.g. typescript, C#.
|
| The problem with Go (for this specific issue, there are
| lots of problems with Go) is that they have wedded
| themselves extremely strongly to zero-defaulting, it's
| absolutely ubiquitous and considered a virtue.
|
| But without null you can't 0-init a pointer, so it's
| incompatible with null-safety.
|
| I think C# pretty much left the idea of everything having
| a default behind when they decided to fix nulls. Though
| obviously the better alternative is to have opt-in
| defaulting instead.
| pcwalton wrote:
| This is a good point. Null pointers are the billion-
| dollar mistake, but the real billion-dollar mistake is
| having "zero values" in your language. In addition to the
| problems with null pointers, zero values make loose
| constructor semantics like C# and Java tempting, where
| objects can exist in a not-fully-initialized state,
| leading to lots of room for confusing bugs. Without zero
| values to fall back on as a crutch, the language design
| is forced to tighten that up so that objects are either
| completely initialized or completely uninitialized, like
| in ML or Rust+, which is a much cleaner semantics. (The
| funny thing is, Go has the tools to get rid of zero
| values by virtue of not having constructors, but it chose
| not to use them for that!)
|
| + Strictly speaking, objects can be partially initialized
| and partially uninitialized in Rust, but this is harmless
| as the borrow checker statically ensures that
| uninitialized fields of objects are never accessed.
| masklinn wrote:
| Yeah technically you can always default init fields to
| None, but then you have to type everything as an Option
| and the ergonomics are so bad you're really incentivised
| not to do that. In that case you bundle that in a
| transient structure (a builder) and you get rid of it
| afterwards.
| usrbinbash wrote:
| > ignoring decades of programming language
|
| True, and because of this, the language can be learned over a
| weekend or during onboarding, new hires can rapidly digest
| codebases and be productive for the company, code is
| straightforward and easy to read, libraries can be quickly
| forked and adapted to suit project needs, and working in
| large teams on the same project is a lot easier than in many
| other languages, the compiler is blazing fast, and it's
| concurrency model is probably the most convenient I have ever
| seen.
|
| Or to put this in less words: Go trades "being-modern" for
| amazing productivity.
|
| > It always feels like a regression when switching from rust
| to go.
|
| It really does, and that's what I love about Go. Don't get me
| wrong I like Rust. I like what it tries to do. But I also
| love the simplicity, and sheer _productiveness_ of Go. If I
| have to deal with the odd nil-based error here and there, I
| consider that a small price to pay.
|
| And judging by the absolute success Go has (measured by
| contributions to Github), many many many many many developers
| agree with me on this.
| insanitybit wrote:
| > And judging by the absolute success Go has (measured by
| contributions to Github), many many many many many
| developers agree with me on this.
|
| Yeah, I truly hate this field
| nicoburns wrote:
| It could have done both though. It could have explicitly
| nullable types like Kotlin/C#. Or sum types like
| zig/rust/Swift. That wouldn't make the language more
| complex to learn.
| usrbinbash wrote:
| By definition, every bit added to a language makes it
| more complex to learn.
|
| Sure, it could be done. Lots of things could be done to
| Go. The people who invented it are among the most
| brilliant computer scientists alive. It's a pretty sure
| bet that they know about, and in great detail, every
| single thing people complain Go doesn't have.
|
| So every thing that is "missing" from Go isn't in it for
| a reason.
|
| "Perfection is achieved, not when there is nothing more
| to add, but when there is nothing left to take away." --
| Antoine de Saint-Exupery, Airman's Odyssey
| nicoburns wrote:
| Not at all. Otherwise brainfuck would be the simplest
| language to learn. How do you currently represent a type
| that is A or B in Go? You have to use an interface.
| That's much more complex than using a sum type would be.
| usrbinbash wrote:
| Well, Brainfuck is simple to learn. The entire
| specification fits comfortably on a single page. Simple
| to learn doesn't automatically imply simple to use for
| any given purpose. The same is true for Go.
|
| > You have to use an interface. That's much more complex
| than using a sum type would be.
|
| More complex how and by what metric?
| tialaramex wrote:
| It's definitely not as simple as "More features = harder
| to learn".
|
| Removing footguns (nulls are a footguns, race-able
| concurrent APIs are a footgun) can make it _easier_ to
| learn even though this may introduce new features (in
| this case sum types) to solve the problem.
| pjmlp wrote:
| It is not like Oberon, Plan 9, Inferno and Limbo were
| such a huge commercial successes.
|
| Had those brilliant computer scientists not been employed
| at Google, it would have been another Oberon or Limbo.
| richbell wrote:
| > True, and because of this...
|
| This is a false dichotomy. One does not imply the other.
|
| Go is also not a simple language. It is deceptively
| difficult with _many_ footguns that could have easily been
| avoided had it not ignored decades of basic programming
| language design.
|
| Many things also aren't straightforward or intuitive. For
| instance, this great list of issues for beginners:
| http://golang50shad.es/
| usrbinbash wrote:
| > This is a false dichotomy. One does not imply the
| other.
|
| No it isn't, and yes it does. By definition, the more
| features I add to something, the more complex it becomes.
| So yes, Go achieves it's simplicity precisely by leaving
| out features.
|
| > this great list of issues
|
| I just picked three examples at random:
|
| "Sending to an Unbuffered Channel Returns As Soon As the
| Target Receiver Is Ready"
|
| "Send and receive operations on a nil channel block
| forver."
|
| "Many languages have increment and decrement operators.
| Unlike other languages, Go doesn't support the prefix
| version of the operations."
|
| All of these are behavior and operators that are
| documented in the language spec. So how is any of these a
| "footgun"?
| pigeonhole123 wrote:
| Not only that but those behaviors are patently not
| footguns or unreasonable in any way
| richbell wrote:
| > No it isn't, and yes it does. By definition, the more
| features I add to something, the more complex it becomes.
| So yes, Go achieves it's simplicity precisely by leaving
| out features.
|
| More complex for whom? Not having generics made the
| compiler simple, but having to copy and paste and
| maintain identical implementations of a function (or use
| interface) adds more complexity for users.
|
| Similarly, adding a better default HTTP client arguably
| makes Go more complex, but the "simple" approach results
| in lots of complexity and frustration for users.
|
| > All of these are behavior and operators that are
| documented in the language spec. So how is any of these a
| "footgun"?
|
| Perhaps I could have been clearer. I didn't mean that the
| entire list was of footguns, just that there are lots of
| confusing and unintuitive things beginners need to learn.
|
| Some actual footguns off the top of my head:
|
| - using Defer in a loop
|
| - having to redeclare variables in a loop
|
| - having to manually close the body of a http response
| even if you don't need it
| adtac wrote:
| I'm sorry but nearly all of them are along the lines of
| "I came from language X and in X we did it this way, but
| Go's syntax is different". That's not a footgun.
|
| You know what's a footgun? Uncaught exceptions popping up
| in places far away from where they were created at which
| point you have very little context to deal with it
| robustly. Use after frees. FactoryFactoryFactories.
| richbell wrote:
| > I'm sorry but nearly all of them are along the lines of
| "I came from language X and in X we did it this way, but
| Go's syntax is different". That's not a footgun.
|
| You're right, I meant to link that in reference to how Go
| can be difficult to learn despite how it simple it seems.
| Not sure how I a sentence.
|
| The overview of that site explains its purpose/necessity
| quite well. Some things are footguns, many are just
| confusing time-wasters. Nevertheless, they are
| frustrating and hamper the learning process.
| adtac wrote:
| > Nevertheless, they are frustrating and hamper the
| learning process
|
| But that _is_ the learning process. What else is there to
| learn in a language if the syntax doesn 't count? They're
| all Turing complete and all of them can do everything.
| All we need to do is learn the exact magic words.
| richbell wrote:
| I never said otherwise. My point is that Go is far harder
| to learn than they're implying. It certainly can't be
| learned over the weekend -- well, maybe it can be, but
| the code you end up writing will Inevitably be full of
| resources leaks, panics, nil pointer issues, improperly
| handled errors, etc. You may be able to put together some
| basic logic, but you are far from understanding the
| language.
|
| I don't think it's honest to parade Go as a language
| that's the paragon of simplicity that's easy to learn
| when that's simply not true. I also don't think it's
| honest for people to argue that addressing any of Go's
| countless warts would somehow make the language more
| complex or harder to learn.
| adtac wrote:
| I agree that it's very unlikely for someone to learn Go
| in a week and start writing flawless code.
|
| But Go's real strength is in its readability, not
| writability. I think it's very much possible to learn Go
| in a week, then read clean Go code like the standard
| library and understand exactly what's going on. At least
| that's my interpretation of what it means for a new grad
| to be productive in Go in less than a week. Nobody is
| expecting someone new to write production-grade libraries
| with intricate concurrency bits in their first week, but
| they're already productive if they can read and
| understand it.
|
| As a rule of thumb we spend 10x more time reading code
| than we do writing it (code reviews, debugging,
| refactors). So why not optimise for it?
| lawrjone wrote:
| I don't have too much an opinion on either side here, but
| as a developer who works full time in Go (and has for >6
| years) all these things exist in Go.
|
| Uncaught exceptions -> panics, like what this nil catcher
| is aiming to solve
|
| Places far away -> easy goroutine creation with no origin
| tracking makes errors appear sometimes very far away from
| source
|
| Use after free -> close after close
|
| FactoryFactoryFactories -> loads of
| BuilderFunc.WithSomething
|
| Lots of other pains I could add that are genuinely novel
| to Go also, but funny that for everything you mentioned
| my head went "yep, just called X"
| ndreas wrote:
| This is the core of the problem. Of course you can learn
| the language in a weekend, but you're bound to make the
| same mistakes developers have been doing for decades.
|
| This may be ok, as you say, if you allow errors here and
| there because you are fine dealing with those problems. But
| at the other end, it may be a user that is affected by the
| error. Which may be ok as well, but why should it be? We
| lament the quality of software all the time.
|
| Compare this to other engineering fields: unless you study
| the knowledge of those who came before you may not even be
| allowed to practice in the field. I would not want to use a
| bridge built by someone who learned bridge building in a
| weekend.
|
| Software is different though, it's rarely a matter of life
| or death. Given that, maybe it's ok to not have the highest
| quality in mind, because the benefit of productivity far
| outweighs the alternative.
|
| I'm torn.
| danenania wrote:
| Go is just making a certain set of tradeoffs. If you try
| to fix all the "mistakes developers have been doing for
| decades", you get Rust. And considering that Rust is
| already Rust, there is not much point in trying to make
| Go another Rust.
|
| The line has to be drawn somewhere. I think everyone has
| certain things they'd put on the other side of that line,
| and strict nils are probably at the top of the list for
| many, but overall it's good that the Go team is stubborn
| about not adding new stuff. If they weren't, maybe there
| would be better nil handling, better error handling, etc.
| but compiles would also get slower and the potential for
| over-engineering, which Go now discourages quite
| effectively, would increase. At a high level, keeping Go
| a simple, pragmatic language with a fast compiler is more
| important than any particular language feature.
| munchler wrote:
| I don't think anyone is suggesting that Go should be like
| Rust. It's too late for that. We're suggesting that
| people should just use Rust (or Haskell, or F#, or any
| other robust functional programming language) instead.
| eviks wrote:
| > And judging by the absolute success Go has (measured by
| contributions to Github), many many many many many
| developers agree with me on this
|
| You can't make up that other devs' opinions / preferences
| are identical to yours just because they use the same
| language, there are other important factors in play (e.g.,
| if your company is using Go, then you'd be more productive
| in it and be more likely to choose to contribute in it even
| it Go is less productive as a language)
| tuetuopay wrote:
| It's funny how I always hear the point about new hires for
| Go. My team is a Rust shop at $DAYJOB that I created
| basically from scratch, so I had to onboard every new hire
| on the codebase. It's amazing how confident they are due to
| the compiler having their back, and how confident I am
| their code won't blow up that much in prod.
|
| > code is straightforward and easy to read
|
| I have to disagree. I don't want to read 3 lines out of
| four that are exactly the same. I don't want to read the
| boilerplate. I don't want to read yet another abs or
| array_contains reimplementation. Yes it's technically easy
| to read, but the actual business logic is buried under so
| much noise that it really hinders my capacity to digest it.
|
| > the compiler is blazing fast
|
| much agreed, that is my #1 pain point in rust (but it's
| getting better!)
|
| > and it's concurrency model is probably the most
| convenient I have ever seen
|
| this so much. this is what I hate the most with go: it
| pioneered a concurrency model and made it available to the
| masses, but it has too many footguns imho. this is no
| surprise other languages picked channels as a first class
| citizen in their stdlib or core language.
|
| > Go trades "being-modern" for amazing productivity.
|
| I don't think those two are incompatible. If we take the
| specific point of the article, which is nil pointers, Go
| would only have to import the sum types concept to have
| Option and maybe Result as a bonus. Would this translate to
| a loss of productivity? I don't think so. (oh and sum types
| hardly are a modern concept)
|
| Also, there may be a false sense of productivity. Go is
| verbose, and you write a lot. Sure if you spend most of
| your time typing then yes you are productive. But is it
| high-value productivity? Some more concise languages leave
| you more time to think about what you are writing and to
| write something correct. The feeling of productivity is not
| there because you are not actively writing code most of the
| time. IIRC plan9 makes heavy use of the mouse, and people
| feel less productive compared to a terminal because they
| are not actively typing. They are not active all the time.
| usrbinbash wrote:
| >It's amazing how confident they are due to the compiler
| having their back, and how confident I am their code
| won't blow up that much in prod.
|
| I get what you're saying, and I'm glad you are having
| such a good experience with it. Disclosure, I am not
| talking down to any language here...in fact I actually
| _like_ Rust as a language, even though I don 't use it
| professionally.
|
| I am just saying that Go is incredibly easy to learn, and
| I don't think there are many people who disagree on this
| point, proponent of Go or not.
|
| > I have to disagree.
|
| We'll have to agree to disagree then :-) Yes, the code is
| verbose, but it's not really noise in my opinion. Noise
| is something like what happens in enterprise Java, where
| we have superfluous abstractions heaped ontop of one
| another. Noise doesn't add to the program. The verbose
| error handling of Go, and the fact that it leaves out a
| lot of magic from other languages doesn't make it noisy
| to me.
|
| > I don't think those two are incompatible.
|
| Neither do I, but that's the path Go has chosen. It may
| also have been poorly worded on my part. A better way of
| putting it: Go doesn't subscribe to the "add as much as
| possible" - mode of language development.
|
| > But is it high-value productivity?
|
| Writing the verbose parts of go, like error checking,
| isn't time consuming, because it's very simple...in fact,
| these days I leave a lot of that to the LLM integration
| of my editor :-)
|
| Is is high value? Yes, I think so, because I don't
| measure productivity by number of lines of code, I
| measure it by features shipped, and issues solves. And
| that's where Go's ... how do I say this ... _obviousness_
| really puts the language into the spotlight for me.
| mcronce wrote:
| > Also, there may be a false sense of productivity. Go is
| verbose, and you write a lot. Sure if you spend most of
| your time typing then yes you are productive. But is it
| high-value productivity? Some more concise languages
| leave you more time to think about what you are writing
| and to write something correct. The feeling of
| productivity is not there because you are not actively
| writing code most of the time. IIRC plan9 makes heavy use
| of the mouse, and people feel less productive compared to
| a terminal because they are not actively typing. They are
| not active all the time.
|
| This is my sense. "False sense of productivity" is an
| accurate statement - I've also found that it seems to be
| for a very specific (and not necessarily useful)
| definition of "productive", such as LOC per day.
|
| It's not as bad as dynamic languages like Python, but
| very frequently Go codebases feel brittle, like any
| change I make might bring down the whole house of cards
| at runtime.
| Conscat wrote:
| > it pioneered a concurrency model and made it available
| to the masses
|
| Isn't it basically just what Cilk did, but with fewer
| feaures?
| marsavar wrote:
| While this is a great piece of engineering, and will certainly
| deliver a huge amount of value to any project, the fact that a
| whole new tool had to be built (and will have to be maintained)
| to address serious, fundamental shortcomings in the language is
| really quite sad.
| foldr wrote:
| It's worth bearing in mind that some of these runtime panics
| would have happened anyway even if the code had been
| implemented in (e.g.) Rust. Ugly real world code tends to make
| quite frequent use of unwrap() or equivalents. For example:
| https://github.com/search?q=repo%3Arust-lang%2Frust+.unwrap%...
| insanitybit wrote:
| You can `grep unwrap` for Rust, you need an entire SAT solver
| for Go.
| tgv wrote:
| What do you think the borrow checker is?
| insanitybit wrote:
| A completely unrelated construct?
| foldr wrote:
| That's almost literally what I just did, but what use is
| it? No-one is really going to go through all those unwrap
| calls and check them.
| mariopt wrote:
| Rust is a lot better in this aspect, but this is a symptom of
| not having proper code review and standards. Do not forget
| that in some scenarios using unwrap is totally fine if a
| panic is acceptable. The same could be said for javascript:
| How many time have we not wrapped JSON.parse inside a try
| catch? More than we would like to admit. Really appreciate
| Rust "forces" you to handles all execution paths.
| foldr wrote:
| My link is to the rust repo on github. Does the Rust
| project not have proper standards for Rust code?
| Philpax wrote:
| > Do not forget that in some scenarios using unwrap is
| totally fine if a panic is acceptable.
|
| Looking through the first few pages, most of these panics
| are easy to audit, and are infallible or in contexts
| where it doesn't matter (internal tooling, etc). That's a
| pretty stark difference to _every_ single reference being
| a potential landmine.
| foldr wrote:
| Yes, you are probably less likely to get a panic caused
| by a nil reference in Rust than in Go. My point is that
| the equivalent software written in Rust (or most other
| languages with option types) would probably have had at
| least _some_ of these very same bugs.
| eviks wrote:
| It's also worth reading the examples you post
|
| Like, one of the first files has only .unwraps in the
| comments (like a dozen of them in a file), some are
| infallible uses, some are irrelevant-to-runtime tooling, etc.
|
| But anyway, "some" is a lot smaller than "all". Just like
| some of memory safety issues would also have happened since
| you can still use unsafe in Rust, yet it's still a big step
| forward in reducing those issues in the ugly real world
| foldr wrote:
| It's a list of all instances of ".unwrap()" in the project,
| so of course it includes instances irrelevant to my point.
| Seems uncharitable to assume that I haven't looked through
| it on that basis.
| shakow wrote:
| Then please pinpoint some problematic ones, so that not
| every reader has to delve into pages to continue the
| discussion.
| foldr wrote:
| The point is precisely that it's not always easy to
| figure out which instances are problematic.
|
| If you think about it a bit, given that bugs are
| relatively rare in a mature project, it's going to be
| difficult to find a use of unwrap that's _definitely_
| bad.
| skybrian wrote:
| I don't see this as a band-aid. It's doing proper type checking
| (static analysis) and that seems quite promising?
|
| Getting good type errors without requiring type annotations
| seems like a win over languages that are annotation-heavy.
| Normally I'd be skeptical about relying on type inference too
| much over explicit type declarations, but maybe it's okay for
| this problem?
|
| This is speculative, but I could see this becoming another win
| for the Go approach of putting off problems that aren't urgent.
| Sort of like having third-party module systems for so many
| years, and then a really good one. Or like generics.
| insanitybit wrote:
| I guess you could call it a bandage? The point wasn't to bad-
| mouth it or undersell it - bandaids are awesome. The point is
| that we need some external thing to patch holes in the
| underlying system.
| bb88 wrote:
| Agree, this is dumb. This should be a part of the compiler if
| the language def can't simplify this for the user.
| aatd86 wrote:
| Very interesting work. I wonder what were the difficulties
| encountered. Aliasing? Variable reassignment wrt short
| declaration shadowing?
|
| Hopefully with time, when exploring union types and perhaps a
| limited form of generalized subtyping (currently it's only
| interface types) we'll be able to deal with nil for good.
|
| Nil is useful, as long as correctly reined in.
| mrkeen wrote:
| > Nil is useful, as long as correctly reined in.
|
| A good way to rein in behaviour is with types. If you need Nil
| in your domain, great! Give it type 'Nil'.
| aatd86 wrote:
| Yes that's part of it. It will probably require a nil type
| which is currently untyped nil when found in interfaces.
|
| The untyped nil _type_ is just not a first-class citizen
| nowadays.
|
| But with type sets, we could probably have ways to track
| nillables at the type system level through type assertions.
|
| And where nillables are required such as map values it would
| be feasible to create some from non nillables then (
| interface{T | nil})
|
| But that's way ahead still.
| candiddevmike wrote:
| It's really easy to check a field of a pointer struct without
| first checking the struct is non nil. Would be interesting if
| go vet or test checked this somehow.
| npalli wrote:
| "The Go monorepo is the largest codebase at Uber, comprising 90
| million lines of code (and growing)"
|
| Is this just a symptom of having a lot of engineers and they keep
| churning code, Golang being verbose or something else. Hard time
| wrapping my head around Uber needing 90+ million lines of
| code(!). What would be some large components of this codebase
| look like?
| gavinray wrote:
| From what I've heard from ex-FAANG, I'd wager that a
| significant portion of the Go is code-generated for things like
| RPC definitions or service skeletons.
| dilyevsky wrote:
| They use bazel so generated rpc code is produced on the fly
| and is not checked in
| vrosas wrote:
| Uber is famous for NIH syndrome. You can tell by their open
| source projects they've basically built every part of their
| infra from scratch. So it's not just the application code but
| everything else that helps run it.
| ianmcgowan wrote:
| Either genius or madness, you be the judge!
| latchkey wrote:
| I personally find uberFX to be a fantastic project. It isn't
| necessary for you to write golang with it, but it certainly
| does provide a great framework for organizing code so that
| you can ensure that writing tests is as easy as it can be.
| foobiekr wrote:
| Basically exactly this. Also, a lot of their "in production"
| open source projects are not "in production" but were
| generated and released as part of their broken promotion
| process.
| vitiral wrote:
| It is the nature of large systems to grow. As software
| engineers we build libraries to build libraries, we build tools
| on top of tools to check our tools.
| adtac wrote:
| Imagine a multidimensional matrix with various payment methods,
| local regulations, cloud providers, third party dependencies,
| web/mobile platforms, etc. Then also add more dimensions for
| internal things like accounting, hiring, payroll, promotions,
| compliance, security, monitoring, etc. Then double it for Uber
| Eats or whatever.
|
| There's a lot of overlap and some invalid combinations, but
| you're still left with a huge number of combinations where Uber
| must simply work. And every time you add a new thing to this
| list, the total number of combinations grows polynomially.
|
| (Also, Go is slightly more verbose than most languages. I think
| that's a feature and not a bug, but it's one more reason.)
| adra wrote:
| Dang, a little more verbose? Understatement of a lifetime.
| It's fine, if you like it not whatever, but it is quite a bit
| more verbose than many languages that I've used. My number 1
| qualms go is with such simple building blocks requiring a
| bunch of redundant boiler plate. You're welcome to disagree
| with my opinion here.
|
| A lot of people seems to gravitate toward languages with less
| dense cognitive load. I have learned to love kotlin, but its
| also a super dense set of syntax to power it's very
| expressive language.
| lopkeny12ko wrote:
| When you have thousands and thousands of engineers, and they
| are evaluated by how much code they produce, and they need to
| justify their job and continued employmwment, you end up with a
| 90M line codebase.
| jheriko wrote:
| how in the hell does uber need engineering problems solved?
|
| mad
| technics256 wrote:
| Is there any movement in the language spec to address this in the
| future with Nil types or something?
| aatd86 wrote:
| If you squint really hard, the work on generics is a step
| toward the future.
|
| If you don't squint, then I don't think so.
| mcronce wrote:
| With generics, can you not make a NonNil<T> struct in Go,
| where the contents of the struct are only a *T that has been
| checked at construction time to not be nil, and doesn't
| expose its inner pointer mutably to the public? I would think
| that would get the job done, but I also haven't really done
| much Go since prior to generics being introduced
|
| Otherwise, since pointers are frequently used to represent
| optional parameters, generics + sum types would get the job
| done; for that use case, it's one of two steps to solve the
| problem. I don't foresee Go adding sum types, though.
| suremarc wrote:
| Every type in Go has a zero value. The zero value for
| pointers is nil. So you can't do it with regular pointers,
| because users can always create an instance of the zero
| value.
| tialaramex wrote:
| This is one of those things which feels like just a small
| trade off against convenience for the language design,
| but then in practice it's a big headache you're stuck
| with in real systems.
|
| It's basically mandating Rust's Default trait or the C++
| default (no argument) constructor. In some places you can
| live with a Default but you wish there wasn't one.
| Default Gender = Male is... not great, but we can live
| with it, some natural languages work like this, and there
| are problems but they're not insurmountable. Default Date
| of Birth is... 1 January 1970 ? 1 January 1900? 0AD ?
| Also not a good idea but if you insist.
|
| But in other places there just is no sane Default. So
| you're forced to create a dummy state, recapitulating the
| NULL problem but for a brand new type. Default file
| descriptor? No. OK, here's a "file descriptor" that's in
| a permanent error state, is that OK? All of my code will
| need to special case this, what a disaster.
| maccard wrote:
| I write a decent amount of go - this isn't a defence of
| the current situation.
|
| > All of my code will need to special case this, what a
| disaster.
|
| No, your code should handle the error state first and
| treat the value as invalid up until that point, e.g.
| foo, err := getVal() if err != nil {
| return } // foo can only be used now
|
| It's infuriating that there's no compiler support to make
| this easier, but c'est la vie.
| andreyvit wrote:
| Default gender male not how this works in practice.
| Instead, you define an extra "invalid" value for almost
| every scalar type, so invalid would be 0, male 1 and
| female 2. Effectively this makes (almost) every scalar
| type nullable. It is surprisingly useful, though, and I
| definitely appreciate this tradeoff most of the time.
|
| (Sometimes your domain type really does have a suitable
| natural default value, and you just make that the zero
| value.)
| codetrotter wrote:
| > Default Gender = Male is... not great
| enum Gender { Unspecified, Male,
| Female, Other, } impl
| Default for Gender { default() -> Self {
| Self::Unspecified } }
|
| or: enum Gender { Male,
| Female, Other, }
|
| and use Option<Gender> instead of Gender directly, with
| Option::None here meaning the same that we would mean by
| Gender::Unspecified
| baby wrote:
| I really love Golang and how it focused on making the job of
| the reader easy. But with today's modern programming language
| the existence of null pointer dereference bugs doesn't really
| make sense anymore. I don't think I would recommend anyone to
| start a project in Golang today.
|
| Maybe we'll get a Golang 3 with sum types...
| __turbobrew__ wrote:
| I don't really buy the usefulness of trying to statically detect
| possible nil panics. In their example of a service panicing 3000+
| times a day why didn't they just check the logs to get the stack
| trace of the panic and fix it there? I don't see why static
| analysis was needed to fix that panic in runtime.
|
| What I would really like golang to have is way to send a "last
| gasp" packet to notify some other system that the runtime is
| panicing. Ideally at large scales it would be really nice to see
| what is panicing where and at what time with also stack traces
| and maybe core dumps. I think that would be much more useful for
| fixing panics in production.
|
| There was a proposal to add this to the runtime, but it got
| turned down: https://github.com/golang/go/issues/32333 Most of
| the arguments against the proposal seem to be that it is hard to
| determine what is safe to run in a global panic handler. I think
| the more reasonable option is to tell the go runtime that you
| want it to send a UDP packet to some address when it panics. That
| allows the runtime to not support calling arbitrary functions
| during panicing as it only has to send a UDP packet and then
| crash.
|
| I could see the static analyzer being useful for helping prevent
| the introduction of new panics, but I would much rather have
| better runtime detection.
| styluss wrote:
| Because they want to find the code paths before deploying the
| code. Surely they have error logging or tracing and can see why
| it panics.
|
| I tried this with a medium sized project and some unexpected
| code that could panic 3 functions away from the nil.
| adtac wrote:
| I'm not sure if that was the best example to showcase NilAway. I
| understand there's a lot of context omitted to focus on NilAway's
| impact, but why is foo returning a channel to bar if bar is just
| going to block on it anyway? Why not just return a *U? If foo's
| function signature was func foo() (*U, error) {}, this wouldn't
| be a problem to begin with.
| iot_devs wrote:
| Wonderful job.
|
| I am toying around with a similar project, with the same goal,
| and it is DIFFICULT.
|
| I'll definitely get to learn from their implementation.
| nirga wrote:
| It amazes me that in 2023 this is not a solved problem by design
| of the language. Why go doesn't adapt the "optional" notion of
| other languages so that if you have a variable you either _know_
| it is not null or _know_ that you must check for nullness. The
| technology exists
| ikari_pl wrote:
| The same reason you can't get map keys without a library or
| looping yourself. "Simplicity" (for go maintainers).
| adtac wrote:
| That's what the `func foo() (*T, error)` pattern is for. It's
| actually better than syntactic sugar for optional values
| because now you also have a descriptive reason for _why_ the
| value is nil.
|
| But if you really cannot afford to return more than one bit of
| information, do `func foo() (*T, bool)`.
| kortex wrote:
| > a descriptive reason for why the value is nil.
|
| Result<T,E> does this. I forget exactly why Result is
| actually different from, and in fact superior to, `func foo()
| (*T, error)` but IIRC it has to do with function composition
| and concrete vs generic types.
| aardvark179 wrote:
| Don't rely on half remembering how specific languages
| implement things, try and internalise the fundamentals. Go
| functions tend to return a tuple which is a product type,
| while rust's result type is sum type. Product types
| contain. Both things (a result and an error) while a sum
| type contains a result or an error.
| progbits wrote:
| Result<T,E> is in one of two states: It either has value of
| type T, or error of type E.
|
| (*T, error) is either T (non-nil, nil), or error
| (nil/undefined, non-nil), or both (non-nil, non-nil), or
| neither (nil, nil). By convention usually only the first
| two are used, but 1) not always, 2) if you rely on
| convention why even have type system, I have conventions in
| Python.
|
| Leaving aside pattern matching and all other things which
| make Rust way more ergonomic and harder to misuse, Go
| simply lacks a proper sum type that can express exactly one
| of two options and won't let you use it wrong. Errors
| should have been done this way from the start, all the
| theory was known and many practical implementations
| existed.
| masklinn wrote:
| As others have mentioned, Result is a sum type so you
| either have a T or an E, there's no situation in which you
| can get both or neither.
|
| The second part is that it's reified as a single value, so
| it works just fine as a normal value e.g. you can map a
| value to a result, or put results in a map, etc... ,
| language doesn't really care.
|
| And then you can build fun utilities around it e.g. a
| transformer from Iterator<Item=Result<T, E>> to
| Result<Vec<T>, E> (iterates and collects values until it
| encounters an error, in which case it aborts and
| immediately returns said error).
| fizx wrote:
| There's much I don't love about Rust, but I feel golang could
| steal the ? operator and keep the spirit of go.
|
| Effectively,
|
| instead of result, err := doSomething()
| if err != nil { return nil, err }
|
| you'd get the same control flow with result
| := doSomething()?
| tialaramex wrote:
| Try (the current incarnation of the ? operator) is actually a
| very clever trait which does rather more than that.
|
| Types for which Try is implemented can Try::branch() to get a
| ControlFlow, a sum type representing the answer to the
| question "Stop now or keep going?". In the use you're
| thinking of where we're using ? on a Result, if we're Err we
| should stop now, returning the error, whereas if we're OK we
| should keep going.
|
| And that's why this works in Rust (today), when you write
| doSomething()? the Try::branch() is executed for your Result
| and resolves into a Break or a Continue which is used to
| decide to return immediately with an error or continue.
|
| But this is also exactly the right shape for other types in
| situations where _failure_ has the opposite expectation, and
| we should _keep going_ if we failed, hoping to succeed later,
| but stop early if we have a good answer now.
| cedws wrote:
| That won't work because in Go you often need to wrap errors
| with additional context.
|
| I have worked with Rust Option/Rust types and found them
| extremely unergonomic and painful. The ?s and method chains
| are an eyesore. Surely PLT has something better for us.
| vips7L wrote:
| Do go errors or rust options include stack traces?
| monksy wrote:
| It's because it's golang and that's how they oninonatedly want
| it. (Seriously, the community is incredibly stubborn and
| controlling)
| pjmlp wrote:
| There are several language design problems solved in the 20th
| century that Go designers decided to ignore, because they
| require PhD level skills to master, apparently.
|
| Hence why the language is full of gotchas like these.
|
| Had it not been for Docker and Kubernetes success, and most
| likely it wouldn't have gotten thus far.
| tehjoker wrote:
| speaking from personal experience, i selected go for a
| project because it is high perf, automatically uses all cores
| w/ goroutines, and is type checked
| triyambakam wrote:
| > type checked
|
| Kinda...
| 65a wrote:
| It is a type safe language, not exactly sure what you're
| hinting at here.
| bb88 wrote:
| And now they're stuck, since they doubled down on not making
| any language changes for the 2.0 release.
|
| They made the language easier and quicker to write a
| compiler, but harder to write programs in, and it doesn't
| look like that will change in Go 2.0.
| pjmlp wrote:
| At least many CNCF projects are now adopting Rust, Java, C#
| and even to a lesser extent C++.
| bb88 wrote:
| There's nothing better than a panic in production caused
| by a third party library.
| pjmlp wrote:
| This whole thread is about the money Uber has spent to
| work around panics in Go.
| andreyvit wrote:
| I write a lot of Go and used to write a lot of Swift. Swift is
| what you'll consider a modern language (optionals, generics,
| strong focus on value types), while Go is Go.
|
| I appreciate both languages, and of course Swift feels like
| what you'd pick any day.
|
| But, after using both nearly side by side and comparing the
| experience directly, I've got to say, I'm so much more
| productive in Go, there's SO much less mental burden when
| writing the code, -- and it does not result in more bugs or
| other sorts of problems.
|
| Thing is, I, of course, am always thinking about types,
| nullability and the like. The mental type model is pretty rich.
| But the more intricacies of it that I have to explain to the
| compiler, the more drag I feel on getting things shipped.
|
| And because Go is so simple, idiomatic, and basically things
| are generally as I expect them to be, maintenance is not an
| issue either. Yes, occasionally you are left wondering if a
| particular field can or cannot be nil / invalid-zero-value, but
| those cases are few enough to not become a problem.
| luxurytent wrote:
| Plenty of Go commentary in this thread but can I just say I'm
| glad to have learned about nilness? Suffered through a few nil
| pointer dereferences after deploying and having this analyser
| enabled in gopls (off by default for me at least) is a nice
| change.
|
| Tested via vim and looks good!
| wg0 wrote:
| 90 million lines of code to .. call a cab?
|
| Genuinely curious what's so much of business logic is for.
| techn00 wrote:
| Uber does way more than calling a cab, however, I was also
| surprised by the number of lines of code
| vore wrote:
| And billing, and reporting, and regulatory compliance, and
| inventory management, and abuse detection, and routing, and
| operations, and...
| yankput wrote:
| I was working on a Grab competitor, you would be surprised
| about the number of subsystems running there.
|
| There are entire teams that are working on just internal
| services that connect some internal tools together.
|
| There was also very little effectivity and efficiency in the
| era of cheap capital so there were tons of talent wasted on
| nonsense. Uber built their own slack for a while!! (before just
| going to mattermost)
|
| People always ask who actually makes money on Uber... I think
| it's not the cab drivers, not the investors, who makes money is
| the programmers. It's a transfer of money from Saudis to
| programmers.
|
| Well it was, anyway.
| robryan wrote:
| A lot of it will be location based. It has come up before here
| in the discussion of why there is so much in the app. They have
| to cater for all the different rules in every jurisdiction.
| jeffrallen wrote:
| Can we not link to scammy engineering blog articles with ads for
| scammy restaurant apps on top please?
|
| Link to the source, or better yet, never link at all to anything
| related to Uber.
| carbocation wrote:
| Just tried this out on some of my own code and it nails the warts
| that I had flagged as TODOs (and a few more...). The tool gives
| helpful info about the source of the nil, too. This is great.
| anonacct37 wrote:
| I do like the approach of static code analysis.
|
| I found it a little funny that their big "win" for the nilness
| checker was some code logging nil panics thousands of time a day.
| Literally an example where their checker wasn't needed because it
| was being logged at runtime.
|
| It's a good idea but they need some examples where their product
| beats running "grep panic".
| hardwaresofton wrote:
| The code:
|
| https://github.com/uber-go/nilaway
| earthboundkid wrote:
| I tried it but got too many false positives to be useful.
___________________________________________________________________
(page generated 2023-11-18 23:00 UTC)