[HN Gopher] Data races in Python, despite the Global Interpreter...
___________________________________________________________________
Data races in Python, despite the Global Interpreter Lock
Author : verdagon
Score : 24 points
Date : 2022-02-21 21:02 UTC (1 hours ago)
(HTM) web link (verdagon.dev)
(TXT) w3m dump (verdagon.dev)
| _ache_ wrote:
| I'm unable to reproduce. Increasing the number of iterations to
| `1000000` and using a temporary variable `c = counter + 1;
| counter = c` doesn't help either.
|
| Why ? I'm on Linux, using Python 3.10. It's only happening using
| Python2.7.
| Aperocky wrote:
| Python 2.x is unsupported and anyone using it should operate on
| the assumption that whatever bug its got is final unless
| they're the one who is going to fix it.
| verdagon wrote:
| Author here, u/skeeto from
| https://www.reddit.com/r/programming/comments/sxy5q4/pythons...
| has some good insight into the 3.10 difference:
|
| > Ironically, CPython 3.10 has gone the opposite direction and
| made thread scheduling much more deterministic. It now only
| releases the GIL on backwards edges in the byte code The
| example in the article always prints 40000000 in CPython 3.10!
| I expect this will ultimately make Python code less reliable in
| the future as many programs will accidentally depend on this
| behavior.
| dehrmann wrote:
| And there's this:
| https://web.archive.org/web/20201108091210/http://effbot.org...
| samwillis wrote:
| Interestingly if you change the line to `counter += 1` it still
| has the race condition. I'm not sure how the byte code is
| different for the two options but it doesn't make a difference, I
| had hoped it would.
| miohtama wrote:
| In the Python VM there is no atomic increment bytecode. So
| `counter += 1` should be exactly same as `counter = counter +
| 1`.
|
| Here is an example what thread safe increment looks like in
| Python:
|
| https://julien.danjou.info/atomic-lock-free-counters-in-pyth...
|
| You need to lock it explicitly.
|
| Note that `INC` instructor for x86 architecture needs explicit
| hints/locks as well, so this should suprise anyone:
|
| https://stackoverflow.com/q/10109679
| jasonhansel wrote:
| In general, "+=" probably needs to be non-atomic to support
| "__add__" overloading; Python wouldn't be able to call
| arbitrary "__add__" methods in a way that could guarantee
| atomicity.
| [deleted]
| tomp wrote:
| Python's GIL does exactly nothing to prevent data races (or any
| other concurrency issues); it merely protects the _runtime_ from
| memory corruption stemming from concurrency.
|
| Obviously, the fundamental issue with concurrency is programmer's
| _intent_. This statement: x += 1; y -= 1
|
| can be interpreted in two ways: atomic {
| x += 1 y -= 1 }
|
| or atomic { x += 1 } atomic { y -= 1 }
|
| The best the compiler can do would be to alert the programmer of
| the ambiguity; I know of no compiler that does that.
| dzqhz wrote:
| I don't see how it could ever be interpreted the first way.
| verdagon wrote:
| The article isn't trying to claim either interpretation, it's
| just using it as an example to show that the GIL doesn't
| actually help protect users from concurrency problems. You'd
| be surprised how many people think that!
|
| I used to think that it did as well, but then my C/Java brain
| kicked in and realized that couldn't be correct. I wrote this
| article to help others see it in action.
| nyanpasu64 wrote:
| If only one thread is reading or writing the counter at a time,
| and holds the GIL while doing so, it's not a data race, but a
| mutex which fails to ensure the atomicity of the read-modify-
| write operation: with GIL: x = counter +
| 1 with GIL: counter = x
| jondgoodwin wrote:
| You claim that a language can guarantee completely deterministic
| runs. How is that possible in Vale?
| verdagon wrote:
| It's tricky but it is possible, if we:
|
| 1. Don't allow any undefined behavior or `unsafe` code in the
| language.
|
| 2. Record all inputs from FFI.
|
| 3. Carefully track the orderings of interactions across
| threads.
|
| The article goes into the first two, but the third one is the
| most interesting IMO:
|
| When we unlock a mutex or send a message, we assign a "sequence
| number" (similar to what we see in TCP packets).
|
| Whenever we lock a mutex or receive a message, we read the
| sequence number and record it to this thread's "recording".
|
| When replaying, we use that sequence number and that file to
| make sure we're reading in the same order as the previous
| execution.
| bb88 wrote:
| Interesting, but how do you know if you've captured all the
| potential states for all possible inputs to a program?
|
| An unexpected state would seem to break the memory model, and
| lead to corrupted data, wouldn't it?
| jasonhansel wrote:
| That's a logical race, not a data race, right? Technically,
| "counter = counter + 1" accesses counter twice (once to read and
| once to write).
|
| The fact that "counter" gets modified between the read and the
| write doesn't imply that multiple accesses were happening
| simultaneously; it just means that accesses from different
| threads were getting interleaved. The GIL would only prevent the
| former but not the latter, since a thread can give up the GIL at
| any point between operations.
| bb88 wrote:
| The GIL is for internal python state. Not for atomic preservation
| of python data types.
|
| I learned very early on that python threads should be treated
| like C threads, and therefore should be avoided. Also there
| really isn't a performance gain to using threads (other than
| maybe waiting for IO completion).
___________________________________________________________________
(page generated 2022-02-21 23:00 UTC)