mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 18:59:04 +08:00
Jeong, YunWon
259ce8ce66
* _imp: read get_frozen_object's data as a whole marshal value
The data argument went through deserialize_code(), which reads a code body
without the type byte in front of it, so nothing marshal.dumps() produces was
accepted. Read it with marshal.loads() and require a code object back.
Assisted-by: Claude
* _imp: bind find_frozen's arguments with FromArgs
The arity check was done by hand on a FuncArgs. Take the arguments through a
FromArgs struct instead, which makes withdata keyword-only, and fill in the
data it asks for: the frozen encoding is not marshal, so the code is
re-serialized into what get_frozen_object() reads back.
Assisted-by: Claude
* itertools: give tee's shared buffer a Python type
The buffer was a PyRc<PyItertoolsTeeData>, which is not a Python object, so the
collector could not walk into it and any cycle running through a tee was
uncollectable. Make it the _tee_dataobject type with a traverse, held by
PyRef, and split the rest along the same lines: tee() is a function, _tee is
the iterator type it builds, and _tee takes a single iterable rather than
returning a tuple from __new__. _tee is weak-referenceable, tee() rejects a
negative n with ValueError and reserves its tuple fallibly.
test_itertools.test_tee passes now; its expectedFailure marker is removed.
Assisted-by: Claude
* contextvars: hold the shared context state under locks
The variable map was a RefCell, the enter flag, the context index, the token's
used flag and the variable hash were Cells, and each carried an unsafe impl
Sync. A Context or ContextVar shared between threads overlapped their borrows
and panicked. The map and the per-variable cache are now PyMutex, the flags and
the index are atomics, entering a context is a compare_exchange, and the three
unsafe impl Sync are gone.
The cache also stopped being read through AtomicCell::as_ptr, which raced a
concurrent store on a value holding a PyObjectRef.
Values displaced from the map or the cache are dropped after the lock is
released: __del__ can come straight back into the same context, and the locks
are not reentrant.
Assisted-by: Claude
* generator: read the frame state under the running claim
send(), send_none(), throw() and close() read `closed` and `frame.lasti()`
before `running` was compare_exchanged, so the frame they went on to resume
could be one another thread had already advanced. A resume that decided from
`lasti() == 0` that the generator had not started pushes no value onto the
value stack, and the code after the yield pops one, which underflows the stack.
The compare_exchange now hands back a guard, taken before those reads and
released after maybe_close(), so the generator is retired while it is still
claimed.
Assisted-by: Claude
* itertools: claim a tee's position and its buffer as one step
`_tee::next` read `index` and moved it on afterwards, and
`_tee_dataobject::get_item` released `running` before the value it fetched
from the source was cached.
Two callers on one `_tee` then read the same index, hand out the same value
twice and advance past a value that was never cached, which leaves `index`
past `values.len()` and indexes the buffer out of bounds. Two callers at the
same index on separate tees each fetch a value from the source, and one of the
two is dropped without reaching a caller.
Both claims now cover the read and the update.
Assisted-by: Claude
* _asyncio: require InvalidStateError to be a type
new_invalid_state_error() called whatever `asyncio.exceptions.InvalidStateError`
names and unwrapped the downcast of the result, so a future asked for a result
it does not have panicked once that attribute was rebound to something that is
not an exception:
asyncio.InvalidStateError = lambda *args: 42
_asyncio.Future(loop=object()).result()
The type is looked up the way get_cancelled_error_type() looks its own up, and
raised with new_exception_msg; a lookup that does not produce an exception type
falls back to RuntimeError, as the other arms already did.
Assisted-by: Claude
96 lines
2.2 KiB
Python
96 lines
2.2 KiB
Python
"""Resume one generator from several threads at once.
|
|
|
|
A generator is resumed by one thread at a time, and whether the sent value is
|
|
pushed onto the frame's value stack depends on whether the generator has
|
|
already started. Deciding that before the generator is claimed reads a frame
|
|
another thread can advance in the meantime, and resuming it then leaves the
|
|
stack short of what the code after the yield expects.
|
|
|
|
Every yielded value still has to reach exactly one caller: threads that lose
|
|
the race get a ValueError instead of a value.
|
|
"""
|
|
|
|
import threading
|
|
|
|
WORKERS = 4
|
|
ROUNDS = 400
|
|
|
|
|
|
def counter():
|
|
yield 1
|
|
yield 2
|
|
yield 3
|
|
|
|
|
|
gens = [counter() for _ in range(ROUNDS)]
|
|
received = [[] for _ in range(ROUNDS)]
|
|
start = threading.Barrier(WORKERS)
|
|
errors = []
|
|
|
|
|
|
def worker():
|
|
try:
|
|
for index, gen in enumerate(gens):
|
|
start.wait()
|
|
for _ in range(3):
|
|
try:
|
|
received[index].append(next(gen))
|
|
except StopIteration:
|
|
break
|
|
except ValueError:
|
|
# another thread is running this generator
|
|
pass
|
|
except Exception as exc: # noqa: BLE001
|
|
errors.append(exc)
|
|
# the other workers are waiting at the barrier for this one
|
|
start.abort()
|
|
|
|
|
|
threads = [threading.Thread(target=worker) for _ in range(WORKERS)]
|
|
for t in threads:
|
|
t.start()
|
|
for t in threads:
|
|
t.join()
|
|
|
|
assert not errors, errors
|
|
for got in received:
|
|
# no value handed out twice, and none skipped
|
|
assert sorted(got) == list(range(1, len(got) + 1)), got
|
|
|
|
|
|
# a generator that is closed while it is being resumed stays consistent
|
|
def loop():
|
|
while True:
|
|
yield 1
|
|
|
|
|
|
shared = loop()
|
|
closed = threading.Barrier(2)
|
|
|
|
|
|
def resumer():
|
|
closed.wait()
|
|
for _ in range(ROUNDS):
|
|
try:
|
|
next(shared)
|
|
except (StopIteration, ValueError):
|
|
pass
|
|
|
|
|
|
def closer():
|
|
closed.wait()
|
|
try:
|
|
shared.close()
|
|
except ValueError:
|
|
# the generator was running
|
|
pass
|
|
|
|
|
|
pair = [threading.Thread(target=resumer), threading.Thread(target=closer)]
|
|
for t in pair:
|
|
t.start()
|
|
for t in pair:
|
|
t.join()
|
|
|
|
print("ok")
|