obj_traverse_i called rb_ary_cancel_sharing on every array it visited,
before the traversal knew whether the object graph was shareable.
So Ractor.shareable? modified frozen arrays even when it returned false.
This commit moves rb_ary_cancel_sharing into mark_shareable, which
runs only after the traversal succeeds.
[Bug #22460]
RARRAY_LENINT raises RangeError for an Array longer than INT_MAX. Ractor.make_shareable failed on one after freezing it, and so did Ractor.shareable? and Ractor.make_shareable(copy: true).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The courier's hash size is an entry count rather than a String or Array length, so it stays `long`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Ractor.make_shareable could previously take the VM lock and then call
rb_funcall(id_freeze), which is not correct to do. Create a new
rb_objspace_reachable_objects_from_local() function that works the same as the
non-local version except doesn't lock. The callback given to it must skip foreign
objects (classes/modules, etc.) as running mark_children could traverse data
structures that could be mutated out from under it.
The reachable_objects_from_local callback must skip foreign objects altogether
instead of checking their shareability because foreign objects that aren't
shareable could be collected or have their page unmapped due to RLGC. Of course,
foreign edges should be shareable anyway. This was a bug that existed in the
locking version as well. Unfortunately, that means we can't check that T_DATA
objects don't refer to unshareables across object spaces while making them
shareable.
The alternative is to check and risk a crash, and raise if we see these illegal
edges.
The registered_globals list made every Ractor's GC scan every slot under
a shared lock. This commit splits it into per-Ractor lists.
We also maintain a global registry that references every Ractor that has
ever registered so that unregistering from a Ractor other than the one
that registered can look up the owning list.
When a Ractor is joined, the dead Ractor's slots are moved to the
joining Ractor's heap before the objspace merge happens.
This change tightens up the requirements for values stored in a
registered address.
Previously, Every Ractor's local GC scanned the shared list. So if a
slot held an unshareable object living in a different Ractor's heap,
that object was always kept alive regardless.
After this change, only the registering Ractor's GC scans its list. So if a
slot holds another Ractor's unshareable object, the owning Ractor's
local GC cannot see that reference, so it can collect the object while
the registered slot still points at it.
Storing such an object was already a Ractor containment violation. It
was just hidden by the shared scan.
[Feature #22277]
Previously, major GC incremental marking was disabled (even on the main Ractor)
when more than 1 Ractor was running. By changing the WB implementation, we now
have it back.
The JIT optimizer sees an `invokebuiltin` and would like to assume that calling it does not write back to the frame. A `Primitive` should never write to the frame by taking a pointer to a local, so this PR removes that functionality.
Partial revert of https://github.com/ruby/ruby/commit/bb8924240d329091eaa38f46791036ac13ccd3ed.
----
* Don't allow writing to locals in cexpr
This makes the few remaining cases of `cexpr`-that-looks-at-locals a
read-only view.
* Remove uses of LOCAL_PTR from ractor.rb
Use two new builtin functions instead.
* Remove LOCAL_PTR from mk_builtin_loader
* Don't bother making the intermediate pointer
* Revert bb8924240d
* Add Primitive.local_self!
When running multiple Ractors, we can't call dfree of a non-thread-safe
T_DATA while other Ractors are running or doing GC work. If there are multiple
Ractors, we save the T_DATA object's `dfree` and `data` pointer in an entry. Once
the entry buffer is full per-Ractor, we publish it to the global list atomically.
Once the threshold for these deferred T_DATAs is reached (right now ~65K), the next
Ractor to run interrupts calls the `dfree` functions for these objects under the VM
barrier. It does not do any other GC work during this postponed job.
If a global GC runs, it also calls the `dfree` functions for all these objects (it drains
the deferred-free queue). When only 1 Ractor is running (`rb_gc_single_objspace_p()`),
these objects are freed normally and aren't added to any queue.
NOTE
----
Embedded T_DATA objects need to be special-cased because their `data` points to inside the
object's slot itself, which we reuse right away now. For non-thread-safe T_DATA that is
embeddable, we need to create it as unembedded - even without Ractors.
Previously the Ractor courier rejected singleton class objects. However
we have existing tests of OpenStruct that those objects can be
Ractor-copied.
dup and clone, which were used prior to RLGC's merge, also just ignore
singleton classes, so I believe this is in line with existing behaviour.
The previous commit integrated Marshal's hooks into the native courier
copying path. That should cover most cases.
One exception I've noticed that this doesn't implement is singleton
classes which have been extended by modules. I would rather we implement
that separately if we find it to be an issue.
Previously the courier only carried the core types natively, and any
other object made the whole message fall back to Marshal.dump on the
sender and Marshal.load on the receiver. That serialized everything
reachable into bytes, so a Ractor::Port or another shareable object
anywhere in the graph couldn't be sent alongside, for example, a Time.
This teaches the courier the same protocols Marshal uses: marshal_dump
with marshal_load, _dump with klass._load, and _dump_data with
_load_data, chosen in Marshal's order of preference. The hook runs on
the sender as before, but instead of being serialized its return value
is captured as an ordinary child node. Sharing, cycles and shareable
references all survive the same way they do for any other node, and
every reference to the same object resolves to the one copy on the
receiver.
Only copy uses this. Move would have to take the source apart, which a
dump hook can't do, so it still refuses these objects.
The preflight in copy_courier_supported_p doesn't walk what the hook
returns, since running it twice isn't an option, so nodes for the
payload are allocated through the growth path rather than the
reservation.
Previously rb_ractor_courier_materialize filled every non-hash node in
id order and then inserted hash entries in a separate pass in reverse
id order, on the assumption that ids are assigned depth-first so a key
always has a larger id than the hash holding it. That isn't true for a
shared key: an object reached earlier in the graph keeps its lower id,
so a hash referencing it later inserted the key while it was still an
empty shell and recorded the wrong #hash for it.
key = { 1 => 2 }
copy_key, copy_hash = Ractor.new { Ractor.receive }.send([key, { key => 1 }]).value
copy_hash[key] # => nil, should be 1
This records the order each node finishes capturing (children before
parents) and materializes in that order, so a node is only filled once
everything below it is complete. That also removes the need for the
separate hash pass. Only a cycle can still reach a node that is being
filled, which was already out of scope.
Previously we had a check that objects which were shareable, but not
frozen on a non-main Ractor would raise an exception if instance
variables were read. This prevented a non-main Ractor reading
non-shareable objects.
This changes the behaviour to forbid all writes to shareable objects
(even when unfrozen). This is done by re-purposing the frozen flag on
the object shape to reflect (FL_FROZEN || FL_SHAREABLE).
With IV writes forbidden on all shareable objects whether or not they
are frozen (with the exception of classes and modules), we can also
remove the check against reading instance variables, since no unsafe
instance variables can be set.
[Bug #22291]
None of these has a caller, a declaration in any header, or a place in
libruby's dynamic symbol table, so an extension cannot reach them either.
* rb_vm_opt_newarray_pack: left over from cb9510f539, which moved the
instruction to the _buffer form.
* rb_autoload: a wrapper around rb_autoload_str, deprecated and dropped
from intern.h in Ruby 2.3 [Feature #11664].
* rb_big_fdiv: a wrapper around rb_big_fdiv_double, unused since Bignum
stopped defining #fdiv.
* rb_io_unbuffered: a wrapper around rb_io_synchronized, which is what it
was renamed to before 1.8.0.
* rb_ractor_hooks: an accessor for cr->pub.hooks; only its sibling
rb_ractor_targeted_hooks is used.
* rb_assert_sig: a leftover debugging aid.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A singleton class belongs to the owner of the object it is attached to, and
Ractor#send(move: true) hands that object over. Without this the receiver could
define singleton methods on a moved object only when its singleton class had not
been materialized before the move. The eigenclass chain above it goes too: each
of those is attached to the class below it, so by the same rule they follow.
The move is refused when any of them holds unshareable constants or variables:
the receiver would become their owner, and so allowed to read them, while the
sender still holds the objects themselves. The check runs in the preflight pass,
before anything is mutated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Ractor id must stay unique for the life of the process, but the 32 bit
counter wraps after ~4e9 Ractors and comes back onto ids that are still in
use. That is not out of reach: a few thousand Ractor.new/s is days, not
years. The wrapped value is 0 too, which some callers read as "the main
Ractor".
There is no portable 64 bit atomic, so RUBY_ATOMIC_FETCH_ADD gives way to the
VM lock, which Ractor.new acquires a few frames later anyway. That lock is a
no-op until there is a second Ractor, which is exactly when the GVL alone
already serializes the counter, so ractor_next_id asserts the same condition
vm_insert_ractor0 does.
rb_serial_t, to match ec->ractor_id and bmethod's defined_ractor_id. Neither
struct grows: rb_ractor_pub had 4 bytes of padding after id.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
move_neutralize_source() rewrites the source's flags to
T_OBJECT | FL_FREEZE | (flags & FL_PROMOTED), which drops FL_FINALIZE
while the finalizer table entry keyed on that slot stays. The two then
disagree, and a RUBY_DEBUG build aborts at shutdown:
gc/default/default.c:4044: Assertion Failed:
rb_gc_impl_shutdown_call_finalizer_i:RB_FL_TEST(obj, FL_FINALIZE)
r = Ractor.new { Ractor.receive }
o = Object.new
ObjectSpace.define_finalizer(o, proc { |id| })
r.send(o, move: true)
Carry FL_FINALIZE over to the shell. The finalizer then runs when the
shell is collected, in the Ractor that defined it; the object rebuilt on
the other side gets fresh flags and does not inherit it, so it still runs
exactly once.
[Bug #21368]
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
When a shared root array is moved to another Ractor, it loses the T_ARRAY
type, which causes it to leak memory. For example, this script leaks memory:
r = Ractor.new { loop { Ractor.receive } }
10.times do
100_000.times do
ary = [1] * 1000
ary.instance_variable_set(:@x, []) # make ary not shareable
ary.freeze
r.send(ary, move: true)
end
puts `ps -o rss= -p #{$$}`
end
Before:
977616
1769680
2552492
3335304
4118116
4900928
5683744
6466640
7249368
8032180
After:
211588
220448
220316
220444
220700
212404
220512
220872
221200
220780
When a shared root string is moved to another Ractor, it loses the T_STRING
type, which causes it to leak memory. For example, this script leaks memory:
r = Ractor.new { loop { Ractor.receive } }
10.times do
100_000.times do
str = "x" * 4096
str.instance_variable_set(:@x, []) # make str not shareable
str.freeze
child_str = str.dup
r.send(str, move: true)
end
puts `ps -o rss= -p #{$$}`
end
Before:
470516
906212
1303752
1683808
2121396
2502312
2898164
3328072
3697172
4114284
After:
129604
135788
136748
136748
136812
136812
136876
136876
136876
136876
rb_ractor_terminate_all() interrupts the other Ractors and waits on a
condvar of its own for them to finish, dropping the VM lock for the wait
and taking it back after. That was rb_vm_cond_timedwait(), whose only
caller it was; rb_vm_cond_wait() had none at all.
Move the wait to rb_ractor_sched_wait_terminate() in thread_sched.c and
give the native thread back for its duration, as a blocking region does.
Waiting natively holds whatever native thread the caller runs on, and an
M:N caller runs on one from the shared pool -- so it would wait, holding
the pool, for Ractors that need the pool to finish. It does not happen
today, the caller being the main Ractor's main thread with a native
thread of its own, but a scheduler-blind wait in the scheduler's way is
worth keeping out of vm_sync.c.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Windows carried a second, much weaker scheduler: a single Win32 mutex as
the GVL, vm->clock for preemption, its own VM barrier over
vm->ractor.sync.barrier_*, its own Ractor wait/wakeup on a per-Ractor
condvar, no ready queue and no time slice. Keeping two made the rest of
the interpreter branch on which one was compiled in, and left Ractors on
Windows without real scheduling.
Move the scheduler out of thread_pthread.c into thread_sched.c and let
both platforms use it. thread.c now includes thread_sched.c, which
declares the scheduler <-> platform contract, includes THREAD_IMPL_SRC,
and then builds the scheduler on the primitives that file provides.
thread_pthread.c keeps only those primitives: synchronization, thread
local storage, native thread creation and stacks, the communication pipe,
the timer thread backend, the fork lock and the event hooks.
The data structures the scheduler works on move to a new thread_sched.h,
included from both platform headers, so struct rb_thread_sched,
struct rb_native_thread and struct rb_thread_sched_item are now shared.
thread_pthread_mn.c becomes thread_sched_mn.c: it is the M:N scheduler,
not a pthread detail. Most of it -- the coroutine threads, the native
thread stack pool, the timer wheel and the fd -> waiters map -- is
platform independent; only the readiness backend is not, and that section
is now marked as the seam where an epoll/kqueue split (or an IOCP backend,
which is what Windows would need for M:N) would go.
What Windows had to add is small: native_thread_interrupt(),
native_thread_default_max_cpu(), timer_thread_polling(),
timer_thread_wakeup_force(), RB_NATIVE_COND_MONOTONIC_P() and
RB_NATIVE_MUTEX_TRYLOCK_DETECTS_SELF() -- the last because a
CRITICAL_SECTION is recursive, so trylock cannot back a "somebody holds
this" assertion. It sets USE_MN_THREADS to 0 and supplies the stubs
thread_sched_mn.c provides elsewhere, so every thread there is dedicated,
as it already was.
With one scheduler the platform branches go away: all 16 uses of
RUBY_THREAD_PTHREAD_H / RUBY_THREAD_WIN32_H are removed from vm_core.h,
ractor.c, ractor_core.h, ractor_sync.c, vm_sync.c, vm.c and thread.c.
ractor_check_blocking() went with them -- it only ever fed the win32
scheduler's vm->ractor.blocking_cnt -- and so did USE_VM_CLOCK, now that
the common time slice drives preemption on Windows too.
Verified on both platforms. Linux: 11 rounds of make test-all (36236
tests) and btest (2066 tests) with no failures, plus 474 runs of a
scheduler and Ractor stress soak. Windows: no test fails that does not
also fail on the same tree without this change, btest passes, and two
40-minute soaks of 438 and 460 runs are clean. Ractors now run in
parallel there -- 3.6x on four Ractors, where before they could not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
coroutine_thread_terminated() leaves the Ractor's living set before handing
the scheduler slot over, then keeps using th for the rest of the function:
thread_sched_to_dead_common() reads th->nt and deregisters th from the
barrier, and the epilogue then stashes th->nt and clears th->sched.context.
Off the set th is unreachable, and once deregistered no barrier waits for
it, so a sweep in another Ractor can free it mid-epilogue. The epilogue
reads freed memory, and co_start() dereferences the NULL it finds in
tctx->nt -- SEGV at 0xc8, the offset of dead_co in struct rb_native_thread.
The removal cannot simply move after the handoff: the GC's root scan walks
r->threads.set without the Ractor lock (relying on the barrier), so the
unlink must stay where no walker can run -- while th is still registered
and before a successor is designated. Nor can the th accesses move before
the removal: deregistration itself writes th's running-list node.
Instead, keep the removal where it is and give the dying thread an explicit
root: r->threads.dying_th, marked by the root scan exactly like a set
member, set just before the removal and cleared inside the scheduler-lock
section after the epilogue's last use of th. Clearing under the lock
serialises the epilogue against its successor -- even a dedicated one woken
by to_dead_common() first runs thread_sched_to_running(), which takes the
same lock -- so successive epilogues cannot overwrite each other's slot,
and the successor cannot publish threads.running_ec before the dying
thread has cleared it. Nothing reads th after the clear: the final unlock
avoids the debug-logging wrapper (which reads th->serial), and whether the
designated successor needs the Ractor enqueued is decided while still
holding the lock, since a dedicated successor may run -- and die, freeing
itself -- the moment the lock is released. The atfork paths reset the slot
so a fork taken mid-epilogue cannot leak a stale root into the child. The
slot is accessed with atomic ops: the root scan may read it while another
Ractor's dying thread writes it.
Also initialise tctx->nt, which native_thread_create_shared() left holding
whatever ruby_xmalloc() returned.
Not addressed, pre-existing on master: rb_postponed_job_trigger_for_ractor()
can copy threads.running_ec and dereference it arbitrarily later (the new
root narrows this but cannot protect an already-copied pointer), and a
freshly created dedicated native thread publishes its EC via
ruby_thread_set_native() before taking the scheduler lock.
Two Ractors looping over Thread.new and GC.start crash master in about
three seconds (19 of 20 runs); an ASAN build reports the heap-use-after-free
directly, at the read of th->nt in coroutine_thread_terminated(), freed by
gc_sweep via rb_thread_free_body. With this change ASAN is clean (0/10 vs
7/10 interleaved), as are the reproducer, s7_nested, both branches of the
epilogue under stress, a Ractor/GC soak, btest and test-all on release and
RUBY_DEBUG=1 builds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rb_ractor_blocking_threads_inc/dec took the VM lock on every blocking
region boundary of a ractor's last runnable thread -- for a 1-thread
ractor, twice per IO operation -- to maintain vm->ractor.blocking_cnt.
That counter is only consumed by the win32 scheduler, so the pthread
build now skips those VM-lock sections; the per-ractor
threads.blocking_cnt and the ractor creation/exit protocol stay.
Ractor#inspect's status becomes running/terminated only ("blocking" is
gone). The finer states were really the win32 scheduler's
in-blocking-region flag leaking into inspect (under M:N, receive/sleep/
IO waits always showed "running" anyway, and the skip above stops the
remaining flips on pthread entirely).
R ractors x 1 thread, pipe write/read round-trips each, 16-HT machine
(Ryzen 9 5900HX), RUBY_MN_THREADS=1, best of 3, mean of 2 alternating
same-tree runs:
before after
1R 342k 356k
4R 877k 1016k
8R 884k 976k
16R 835k 884k
24R 793k 839k
The remaining ceiling is the scheduler-lock contention addressed by
"thread: keep context switches off the scheduler lock"; combined, 16R
reaches 3.6M rt/s.
Suggested-by: Koichi Sasada <ko1@atdot.net>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
thread_sched_setup_running_threads runs on every context switch (park,
resume, and both ends of every blocking region) and took the global
ractor.sched.lock to maintain the running_threads and timeslice lists
and running_cnt. With many ractors switching concurrently this lock is
the scaling ceiling.
Replace all three with ntlist, a per-location registration of who runs
where, and no shared counter:
- a thread on a shared nt registers in that nt's running_th, under the
nt's own lock -- uncontended, no cross-core traffic
- a dedicated nt goes on ntlist.running_dnts while its thread runs (the
switch rate of dedicated threads is what it always was; only the lock
shrank), so the fast path writes no shared cache line at all
The consumers walk the registrations: the barrier's interrupt walk also
counts them into a snapshot (barrier_running_cnt) and stamps each nt
with the barrier serial; a stamped nt deregistering during that barrier
decrements the snapshot under sched.lock, so the completion check stays
O(1) however many runners there are.
The timeslice list becomes timeslice.scheds, the scheds whose readyq
holds waiters. The enq that fills an empty readyq (and the add path
taking a turn with waiters already queued) links the sched; the timer
thread interrupts each listed sched's running thread and prunes entries
whose readyq drained. A steadily contended sched stays linked, so the
switch path never touches the list, and idle scheds are not walked at
all. The timer takes each sched's lock by trylock, because the
switchers nest sched.lock -> timeslice.lock; pruning is lazy, so
ractor_free delists the dying ractor's sched.
Barrier pairing needs no fences: the barrier sets barrier_is_waiting
and then walks the registrations under their locks; a switch moves its
registration under the same lock and then reads the flag. A walk that
missed a registration ran before that registration's critical section,
so its flag store is visible to that switcher. Nothing may start
running while the flag stands -- an interrupt flag alone would let a
resuming thread run, and allocate, inside the stop-the-world section --
so the add path takes its registration back and waits for
rb_ractor_sched_barrier_end's broadcast. While a scan holds a
running_th lock the registered thread cannot finish parking, so it
cannot die under the scan. An snt turned dedicated under
rb_thread_lock_native_thread has no creation-time running_thread, so
the running_dnts registration writes nt->running_thread itself.
R ractor pairs ping-ponging via ports (msgs/sec) and R ractors doing
pipe write/read round-trips (rt/sec), 16-HT machine (Ryzen 9 5900HX),
mean of 2 alternating same-tree runs:
before after
port pairs 8 503k 815k +62%
port pairs 32 453k 799k +76%
pipe self 4R 837k 940k +12%
pipe self 16R 820k 898k +10%
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ractor::Port#receive, Ractor.receive and Ractor.select gain a `timeout:`
keyword and return nil when it passes.
The wait itself stays where it is: rb_ractor_sched_wait() keeps parking the
thread in the thread scheduler, so an M:N thread still hands its native
thread back instead of becoming dedicated. How the deadline is taken
depends on which kind of thread waits:
A dedicated native thread parks on its own condvar, so it takes the deadline
there, the way native_cond_sleep() does. Nothing else is involved: the
condvar it waits on is the one a send already signals.
An M:N thread has no condvar of its own, so its deadline is armed on the
timer thread as a timeout-only wheel entry, and the timer thread wakes it
through thread_sched_to_ready_common(), which is exactly how
rb_ractor_sched_wakeup() wakes it for a send. That gives such a waiter two
wakers, so rb_ractor_sched_wakeup() now takes an armed timeout back before
waking, skips a thread that a fired timeout already made runnable, and bumps
the scheduler event serial so a timeout that has not fired cannot wake it a
second time.
`timeout: 0` is a poll: it never blocks, converts nothing and reads no clock,
so it costs what a receive of a waiting message costs. It still delivers the
incoming queue first, so a message another thread just sent is not missed.
A timeout that expires leaves the waiter on the Ractor's waiter list, as a
spurious wakeup does, and ractor_wait_receive() still delivers pending
messages before it reports the timeout, so a message that arrived just before
the deadline is not lost.
Both kinds of wait exist on every pthread platform, including builds without
the timer wheel (USE_MN_THREADS == 0), where every thread is dedicated. On
win32 the wait is a condvar wait, which takes the timeout directly.
A Ractor whose block ends while one of its own threads is still running took
a full second to terminate: rb_thread_terminate_all() sleeps for a second per
round and relies on the last sub-thread to wake it, but that wakeup was sent
only when the Ractor's main thread already had THREAD_KILLED, which
thread_start_func_2 sets after rb_thread_terminate_all() returns. For the
main Ractor rb_ec_cleanup() sets it beforehand, which is why only Ractors
were affected.
Setting THREAD_KILLED earlier is not an option here: thread_sched_switch()
reads it as to_dead, promising the M:N scheduler that the coroutine is never
resumed, and this thread still parks and resumes inside the wait. So mark
the wait itself instead: rb_thread_terminate_all() sets threads.terminating
before it waits, and the exiting thread wakes `main` on that. Nothing clears
it afterwards, because no thread of that Ractor runs again; the Ractor that
survives a fork gets it reset along with the rest of the thread set.
A port's queue lives in the receiving Ractor's table keyed by the port id, so
nothing ties it to the Ractor::Port object: when the port becomes unreachable
the queue stays, and so does every message left in it. Nobody can receive
them -- receiving needs the port -- but they are still GC roots.
A message reaching a Ractor makes that permanent. Port#<< of a port is the
natural shape of a credit token, and a token nobody takes holds the sender's
port, which marks the sender's Ractor object, which keeps its objspace in
zombie_objspaces for the life of the process:
200.times { port = Ractor::Port.new
Ractor.new(port) { |p| p << Ractor::Port.new; nil }.join }
6.times { GC.start }
ObjectSpace.each_object(Ractor).count # 201 before, 1 after
Mark and sweep the table itself. ractor_port_mark notes the ids a reachable
port still carries, and rb_ractor_finish_marking drops the queues nobody
noted, which is where the ractor-local keys are already reaped for the same
reason. Both run only where an unmarked port really is dead: after a mark
that covered every objspace -- a global GC or a full single-objspace one --
which is also when the world is stopped enough to read the table without the
owner's lock. Hence the new argument to rb_ractor_finish_marking: a minor
mark leaves live old objects unmarked.
The single-objspace question is asked first because mmtk marks from its GC
worker threads, which cannot reach the VM to answer the other one. Those
threads race on the flag, but they all store the same value and the reap
reads it once marking is over.
A queue starts alive so that one created between its port being marked and
the reap, which incremental marking allows, is not taken for an orphan. It
costs the queue one more cycle before it goes.
2000 rounds, dropping one message each, KB of RSS retained per round:
payload before after
Integer 13.0 13.3
Ractor::Port leaks 14.3
shareable 10KB 89.1 62.5
The last row is not retention: the payload is collected (0 of 500 strings
survive), but the reap only runs in a global GC, so between two of them the
dropped messages hold pages. Collecting every 20 rounds instead brings it to
1.5 KB.
The reap runs from rb_ractor_finish_marking, which only the default GC calls
-- mmtk has never called it, so the ractor-local keys it also frees are not
freed there either. Under mmtk a dropped message still holds its port, so
the test for that is a test-all case that omits itself, next to the other
tests for per-Ractor objspace semantics.
ractor-pipeline, which sends credit tokens this way, over 60 pipelines:
455MB growing by 5.6MB each to 272MB growing by 1.5MB, with the library
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The root scan marked r->sync.legacy for every zombie objspace, deliberately
"without depending on wrapper reachability". That makes the value a GC root,
and a Ractor whose return value can reach the Ractor object roots itself:
Ractor.new { Ractor.current }.join
Nothing else names either one, but the value is a root, the value reaches the
wrapper, so the wrapper never dies, so ractor_free never runs,
rb_gc_objspace_disown is never called, and the objspace stays in
zombie_objspaces for the life of the process. The count never returning to
zero also holds rb_gc_single_objspace_p() false forever, which is the gate on
page_pool_reclaim, so the page pool stops returning arenas to the OS. A
Ractor::Port, an array, an ivar -- any path from the value back to the wrapper
does it; 10000 rounds of the line above leave 10001 live Ractors.
The value is of use only to whoever can still call Ractor#value, and that
means holding the wrapper, so mark it as the wrapper's child instead. A
self-referential pair is then ordinary garbage, and the transitive case (one
Ractor's value naming another) falls out of normal tracing rather than needing
a pass that iterates to a fixpoint.
Only during a global GC: it stops the world and marks every objspace together,
which is what lets the shareable wrapper reach an unshareable value. A local
GC must not, and never did -- the zombie scan this replaces also ran only in a
global GC.
300 rounds of the line above 10001 -> 1 live Ractors
8000 rounds with a dropped message 2.3 GB -> 216 MB, and it now plateaus
(287 MB at 20000, 288 MB at 40000)
Verified with RUBY_DEBUG=1: bootstraptest/test_ractor.rb, the GC and Ractor
unit tests, GC.verify_internal_consistency after holding 100 terminated
Ractors across GC.compact (their values, including self-references, all still
readable), and the same under GC.stress.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rb_ractor_mark_in_flight_for_single_objspace marked r->sync.legacy and
nothing else once the pin machinery went away, which is what
rb_ractor_mark_terminated_join_value next to it already does -- two functions
over the same field, and the surviving one names it correctly. Call that one
from the single-objspace branch and drop the other.
It also pins, which the plain rb_gc_mark did not. sync.legacy is a C-struct
slot with no compaction update hook anywhere, so pinning is the treatment the
field needs; the comment on rb_ractor_mark_terminated_join_value says so.
The name had stopped describing the function: "in flight" meant the sending
basket and the pinned copy snapshot it used to mark, and both are gone. The
word is also the GC-cycle sense elsewhere in this code, which is why the
wrapper it sat next to is now rb_gc_rest.
Only the ractor.c translation unit used it -- ractor.c includes
ractor_sync.c -- so the ractor_core.h declaration goes too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rb_gc_finish_in_flight_gc is a one-line wrapper around rb_gc_impl_gc_rest,
and every other layer of the same call already says "gc rest": gc_rest in
gc/default/default.c, rb_gc_impl_gc_rest across the modular GC boundary. The
wrapper exists only because ractor.c cannot call rb_gc_impl_* itself, so it
should not rename the operation on the way out.
"in flight" also means a message in transit elsewhere in the Ractor code
(rb_ractor_repin_in_flight, rb_ractor_mark_in_flight_for_single_objspace),
which is a second sense for the word in the subsystem where both GC cycles
and messages are the subject.
The comment above the definition described the one caller; the caller already
carries that, so drop it there and keep the reason where the call is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
courier_alloc_node bumped c->count and then wrote the placeholder fields, so
for those few stores the mark walk over nodes[0, count) covered a slot that
had not been written yet. courier_alloc_ref already had the other order and
says why; make the two match.
Not reachable as written: nothing between the bump and the stores allocates
or reaches a safepoint, so no collection can start there, and the growth
above it happens before the bump. The order is what keeps that true without
having to check.
Found by the Copilot review on ruby/ruby#18392.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The off-heap courier started out as the move path's own structure, so every
part of it was named move_*. Copy now travels in the same structure and
through the same capture walk, with courier_build.copy selecting whether the
source is read or taken apart, which left the prefix claiming something
untrue: move_capture is what a copy goes through too.
Rename the shared machinery to courier_*: the struct and its build/
materialize/mark/free entry points, the node kinds, the capture and
materialize walks, and the array allocators. The two builders become
rb_ractor_courier_build_copy and rb_ractor_courier_build_move so the pair
reads as one family. What is still move-only keeps the move_ prefix --
move_preflight (copy has its own walk, copy_courier_supported_p) and
move_neutralize_source, which is exactly the step copy skips. In re.c the
MatchData transfer helpers become rb_match_blob_*, matching the neighbouring
rb_backtrace_blob_* it sits beside in the same node.
No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two costs on the courier path that the on-heap snapshot did not have.
The node and ref arrays were grown by doubling while the graph was captured,
and growing them is not a realloc: the courier is a GC root while it is being
built, so a realloc would leave the old pointer live across a window where it
may already have been freed. Copying into a fresh array instead is correct
but not free, and it also made move_alloc_node/move_alloc_ref too big to
inline, which costs once per captured object.
Both walks that run before capture -- copy_courier_supported_p and
move_preflight -- already visit the whole graph, so let them count what
capture will allocate: one node per distinct unshareable object, one ref per
occurrence of a shareable one. move_courier_reserve then sizes both arrays
once and the growth path never runs (verified: it does not fire for any of
the benchmark payloads). It stays in place, out of line, in case a count
ever comes out short.
Materializing an array pushed its elements one at a time, through the
capacity check, although the length is known and the shell was allocated with
it. Set the length once and write the slots.
Message cost, 20k messages per trial, 9 trials, median us per message,
measured against master:
40B String 1.40 -> 1.04 4KB String 17.28 -> 8.66
bare Object 1.15 -> 1.05 20-key Hash 12.35 -> 7.61
nested 5.95 -> 5.48 100-elem Integer array 5.70 -> 6.46
The array of immediates is the one shape still behind master. Copy walks the
graph twice there (the preflight, then the capture) where the on-heap snapshot
walked it once; dropping the preflight would mean capture bailing out on an
unsupported type and falling back to Marshal, which also gives up the sizing
above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A basket is rooted by whoever holds it, and nothing else: the port queue it
waits on, or -- while it is on no queue -- its holder's off_queue_baskets
list. rb_ractor_sync grows that list and the basket a second ccan_list_node,
so it can sit on one or the other.
send alloc the basket, put it on the sender's list, then build the
payload into it: rb_ractor_{move,copy}_courier_build publish the
courier into the basket right after ZALLOC, so even a half-built
one is rooted. A move needs this -- it husks the sources as it
captures them, so what the courier has collected loses its other
root as the walk proceeds.
enqueue off the sender's list, onto the receiver's queue
receive off the queue, onto the receiver's list until the basket is freed
The publish is the handover of ownership too: when the capture raises, the
builder re-raises and leaves the courier where it is, and the basket frees it
exactly once as ractor_basket_new unwinds. (The raise path was pointed out by
the Copilot review on ruby/ruby#18392.)
Before this, a courier was the one payload ractor_basket_mark skipped: it
left the whole job to vm->ractor.move_courier_registry, which only a global
GC walks. Between the send and the next global GC nothing rooted the
payload, and a shareable object reachable from the courier alone was
collected -- a dynamic Symbol loses its fstr, a frozen Array its elements,
and reading it crashes. A shareable String survived only by being an
fstring, rooted elsewhere. That is already broken in master for move; it
becomes reachable from an ordinary send once copy travels by courier. Two
reproducers, both crashing unpatched master and fine on 4.0.2, now pass:
ports = 5.times.map do |i|
port = Ractor::Port.new
r = Ractor.new(port, i) { |p, n| p.send([:"sym_#{n}"], move: true) }
r.join
port
end
5.times { GC.start }
ports.each { |p| puts p.receive[0].inspect }
(the symbol has to be interpolated: a literal one is held by the iseq) The
second sends Ractor.make_shareable([1, "child"]) the same way.
ractor_sync_mark walks the list next to the queues it already walks, so this
needs no new root pass, and it walks both in every collection. "What a
courier holds is shareable, and only a global GC frees a shareable" does not
hold: pinned_roots_mark, which roots a shareable from its page bit, is
skipped once the process is back to a single Ractor
(rb_gc_single_objspace_p), and then an ordinary local GC frees one that
nothing else names. A payload in flight is named by its basket and nothing
else. The list needs no sync lock even so -- only its owner touches it,
unlike the queues a foreign sender writes. The one caller that does skip it
is the single-objspace root pass, where ractor_mark_unshareable_parts has
just walked the same baskets through ractor_sync_mark.
This retires the registry, its lock, its fork re-initialization and
rb_ractor_move_courier_registry_mark. Walking the registry in every
collection instead of the basket would not do: a receiver's local GC, which
materialize's own allocation can trigger, would then walk a courier another
Ractor is inside move_alloc_ref for -- observed as a crash.
move_alloc_node and move_alloc_ref grow by swapping in a fresh array rather
than realloc, for the same reason the nodes are initialized mark-safe: the
courier is a GC root while it is built, and a realloc can leave c->nodes
pointing at a block it has already freed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Materializing a string node called rb_enc_str_new, which copies the bytes a
second time: once into the courier when the node was built, once out of it
when the message arrives. Give the buffer to the String instead.
rb_str_new_owned takes an xmalloc'd buffer as a heap String's body, the way
rb_str_new_static does for a static one, except the String owns it and frees
it like any other. The courier's node hands its pointer over and forgets it,
so courier_free no longer frees what the String now owns.
Message cost, 20k messages per trial, 9 trials, median us per message:
payload 4.0.2 master before here
1KB 2.97 4.11 3.4 3.10
4KB 3.52 8.75 9.4 4.22
8KB 4.26 12.39 12.0 5.08
16KB 5.43 16.24 16.5 6.53
master pulls away from 4.0.2 as the payload grows; this tracks it within
about 20% across the range. Small strings are embedded and never had a
buffer, so they are unchanged.
A heap String is freed by its size (STR_HEAP_SIZE = capa + terminator), so the
node carries the capacity of the allocation rather than the bytes in use, and
reserves the encoding's terminator length instead of a single NUL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With MatchData and an exception's backtrace on the courier, every copy
payload is off-heap: the courier carries the core types and anything else
travels as marshaled bytes. The sender-resident snapshot is gone, and with
it all the machinery that kept one alive across a global GC.
MatchData: rb_match_move_dump takes a release_source flag, since only a move
takes the source's onig region and char_offset apart.
Backtrace: rb_backtrace_blob_dump / _load / _mark copy the frames into an
off-heap blob. A frame only references shareable iseq and method-entry
imemos, so the blob can carry them as they are; it has no compaction update
hook, so the mark pins them (rb_gc_mark, not _movable).
Removed, all of it unreachable once no payload lives in the sender's heap:
rb_ractor_t pin_capture / _cnt / _capa, sending_basket,
gen_fields_capturing, ractor_pin_capture_push
basket p.pinned / p.pinned_cnt, ractor_basket_repin_in_flight,
rb_ractor_repin_in_flight and its queue walks
receiver struct ractor_materialize_frame, ec->materialize_frames and
its mark loop, sync.materializing_copies,
rb_ractor_materializing_p
gc rb_gc_pin_in_flight_message and the whole
rb_gc_impl_pin_in_flight_message hook (default, mmtk, the
modular function table), plus the two verifier relaxations
that existed for a snapshot being materialized
shref bits stay: the write barrier still records them for other reasons.
ractor_copy_native_try and copy_enter stay too, since
Ractor.make_shareable(obj, copy: true) still deep-copies within one objspace.
Message cost, 20k messages per trial, 9 trials, median us per message:
4.0.2 master here
4KB String 3.8 19.4 9.4
20-key Hash 3.3 11.9 7.5
nested Hash/Array/Str 5.3 7.1 5.1
100-elem Integer Array 4.4 5.7 6.9
40B String, Object same across all three
master is well behind 4.0.2 on multi-KB payloads; this recovers about half
of that and puts nested graphs back at 4.0.2's level.
Known issue, already in master and not introduced here: the courier's
registry mark does not traverse a shareable payload, so its children are
collected if a global GC runs while the message is in flight. Unpatched
master crashes on both repros (a dynamic Symbol loses its fstr, a frozen
Array loses its elements) and 4.0.2 does not. It only shows up for move
today; routing copy through the courier makes it reachable from a common
path, so it wants fixing before this ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 took the marshaled copy payload off-heap; this does the same for the
natively copied one, so an unreceived copy no longer pins a page of the
sender's heap for any payload type.
move_capture gains a copy mode: it reads the sources instead of taking them
apart (no husk, no buffer hand-over, no freeing the source's internals), and
copy_courier_supported_p decides up front whether the courier can carry the
whole graph. MatchData, IO, any other T_DATA and a singleton class are
rejected there and keep using the on-heap snapshot path, which handles or
rejects them exactly as before.
Retention with 3000 send-and-never-receive rounds (KB per sending Ractor,
before -> after):
String 79.5 -> 14.0 Array 79.5 -> 14.0
Object 79.5 -> 14.0 Hash 79.5 -> 14.0
14 KB is what a Ractor that sends nothing retains, and that part plateaus.
A courier is an in-process format, so a shareable payload can be embedded as
the VALUE itself rather than costing a whole move_node: child slots now hold
either a node id or, with the top bit set, an index into a compact c->refs
array that the registry marks. Without that, an array of immediates paid a
72-byte node, a hash lookup and an insert per element, and the message path
cost 2.5x what the native copier did (it skipped shareable children).
Message throughput (20k messages, n=9, interleaved, median us/message):
100-element Integer array 5.1 -> 4.7
4KB String 16.6 -> 9.3 (one memcpy, not two copies)
20-key Hash 12.0 -> 9.4
nested Hash/Array/String 10.1 -> 7.2
40B String, bare Object unchanged
It also restores a behaviour master lost: a String or Array subclass now
survives the copy again (4.0.2 copies with #clone and keeps the class;
master's native copier builds a base-class object). MyStr < String arrives
as MyStr, as it does under Marshal and #clone.
move_capture keeps testing the dedup table before shareability: move husks
each source as it goes, and a husk is a frozen field-less object that
rb_ractor_shareable_p answers true for, so the other order embedded the husk
instead of resolving a second occurrence to the first one's node
(bootstraptest/test_ractor.rb:661). An immediate skips the lookup, since
only captured objects are ever inserted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ractor_mark_unshareable_parts asserts that its Ractor is
not terminated. This is only relevant in the multi-objspace
root scan, where a terminated Ractor has left the set and
zombie_objspaces keeps its possessions alive.
With a single objspace the wrapper mark function also calls
this helper, and a terminated Ractor wrapper stays reachable
on the heap.
MMTk hits this assertion:
ractor.c:259: Assertion Failed: ractor_mark_unshareable_parts:!rb_ractor_status_p(r, ractor_terminated)
Ractor#join waits on a monitor port, and the exit token was sent from
ractor_notify_exit as the Ractor's block ends. Ractor#value then has to wait
for ractor_terminated on its own -- vm_remove_ractor retires the objspace the
joiner is about to absorb -- and it does so by spinning on rb_thread_schedule().
Now that the dying thread collects its own objspace post-mortem, that spin
covers a whole local GC: the joiner burns a core for as long as the child's
final collection takes, which grows with the child's heap.
Split the notification: the value is published at atexit as before, but the
tokens go out after the collection, from rb_ractor_postmortem() so that the
order of the two steps is stated in one place. What the joiner spins over is
then only the scheduler handoff and vm_remove_ractor, which is constant.
The send cannot go through ractor_basket_new there: thread_cleanup_func has
already dropped the EC's VM stack, and EC_PUSH_TAG reads ec->cfp when ZJIT is
built in. A token is a static Symbol, so ractor_basket_new_ref builds its
basket without a tag or any payload preparation.
Post-join spin, by the number of objects the child allocated:
0: 13us -> 3us
10000: 961us -> 3us
100000: 5211us -> 199us (the remainder is the absorb itself)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every Ractor spawn eagerly built three IO objects (its $stdin, $stdout
and $stderr) under the VM lock in the setup section, though most
Ractors never touch stdio. Build them in rb_ractor_stdin and friends
on first access instead: the accessor runs in the owning Ractor, so
the objects still land in its own objspace. The slots stay 0 until
then -- marking 0 is a no-op, and the setters are unchanged.
A Ractor that never uses stdio now leaves nothing for its final
collection to finalize, which composes with the previous two commits:
the death of Ractor.new { :ok } releases every page it allocated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The retire GC at notify_exit could not reclaim the terminating
thread's scaffolding -- the Thread and Fiber wrappers, the ThreadGroup,
the Ractor-local stdio, whatever the machine stack still references --
because at that point they were all still rooted. Those ~11 objects
kept ~2 heap pages alive per dead Ractor, merged into the inheritor to
wait for its next major GC, so both a spawn-and-join loop and a
fire-and-forget loop grew the heap by two pages per Ractor with
nothing reclaiming them.
Instead, run the final collection at the tail of thread_start_func_2,
just before the scheduler handoff. The thread still holds the GVL, so
the collection runs lock-free like any local GC (a concurrent global
GC waits for its safepoint), and vm_remove_ractor keeps its light
lock-only section. r->postmortem makes rb_ractor_mark_local_roots
root only what must survive the death -- the join value
(r->sync.legacy) and the rb_gc_register_mark_object pins -- and the
zombie_objspaces entry recorded later measures exactly the pages the
joiner will inherit. While the Ractor is still in vm->ractor.set,
rb_vm_mark and ractor_mark keep the Ractor object itself and its
default port alive as before; everything they mark is foreign to or
pinned by this collection.
Two of the swept wrappers are the dying thread's own: its Thread (the
rest of the collection resolves GET_EC through ec->thread_ptr) and its
root Fiber (whose struct embeds that ec). thread_free and fiber_free
recognize "the struct I am standing on" (ec == the current ec), keep
the struct, and leave an in-band mark (th->self = 0; cont.self = 0,
which already means "no wrapper" on the never-wrapped path). Found as
a 1/30 heap corruption, pinned by ASAN. The collecting thread
captures those two pointers right after the collection, while its GVL
still excludes any global GC, and frees exactly the captured structs
at its very last step -- after vm_remove_ractor unlinks the Ractor,
anything it did not sweep may already be freed by a concurrent global
GC, so the epilogue reads no shared memory at all. Those frees would
resolve their objspace through the TLS ec -- which sits inside the
fiber struct being freed and leads to the retired Ractor -- so
rb_ractor_postmortem_free severs the TLS ec first (the MN epilogue
already had) and they fall back to the main objspace.
While a thread-event hook is registered the postmortem flag stays off
and the collection keeps the ordinary retire roots: callbacks may
retain any Ractor's Thread object (rb_internal_thread_event_hook_t),
and such references are invisible to the reduced roots. The check
takes the hook rwlock, so it happens-after any completed registration;
a hook registered after it gets no event from this thread --
rb_thread_execute_hooks skips a thread whose wrapper is gone
(th->self == 0) -- so it never receives what the collection swept.
On platforms without hook support (win32, none) the check reports
false.
The collection runs after thread_cleanup_func, so the dying thread's
machine stack is already torn down: the root scan skips its machine
context (rb_gc_impl_during_postmortem_p), which holds nothing the
reduced roots do not already cover and faults when scanned on some
platforms (windows-arm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
move_preflight did not recurse into the regexp and the matched
string of a MatchData. A Regexp subclass is not shareable, so it
passed preflight. move_capture then mutated earlier sources in
the graph before the raise. This left the sources destroyed.
Recurse into rm->regexp and rm->str in preflight. This mirrors
move_capture. The raise now happens before any mutation.
The Ractor root scan marked each thread's ec directly and then marked the
wrapper of the fiber that ec lives in. Both reach rb_execution_context_mark,
so every thread's machine stack was scanned conservatively twice per
collection -- with 5000 threads, 750270 scans where 362326 were needed.
Marking the fiber wrapper is enough: th->ec is &fiber->cont.saved_ec (thread_mark
asserts as much), the wrapper goes on the mark stack during the root scan, and
the stack drains before anything is swept. Only when there is no wrapper yet,
mid-creation or teardown, does the ec still need marking of its own.
benchmark/vm_thread_pass_flood.rb, which floods 5000 threads:
collections 157 157 (unchanged)
time in GC 0.84s -> 0.49s
thread creation 1.17s -> 0.81s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A move courier carries an IO by taking over its rb_io, so the source IO is
already a RactorMovedObject and nobody else can close the descriptor. When
the courier is freed without being delivered -- Ractor#send raising because
the port is closed is enough -- the branch for MOVE_KIND_IO did nothing and
the fd was leaked. Sending 20 IOs to a closed Ractor leaked 20 descriptors.
Finalize the rb_io instead, which closes the fd and frees the buffers. A
delivered IO already left fptr == NULL, so it is untouched.
Found by an automated review of the branch.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
RACTOR_CHECK_MODE stamped every object with the id of the Ractor that
allocated it (a 4 byte suffix on every slot), and re-stamped whole object
graphs whenever a value crossed a Ractor boundary. Per-Ractor objspaces
already record the owner: an object's page knows its objspace, so
rb_gc_obj_foreign_p() answers "is this another Ractor's object" with one
load, and the id, its suffix and the re-stamping traversals all go away.
Deriving it is also more accurate. The stamped id was never rewritten
when a zombie objspace was absorbed, which is what ractor_reset_belonging()
had to paper over by re-stamping received and inherited values. Page
ownership moves with the absorb, so nothing is left stale.
thread.c stamped the child's main Thread wrapper with the child's id
because upstream allocates that wrapper in the parent's objspace. This
branch already allocates it in the child's objspace, so the site goes too.
With ractor_reset_belonging() gone, no rb_obj_traverse() caller has an
enter function that descends into arbitrary types: make_shareable raises
on T_FILE and shareable_p stops there, so obj_traverse_i() can no longer
reach T_FILE and its case goes as well.
GC impls that keep a single objspace (mmtk) cannot tell owners apart and
report every object as their own, so the check degrades to a no-op there;
in exchange, dropping the suffix re-enables mmtk's ZJIT allocation
fastpath.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Matt Valentine-House