Windows carried a second, much weaker scheduler: a single Win32 mutex as
the GVL, vm->clock for preemption, its own VM barrier over
vm->ractor.sync.barrier_*, its own Ractor wait/wakeup on a per-Ractor
condvar, no ready queue and no time slice. Keeping two made the rest of
the interpreter branch on which one was compiled in, and left Ractors on
Windows without real scheduling.
Move the scheduler out of thread_pthread.c into thread_sched.c and let
both platforms use it. thread.c now includes thread_sched.c, which
declares the scheduler <-> platform contract, includes THREAD_IMPL_SRC,
and then builds the scheduler on the primitives that file provides.
thread_pthread.c keeps only those primitives: synchronization, thread
local storage, native thread creation and stacks, the communication pipe,
the timer thread backend, the fork lock and the event hooks.
The data structures the scheduler works on move to a new thread_sched.h,
included from both platform headers, so struct rb_thread_sched,
struct rb_native_thread and struct rb_thread_sched_item are now shared.
thread_pthread_mn.c becomes thread_sched_mn.c: it is the M:N scheduler,
not a pthread detail. Most of it -- the coroutine threads, the native
thread stack pool, the timer wheel and the fd -> waiters map -- is
platform independent; only the readiness backend is not, and that section
is now marked as the seam where an epoll/kqueue split (or an IOCP backend,
which is what Windows would need for M:N) would go.
What Windows had to add is small: native_thread_interrupt(),
native_thread_default_max_cpu(), timer_thread_polling(),
timer_thread_wakeup_force(), RB_NATIVE_COND_MONOTONIC_P() and
RB_NATIVE_MUTEX_TRYLOCK_DETECTS_SELF() -- the last because a
CRITICAL_SECTION is recursive, so trylock cannot back a "somebody holds
this" assertion. It sets USE_MN_THREADS to 0 and supplies the stubs
thread_sched_mn.c provides elsewhere, so every thread there is dedicated,
as it already was.
With one scheduler the platform branches go away: all 16 uses of
RUBY_THREAD_PTHREAD_H / RUBY_THREAD_WIN32_H are removed from vm_core.h,
ractor.c, ractor_core.h, ractor_sync.c, vm_sync.c, vm.c and thread.c.
ractor_check_blocking() went with them -- it only ever fed the win32
scheduler's vm->ractor.blocking_cnt -- and so did USE_VM_CLOCK, now that
the common time slice drives preemption on Windows too.
Verified on both platforms. Linux: 11 rounds of make test-all (36236
tests) and btest (2066 tests) with no failures, plus 474 runs of a
scheduler and Ractor stress soak. Windows: no test fails that does not
also fail on the same tree without this change, btest passes, and two
40-minute soaks of 438 and 460 runs are clean. Ractors now run in
parallel there -- 3.6x on four Ractors, where before they could not.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
coroutine_thread_terminated() leaves the Ractor's living set before handing
the scheduler slot over, then keeps using th for the rest of the function:
thread_sched_to_dead_common() reads th->nt and deregisters th from the
barrier, and the epilogue then stashes th->nt and clears th->sched.context.
Off the set th is unreachable, and once deregistered no barrier waits for
it, so a sweep in another Ractor can free it mid-epilogue. The epilogue
reads freed memory, and co_start() dereferences the NULL it finds in
tctx->nt -- SEGV at 0xc8, the offset of dead_co in struct rb_native_thread.
The removal cannot simply move after the handoff: the GC's root scan walks
r->threads.set without the Ractor lock (relying on the barrier), so the
unlink must stay where no walker can run -- while th is still registered
and before a successor is designated. Nor can the th accesses move before
the removal: deregistration itself writes th's running-list node.
Instead, keep the removal where it is and give the dying thread an explicit
root: r->threads.dying_th, marked by the root scan exactly like a set
member, set just before the removal and cleared inside the scheduler-lock
section after the epilogue's last use of th. Clearing under the lock
serialises the epilogue against its successor -- even a dedicated one woken
by to_dead_common() first runs thread_sched_to_running(), which takes the
same lock -- so successive epilogues cannot overwrite each other's slot,
and the successor cannot publish threads.running_ec before the dying
thread has cleared it. Nothing reads th after the clear: the final unlock
avoids the debug-logging wrapper (which reads th->serial), and whether the
designated successor needs the Ractor enqueued is decided while still
holding the lock, since a dedicated successor may run -- and die, freeing
itself -- the moment the lock is released. The atfork paths reset the slot
so a fork taken mid-epilogue cannot leak a stale root into the child. The
slot is accessed with atomic ops: the root scan may read it while another
Ractor's dying thread writes it.
Also initialise tctx->nt, which native_thread_create_shared() left holding
whatever ruby_xmalloc() returned.
Not addressed, pre-existing on master: rb_postponed_job_trigger_for_ractor()
can copy threads.running_ec and dereference it arbitrarily later (the new
root narrows this but cannot protect an already-copied pointer), and a
freshly created dedicated native thread publishes its EC via
ruby_thread_set_native() before taking the scheduler lock.
Two Ractors looping over Thread.new and GC.start crash master in about
three seconds (19 of 20 runs); an ASAN build reports the heap-use-after-free
directly, at the read of th->nt in coroutine_thread_terminated(), freed by
gc_sweep via rb_thread_free_body. With this change ASAN is clean (0/10 vs
7/10 interleaved), as are the reproducer, s7_nested, both branches of the
epilogue under stress, a Ractor/GC soak, btest and test-all on release and
RUBY_DEBUG=1 builds.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rb_ractor_blocking_threads_inc/dec took the VM lock on every blocking
region boundary of a ractor's last runnable thread -- for a 1-thread
ractor, twice per IO operation -- to maintain vm->ractor.blocking_cnt.
That counter is only consumed by the win32 scheduler, so the pthread
build now skips those VM-lock sections; the per-ractor
threads.blocking_cnt and the ractor creation/exit protocol stay.
Ractor#inspect's status becomes running/terminated only ("blocking" is
gone). The finer states were really the win32 scheduler's
in-blocking-region flag leaking into inspect (under M:N, receive/sleep/
IO waits always showed "running" anyway, and the skip above stops the
remaining flips on pthread entirely).
R ractors x 1 thread, pipe write/read round-trips each, 16-HT machine
(Ryzen 9 5900HX), RUBY_MN_THREADS=1, best of 3, mean of 2 alternating
same-tree runs:
before after
1R 342k 356k
4R 877k 1016k
8R 884k 976k
16R 835k 884k
24R 793k 839k
The remaining ceiling is the scheduler-lock contention addressed by
"thread: keep context switches off the scheduler lock"; combined, 16R
reaches 3.6M rt/s.
Suggested-by: Koichi Sasada <ko1@atdot.net>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
thread_sched_setup_running_threads runs on every context switch (park,
resume, and both ends of every blocking region) and took the global
ractor.sched.lock to maintain the running_threads and timeslice lists
and running_cnt. With many ractors switching concurrently this lock is
the scaling ceiling.
Replace all three with ntlist, a per-location registration of who runs
where, and no shared counter:
- a thread on a shared nt registers in that nt's running_th, under the
nt's own lock -- uncontended, no cross-core traffic
- a dedicated nt goes on ntlist.running_dnts while its thread runs (the
switch rate of dedicated threads is what it always was; only the lock
shrank), so the fast path writes no shared cache line at all
The consumers walk the registrations: the barrier's interrupt walk also
counts them into a snapshot (barrier_running_cnt) and stamps each nt
with the barrier serial; a stamped nt deregistering during that barrier
decrements the snapshot under sched.lock, so the completion check stays
O(1) however many runners there are.
The timeslice list becomes timeslice.scheds, the scheds whose readyq
holds waiters. The enq that fills an empty readyq (and the add path
taking a turn with waiters already queued) links the sched; the timer
thread interrupts each listed sched's running thread and prunes entries
whose readyq drained. A steadily contended sched stays linked, so the
switch path never touches the list, and idle scheds are not walked at
all. The timer takes each sched's lock by trylock, because the
switchers nest sched.lock -> timeslice.lock; pruning is lazy, so
ractor_free delists the dying ractor's sched.
Barrier pairing needs no fences: the barrier sets barrier_is_waiting
and then walks the registrations under their locks; a switch moves its
registration under the same lock and then reads the flag. A walk that
missed a registration ran before that registration's critical section,
so its flag store is visible to that switcher. Nothing may start
running while the flag stands -- an interrupt flag alone would let a
resuming thread run, and allocate, inside the stop-the-world section --
so the add path takes its registration back and waits for
rb_ractor_sched_barrier_end's broadcast. While a scan holds a
running_th lock the registered thread cannot finish parking, so it
cannot die under the scan. An snt turned dedicated under
rb_thread_lock_native_thread has no creation-time running_thread, so
the running_dnts registration writes nt->running_thread itself.
R ractor pairs ping-ponging via ports (msgs/sec) and R ractors doing
pipe write/read round-trips (rt/sec), 16-HT machine (Ryzen 9 5900HX),
mean of 2 alternating same-tree runs:
before after
port pairs 8 503k 815k +62%
port pairs 32 453k 799k +76%
pipe self 4R 837k 940k +12%
pipe self 16R 820k 898k +10%
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ractor::Port#receive, Ractor.receive and Ractor.select gain a `timeout:`
keyword and return nil when it passes.
The wait itself stays where it is: rb_ractor_sched_wait() keeps parking the
thread in the thread scheduler, so an M:N thread still hands its native
thread back instead of becoming dedicated. How the deadline is taken
depends on which kind of thread waits:
A dedicated native thread parks on its own condvar, so it takes the deadline
there, the way native_cond_sleep() does. Nothing else is involved: the
condvar it waits on is the one a send already signals.
An M:N thread has no condvar of its own, so its deadline is armed on the
timer thread as a timeout-only wheel entry, and the timer thread wakes it
through thread_sched_to_ready_common(), which is exactly how
rb_ractor_sched_wakeup() wakes it for a send. That gives such a waiter two
wakers, so rb_ractor_sched_wakeup() now takes an armed timeout back before
waking, skips a thread that a fired timeout already made runnable, and bumps
the scheduler event serial so a timeout that has not fired cannot wake it a
second time.
`timeout: 0` is a poll: it never blocks, converts nothing and reads no clock,
so it costs what a receive of a waiting message costs. It still delivers the
incoming queue first, so a message another thread just sent is not missed.
A timeout that expires leaves the waiter on the Ractor's waiter list, as a
spurious wakeup does, and ractor_wait_receive() still delivers pending
messages before it reports the timeout, so a message that arrived just before
the deadline is not lost.
Both kinds of wait exist on every pthread platform, including builds without
the timer wheel (USE_MN_THREADS == 0), where every thread is dedicated. On
win32 the wait is a condvar wait, which takes the timeout directly.
A Ractor whose block ends while one of its own threads is still running took
a full second to terminate: rb_thread_terminate_all() sleeps for a second per
round and relies on the last sub-thread to wake it, but that wakeup was sent
only when the Ractor's main thread already had THREAD_KILLED, which
thread_start_func_2 sets after rb_thread_terminate_all() returns. For the
main Ractor rb_ec_cleanup() sets it beforehand, which is why only Ractors
were affected.
Setting THREAD_KILLED earlier is not an option here: thread_sched_switch()
reads it as to_dead, promising the M:N scheduler that the coroutine is never
resumed, and this thread still parks and resumes inside the wait. So mark
the wait itself instead: rb_thread_terminate_all() sets threads.terminating
before it waits, and the exiting thread wakes `main` on that. Nothing clears
it afterwards, because no thread of that Ractor runs again; the Ractor that
survives a fork gets it reset along with the rest of the thread set.
A port's queue lives in the receiving Ractor's table keyed by the port id, so
nothing ties it to the Ractor::Port object: when the port becomes unreachable
the queue stays, and so does every message left in it. Nobody can receive
them -- receiving needs the port -- but they are still GC roots.
A message reaching a Ractor makes that permanent. Port#<< of a port is the
natural shape of a credit token, and a token nobody takes holds the sender's
port, which marks the sender's Ractor object, which keeps its objspace in
zombie_objspaces for the life of the process:
200.times { port = Ractor::Port.new
Ractor.new(port) { |p| p << Ractor::Port.new; nil }.join }
6.times { GC.start }
ObjectSpace.each_object(Ractor).count # 201 before, 1 after
Mark and sweep the table itself. ractor_port_mark notes the ids a reachable
port still carries, and rb_ractor_finish_marking drops the queues nobody
noted, which is where the ractor-local keys are already reaped for the same
reason. Both run only where an unmarked port really is dead: after a mark
that covered every objspace -- a global GC or a full single-objspace one --
which is also when the world is stopped enough to read the table without the
owner's lock. Hence the new argument to rb_ractor_finish_marking: a minor
mark leaves live old objects unmarked.
The single-objspace question is asked first because mmtk marks from its GC
worker threads, which cannot reach the VM to answer the other one. Those
threads race on the flag, but they all store the same value and the reap
reads it once marking is over.
A queue starts alive so that one created between its port being marked and
the reap, which incremental marking allows, is not taken for an orphan. It
costs the queue one more cycle before it goes.
2000 rounds, dropping one message each, KB of RSS retained per round:
payload before after
Integer 13.0 13.3
Ractor::Port leaks 14.3
shareable 10KB 89.1 62.5
The last row is not retention: the payload is collected (0 of 500 strings
survive), but the reap only runs in a global GC, so between two of them the
dropped messages hold pages. Collecting every 20 rounds instead brings it to
1.5 KB.
The reap runs from rb_ractor_finish_marking, which only the default GC calls
-- mmtk has never called it, so the ractor-local keys it also frees are not
freed there either. Under mmtk a dropped message still holds its port, so
the test for that is a test-all case that omits itself, next to the other
tests for per-Ractor objspace semantics.
ractor-pipeline, which sends credit tokens this way, over 60 pipelines:
455MB growing by 5.6MB each to 272MB growing by 1.5MB, with the library
unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The root scan marked r->sync.legacy for every zombie objspace, deliberately
"without depending on wrapper reachability". That makes the value a GC root,
and a Ractor whose return value can reach the Ractor object roots itself:
Ractor.new { Ractor.current }.join
Nothing else names either one, but the value is a root, the value reaches the
wrapper, so the wrapper never dies, so ractor_free never runs,
rb_gc_objspace_disown is never called, and the objspace stays in
zombie_objspaces for the life of the process. The count never returning to
zero also holds rb_gc_single_objspace_p() false forever, which is the gate on
page_pool_reclaim, so the page pool stops returning arenas to the OS. A
Ractor::Port, an array, an ivar -- any path from the value back to the wrapper
does it; 10000 rounds of the line above leave 10001 live Ractors.
The value is of use only to whoever can still call Ractor#value, and that
means holding the wrapper, so mark it as the wrapper's child instead. A
self-referential pair is then ordinary garbage, and the transitive case (one
Ractor's value naming another) falls out of normal tracing rather than needing
a pass that iterates to a fixpoint.
Only during a global GC: it stops the world and marks every objspace together,
which is what lets the shareable wrapper reach an unshareable value. A local
GC must not, and never did -- the zombie scan this replaces also ran only in a
global GC.
300 rounds of the line above 10001 -> 1 live Ractors
8000 rounds with a dropped message 2.3 GB -> 216 MB, and it now plateaus
(287 MB at 20000, 288 MB at 40000)
Verified with RUBY_DEBUG=1: bootstraptest/test_ractor.rb, the GC and Ractor
unit tests, GC.verify_internal_consistency after holding 100 terminated
Ractors across GC.compact (their values, including self-references, all still
readable), and the same under GC.stress.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rb_ractor_mark_in_flight_for_single_objspace marked r->sync.legacy and
nothing else once the pin machinery went away, which is what
rb_ractor_mark_terminated_join_value next to it already does -- two functions
over the same field, and the surviving one names it correctly. Call that one
from the single-objspace branch and drop the other.
It also pins, which the plain rb_gc_mark did not. sync.legacy is a C-struct
slot with no compaction update hook anywhere, so pinning is the treatment the
field needs; the comment on rb_ractor_mark_terminated_join_value says so.
The name had stopped describing the function: "in flight" meant the sending
basket and the pinned copy snapshot it used to mark, and both are gone. The
word is also the GC-cycle sense elsewhere in this code, which is why the
wrapper it sat next to is now rb_gc_rest.
Only the ractor.c translation unit used it -- ractor.c includes
ractor_sync.c -- so the ractor_core.h declaration goes too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
rb_gc_finish_in_flight_gc is a one-line wrapper around rb_gc_impl_gc_rest,
and every other layer of the same call already says "gc rest": gc_rest in
gc/default/default.c, rb_gc_impl_gc_rest across the modular GC boundary. The
wrapper exists only because ractor.c cannot call rb_gc_impl_* itself, so it
should not rename the operation on the way out.
"in flight" also means a message in transit elsewhere in the Ractor code
(rb_ractor_repin_in_flight, rb_ractor_mark_in_flight_for_single_objspace),
which is a second sense for the word in the subsystem where both GC cycles
and messages are the subject.
The comment above the definition described the one caller; the caller already
carries that, so drop it there and keep the reason where the call is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
courier_alloc_node bumped c->count and then wrote the placeholder fields, so
for those few stores the mark walk over nodes[0, count) covered a slot that
had not been written yet. courier_alloc_ref already had the other order and
says why; make the two match.
Not reachable as written: nothing between the bump and the stores allocates
or reaches a safepoint, so no collection can start there, and the growth
above it happens before the bump. The order is what keeps that true without
having to check.
Found by the Copilot review on ruby/ruby#18392.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The off-heap courier started out as the move path's own structure, so every
part of it was named move_*. Copy now travels in the same structure and
through the same capture walk, with courier_build.copy selecting whether the
source is read or taken apart, which left the prefix claiming something
untrue: move_capture is what a copy goes through too.
Rename the shared machinery to courier_*: the struct and its build/
materialize/mark/free entry points, the node kinds, the capture and
materialize walks, and the array allocators. The two builders become
rb_ractor_courier_build_copy and rb_ractor_courier_build_move so the pair
reads as one family. What is still move-only keeps the move_ prefix --
move_preflight (copy has its own walk, copy_courier_supported_p) and
move_neutralize_source, which is exactly the step copy skips. In re.c the
MatchData transfer helpers become rb_match_blob_*, matching the neighbouring
rb_backtrace_blob_* it sits beside in the same node.
No behaviour change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two costs on the courier path that the on-heap snapshot did not have.
The node and ref arrays were grown by doubling while the graph was captured,
and growing them is not a realloc: the courier is a GC root while it is being
built, so a realloc would leave the old pointer live across a window where it
may already have been freed. Copying into a fresh array instead is correct
but not free, and it also made move_alloc_node/move_alloc_ref too big to
inline, which costs once per captured object.
Both walks that run before capture -- copy_courier_supported_p and
move_preflight -- already visit the whole graph, so let them count what
capture will allocate: one node per distinct unshareable object, one ref per
occurrence of a shareable one. move_courier_reserve then sizes both arrays
once and the growth path never runs (verified: it does not fire for any of
the benchmark payloads). It stays in place, out of line, in case a count
ever comes out short.
Materializing an array pushed its elements one at a time, through the
capacity check, although the length is known and the shell was allocated with
it. Set the length once and write the slots.
Message cost, 20k messages per trial, 9 trials, median us per message,
measured against master:
40B String 1.40 -> 1.04 4KB String 17.28 -> 8.66
bare Object 1.15 -> 1.05 20-key Hash 12.35 -> 7.61
nested 5.95 -> 5.48 100-elem Integer array 5.70 -> 6.46
The array of immediates is the one shape still behind master. Copy walks the
graph twice there (the preflight, then the capture) where the on-heap snapshot
walked it once; dropping the preflight would mean capture bailing out on an
unsupported type and falling back to Marshal, which also gives up the sizing
above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A basket is rooted by whoever holds it, and nothing else: the port queue it
waits on, or -- while it is on no queue -- its holder's off_queue_baskets
list. rb_ractor_sync grows that list and the basket a second ccan_list_node,
so it can sit on one or the other.
send alloc the basket, put it on the sender's list, then build the
payload into it: rb_ractor_{move,copy}_courier_build publish the
courier into the basket right after ZALLOC, so even a half-built
one is rooted. A move needs this -- it husks the sources as it
captures them, so what the courier has collected loses its other
root as the walk proceeds.
enqueue off the sender's list, onto the receiver's queue
receive off the queue, onto the receiver's list until the basket is freed
The publish is the handover of ownership too: when the capture raises, the
builder re-raises and leaves the courier where it is, and the basket frees it
exactly once as ractor_basket_new unwinds. (The raise path was pointed out by
the Copilot review on ruby/ruby#18392.)
Before this, a courier was the one payload ractor_basket_mark skipped: it
left the whole job to vm->ractor.move_courier_registry, which only a global
GC walks. Between the send and the next global GC nothing rooted the
payload, and a shareable object reachable from the courier alone was
collected -- a dynamic Symbol loses its fstr, a frozen Array its elements,
and reading it crashes. A shareable String survived only by being an
fstring, rooted elsewhere. That is already broken in master for move; it
becomes reachable from an ordinary send once copy travels by courier. Two
reproducers, both crashing unpatched master and fine on 4.0.2, now pass:
ports = 5.times.map do |i|
port = Ractor::Port.new
r = Ractor.new(port, i) { |p, n| p.send([:"sym_#{n}"], move: true) }
r.join
port
end
5.times { GC.start }
ports.each { |p| puts p.receive[0].inspect }
(the symbol has to be interpolated: a literal one is held by the iseq) The
second sends Ractor.make_shareable([1, "child"]) the same way.
ractor_sync_mark walks the list next to the queues it already walks, so this
needs no new root pass, and it walks both in every collection. "What a
courier holds is shareable, and only a global GC frees a shareable" does not
hold: pinned_roots_mark, which roots a shareable from its page bit, is
skipped once the process is back to a single Ractor
(rb_gc_single_objspace_p), and then an ordinary local GC frees one that
nothing else names. A payload in flight is named by its basket and nothing
else. The list needs no sync lock even so -- only its owner touches it,
unlike the queues a foreign sender writes. The one caller that does skip it
is the single-objspace root pass, where ractor_mark_unshareable_parts has
just walked the same baskets through ractor_sync_mark.
This retires the registry, its lock, its fork re-initialization and
rb_ractor_move_courier_registry_mark. Walking the registry in every
collection instead of the basket would not do: a receiver's local GC, which
materialize's own allocation can trigger, would then walk a courier another
Ractor is inside move_alloc_ref for -- observed as a crash.
move_alloc_node and move_alloc_ref grow by swapping in a fresh array rather
than realloc, for the same reason the nodes are initialized mark-safe: the
courier is a GC root while it is built, and a realloc can leave c->nodes
pointing at a block it has already freed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Materializing a string node called rb_enc_str_new, which copies the bytes a
second time: once into the courier when the node was built, once out of it
when the message arrives. Give the buffer to the String instead.
rb_str_new_owned takes an xmalloc'd buffer as a heap String's body, the way
rb_str_new_static does for a static one, except the String owns it and frees
it like any other. The courier's node hands its pointer over and forgets it,
so courier_free no longer frees what the String now owns.
Message cost, 20k messages per trial, 9 trials, median us per message:
payload 4.0.2 master before here
1KB 2.97 4.11 3.4 3.10
4KB 3.52 8.75 9.4 4.22
8KB 4.26 12.39 12.0 5.08
16KB 5.43 16.24 16.5 6.53
master pulls away from 4.0.2 as the payload grows; this tracks it within
about 20% across the range. Small strings are embedded and never had a
buffer, so they are unchanged.
A heap String is freed by its size (STR_HEAP_SIZE = capa + terminator), so the
node carries the capacity of the allocation rather than the bytes in use, and
reserves the encoding's terminator length instead of a single NUL.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With MatchData and an exception's backtrace on the courier, every copy
payload is off-heap: the courier carries the core types and anything else
travels as marshaled bytes. The sender-resident snapshot is gone, and with
it all the machinery that kept one alive across a global GC.
MatchData: rb_match_move_dump takes a release_source flag, since only a move
takes the source's onig region and char_offset apart.
Backtrace: rb_backtrace_blob_dump / _load / _mark copy the frames into an
off-heap blob. A frame only references shareable iseq and method-entry
imemos, so the blob can carry them as they are; it has no compaction update
hook, so the mark pins them (rb_gc_mark, not _movable).
Removed, all of it unreachable once no payload lives in the sender's heap:
rb_ractor_t pin_capture / _cnt / _capa, sending_basket,
gen_fields_capturing, ractor_pin_capture_push
basket p.pinned / p.pinned_cnt, ractor_basket_repin_in_flight,
rb_ractor_repin_in_flight and its queue walks
receiver struct ractor_materialize_frame, ec->materialize_frames and
its mark loop, sync.materializing_copies,
rb_ractor_materializing_p
gc rb_gc_pin_in_flight_message and the whole
rb_gc_impl_pin_in_flight_message hook (default, mmtk, the
modular function table), plus the two verifier relaxations
that existed for a snapshot being materialized
shref bits stay: the write barrier still records them for other reasons.
ractor_copy_native_try and copy_enter stay too, since
Ractor.make_shareable(obj, copy: true) still deep-copies within one objspace.
Message cost, 20k messages per trial, 9 trials, median us per message:
4.0.2 master here
4KB String 3.8 19.4 9.4
20-key Hash 3.3 11.9 7.5
nested Hash/Array/Str 5.3 7.1 5.1
100-elem Integer Array 4.4 5.7 6.9
40B String, Object same across all three
master is well behind 4.0.2 on multi-KB payloads; this recovers about half
of that and puts nested graphs back at 4.0.2's level.
Known issue, already in master and not introduced here: the courier's
registry mark does not traverse a shareable payload, so its children are
collected if a global GC runs while the message is in flight. Unpatched
master crashes on both repros (a dynamic Symbol loses its fstr, a frozen
Array loses its elements) and 4.0.2 does not. It only shows up for move
today; routing copy through the courier makes it reachable from a common
path, so it wants fixing before this ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 took the marshaled copy payload off-heap; this does the same for the
natively copied one, so an unreceived copy no longer pins a page of the
sender's heap for any payload type.
move_capture gains a copy mode: it reads the sources instead of taking them
apart (no husk, no buffer hand-over, no freeing the source's internals), and
copy_courier_supported_p decides up front whether the courier can carry the
whole graph. MatchData, IO, any other T_DATA and a singleton class are
rejected there and keep using the on-heap snapshot path, which handles or
rejects them exactly as before.
Retention with 3000 send-and-never-receive rounds (KB per sending Ractor,
before -> after):
String 79.5 -> 14.0 Array 79.5 -> 14.0
Object 79.5 -> 14.0 Hash 79.5 -> 14.0
14 KB is what a Ractor that sends nothing retains, and that part plateaus.
A courier is an in-process format, so a shareable payload can be embedded as
the VALUE itself rather than costing a whole move_node: child slots now hold
either a node id or, with the top bit set, an index into a compact c->refs
array that the registry marks. Without that, an array of immediates paid a
72-byte node, a hash lookup and an insert per element, and the message path
cost 2.5x what the native copier did (it skipped shareable children).
Message throughput (20k messages, n=9, interleaved, median us/message):
100-element Integer array 5.1 -> 4.7
4KB String 16.6 -> 9.3 (one memcpy, not two copies)
20-key Hash 12.0 -> 9.4
nested Hash/Array/String 10.1 -> 7.2
40B String, bare Object unchanged
It also restores a behaviour master lost: a String or Array subclass now
survives the copy again (4.0.2 copies with #clone and keeps the class;
master's native copier builds a base-class object). MyStr < String arrives
as MyStr, as it does under Marshal and #clone.
move_capture keeps testing the dedup table before shareability: move husks
each source as it goes, and a husk is a frozen field-less object that
rb_ractor_shareable_p answers true for, so the other order embedded the husk
instead of resolving a second occurrence to the first one's node
(bootstraptest/test_ractor.rb:661). An immediate skips the lookup, since
only captured objects are ever inserted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ractor_mark_unshareable_parts asserts that its Ractor is
not terminated. This is only relevant in the multi-objspace
root scan, where a terminated Ractor has left the set and
zombie_objspaces keeps its possessions alive.
With a single objspace the wrapper mark function also calls
this helper, and a terminated Ractor wrapper stays reachable
on the heap.
MMTk hits this assertion:
ractor.c:259: Assertion Failed: ractor_mark_unshareable_parts:!rb_ractor_status_p(r, ractor_terminated)
Ractor#join waits on a monitor port, and the exit token was sent from
ractor_notify_exit as the Ractor's block ends. Ractor#value then has to wait
for ractor_terminated on its own -- vm_remove_ractor retires the objspace the
joiner is about to absorb -- and it does so by spinning on rb_thread_schedule().
Now that the dying thread collects its own objspace post-mortem, that spin
covers a whole local GC: the joiner burns a core for as long as the child's
final collection takes, which grows with the child's heap.
Split the notification: the value is published at atexit as before, but the
tokens go out after the collection, from rb_ractor_postmortem() so that the
order of the two steps is stated in one place. What the joiner spins over is
then only the scheduler handoff and vm_remove_ractor, which is constant.
The send cannot go through ractor_basket_new there: thread_cleanup_func has
already dropped the EC's VM stack, and EC_PUSH_TAG reads ec->cfp when ZJIT is
built in. A token is a static Symbol, so ractor_basket_new_ref builds its
basket without a tag or any payload preparation.
Post-join spin, by the number of objects the child allocated:
0: 13us -> 3us
10000: 961us -> 3us
100000: 5211us -> 199us (the remainder is the absorb itself)
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every Ractor spawn eagerly built three IO objects (its $stdin, $stdout
and $stderr) under the VM lock in the setup section, though most
Ractors never touch stdio. Build them in rb_ractor_stdin and friends
on first access instead: the accessor runs in the owning Ractor, so
the objects still land in its own objspace. The slots stay 0 until
then -- marking 0 is a no-op, and the setters are unchanged.
A Ractor that never uses stdio now leaves nothing for its final
collection to finalize, which composes with the previous two commits:
the death of Ractor.new { :ok } releases every page it allocated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The retire GC at notify_exit could not reclaim the terminating
thread's scaffolding -- the Thread and Fiber wrappers, the ThreadGroup,
the Ractor-local stdio, whatever the machine stack still references --
because at that point they were all still rooted. Those ~11 objects
kept ~2 heap pages alive per dead Ractor, merged into the inheritor to
wait for its next major GC, so both a spawn-and-join loop and a
fire-and-forget loop grew the heap by two pages per Ractor with
nothing reclaiming them.
Instead, run the final collection at the tail of thread_start_func_2,
just before the scheduler handoff. The thread still holds the GVL, so
the collection runs lock-free like any local GC (a concurrent global
GC waits for its safepoint), and vm_remove_ractor keeps its light
lock-only section. r->postmortem makes rb_ractor_mark_local_roots
root only what must survive the death -- the join value
(r->sync.legacy) and the rb_gc_register_mark_object pins -- and the
zombie_objspaces entry recorded later measures exactly the pages the
joiner will inherit. While the Ractor is still in vm->ractor.set,
rb_vm_mark and ractor_mark keep the Ractor object itself and its
default port alive as before; everything they mark is foreign to or
pinned by this collection.
Two of the swept wrappers are the dying thread's own: its Thread (the
rest of the collection resolves GET_EC through ec->thread_ptr) and its
root Fiber (whose struct embeds that ec). thread_free and fiber_free
recognize "the struct I am standing on" (ec == the current ec), keep
the struct, and leave an in-band mark (th->self = 0; cont.self = 0,
which already means "no wrapper" on the never-wrapped path). Found as
a 1/30 heap corruption, pinned by ASAN. The collecting thread
captures those two pointers right after the collection, while its GVL
still excludes any global GC, and frees exactly the captured structs
at its very last step -- after vm_remove_ractor unlinks the Ractor,
anything it did not sweep may already be freed by a concurrent global
GC, so the epilogue reads no shared memory at all. Those frees would
resolve their objspace through the TLS ec -- which sits inside the
fiber struct being freed and leads to the retired Ractor -- so
rb_ractor_postmortem_free severs the TLS ec first (the MN epilogue
already had) and they fall back to the main objspace.
While a thread-event hook is registered the postmortem flag stays off
and the collection keeps the ordinary retire roots: callbacks may
retain any Ractor's Thread object (rb_internal_thread_event_hook_t),
and such references are invisible to the reduced roots. The check
takes the hook rwlock, so it happens-after any completed registration;
a hook registered after it gets no event from this thread --
rb_thread_execute_hooks skips a thread whose wrapper is gone
(th->self == 0) -- so it never receives what the collection swept.
On platforms without hook support (win32, none) the check reports
false.
The collection runs after thread_cleanup_func, so the dying thread's
machine stack is already torn down: the root scan skips its machine
context (rb_gc_impl_during_postmortem_p), which holds nothing the
reduced roots do not already cover and faults when scanned on some
platforms (windows-arm).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
move_preflight did not recurse into the regexp and the matched
string of a MatchData. A Regexp subclass is not shareable, so it
passed preflight. move_capture then mutated earlier sources in
the graph before the raise. This left the sources destroyed.
Recurse into rm->regexp and rm->str in preflight. This mirrors
move_capture. The raise now happens before any mutation.
The Ractor root scan marked each thread's ec directly and then marked the
wrapper of the fiber that ec lives in. Both reach rb_execution_context_mark,
so every thread's machine stack was scanned conservatively twice per
collection -- with 5000 threads, 750270 scans where 362326 were needed.
Marking the fiber wrapper is enough: th->ec is &fiber->cont.saved_ec (thread_mark
asserts as much), the wrapper goes on the mark stack during the root scan, and
the stack drains before anything is swept. Only when there is no wrapper yet,
mid-creation or teardown, does the ec still need marking of its own.
benchmark/vm_thread_pass_flood.rb, which floods 5000 threads:
collections 157 157 (unchanged)
time in GC 0.84s -> 0.49s
thread creation 1.17s -> 0.81s
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A move courier carries an IO by taking over its rb_io, so the source IO is
already a RactorMovedObject and nobody else can close the descriptor. When
the courier is freed without being delivered -- Ractor#send raising because
the port is closed is enough -- the branch for MOVE_KIND_IO did nothing and
the fd was leaked. Sending 20 IOs to a closed Ractor leaked 20 descriptors.
Finalize the rb_io instead, which closes the fd and frees the buffers. A
delivered IO already left fptr == NULL, so it is untouched.
Found by an automated review of the branch.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
RACTOR_CHECK_MODE stamped every object with the id of the Ractor that
allocated it (a 4 byte suffix on every slot), and re-stamped whole object
graphs whenever a value crossed a Ractor boundary. Per-Ractor objspaces
already record the owner: an object's page knows its objspace, so
rb_gc_obj_foreign_p() answers "is this another Ractor's object" with one
load, and the id, its suffix and the re-stamping traversals all go away.
Deriving it is also more accurate. The stamped id was never rewritten
when a zombie objspace was absorbed, which is what ractor_reset_belonging()
had to paper over by re-stamping received and inherited values. Page
ownership moves with the absorb, so nothing is left stale.
thread.c stamped the child's main Thread wrapper with the child's id
because upstream allocates that wrapper in the parent's objspace. This
branch already allocates it in the child's objspace, so the site goes too.
With ractor_reset_belonging() gone, no rb_obj_traverse() caller has an
enter function that descends into arbitrary types: make_shareable raises
on T_FILE and shareable_p stops there, so obj_traverse_i() can no longer
reach T_FILE and its case goes as well.
GC impls that keep a single objspace (mmtk) cannot tell owners apart and
report every object as their own, so the check degrades to a no-op there;
in exchange, dropping the suffix re-enables mmtk's ZJIT allocation
fastpath.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ractor#value used to be repeatable: the successor could call it again
and get the same object back, because the value stayed reachable from
the dead Ractor's C struct (sync.legacy). Keeping that slot valid
after the Ractor is gone is what forced the successor to root and pin
every Ractor it absorbed, through a VM-wide list guarded by its own
lock, with add, unlink and scan coming from three different threads.
Take the value out of the struct on the first call and raise
Ractor::Error afterwards. The value is then rooted by the caller like
any other return value, so the list, its lock and the pinning go away.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Implement send(move: true) with an off-heap courier that carries the
payload between objspaces, neutralizing the source into a
RactorMovedObject shell. Handle absorbing a terminated Ractor's heap,
keeping a zombie owner alive while another Ractor still reads its value,
and per-Ractor ownership of the VM tables.
Co-authored-by: Claude <noreply@anthropic.com>
Previously, JIT single-ractor mode invalidation happened on every `Ractor.new` call.
This invalidation takes the VM lock + VM barrier so slows down these calls. We only
need to invalidate compiled code that makes these single-ractor assumptions on the
transition from single-ractor to multi-ractor mode. The JITs never make single-ractor
assumptions when multi-ractor mode has been enabled.
If a process in multi-ractor mode forks, the forked process is set back to single-ractor
mode. The JITs can therefore make single-ractor assumptions in this new process, and the
invalidation can occur again when the first non-main Ractor is created in that process.
The array counterpart of f4e50b9993. A frozen array is handed out as a shared
root as it is, without the shared root flag (ary_make_shared), so a subseq or a
dup of an embedded one reads the elements straight out of its slot:
ary = Array.new(40) { |i| i + 1 }
ary.instance_variable_set(:@iv, []) # unshareable, so it is moved
ary.freeze
sharer = ary[1, 38] # reads ary's elements in place
r.send(ary, move: true)
sharer #=> [false, false, ...]
Two things go wrong once the root is hollowed out. The wipe added by 556296cbe0
zeroes the slot the sharer is reading, which is the visible breakage above; skip
it for an embedded shared root, as the string side already does.
The other one predates the wipe. Arrays let a shared root move and re-point the
sharer in gc_ref_update_array(), which needs to know that the payload lives in
the root's slot -- and a Ractor::MovedObject no longer says so, so the sharer was
left pointing into the old slot. Pin such a root instead, the way strings have
handled embedded roots since 80ea7fbad8.
Reachable since arrays moved to variable width allocation (a51f30c671): before
that an embedded array was at most 3 elements, and ary_make_partial copies rather
than shares at that size, so an embedded array was never a shared root.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Wiping the body of a hollowed-out object (556296cbe0) is wrong for a string
whose bytes another string reads in place:
r = Ractor.new { Ractor.receive }
str = "x" * 100
str.instance_variable_set(:@iv, []) # unshareable, so it is moved
str.freeze
dup = str.dup # reads str's bytes in place
r.send(str, move: true)
dup #=> "\0\0\0..." (was "xxx...")
str_replace_shared_without_enc() shares instead of copying whenever the target's
embedded capacity is too small, and the shared bytes are the source's own slot
when the source is embedded. The sharer keeps the root alive and reads that slot
for as long as it lives, so the move must leave it as it is. Skip the wipe for an
embedded shared root; every other case (a private heap buffer, a plain embedded
string) keeps it.
The default GC embeds strings up to a few hundred bytes and mmtk embeds any size,
so this is reachable with an ordinary dup of a frozen string.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ractor#send(move: true) turns the source object into a frozen
Ractor::MovedObject in place: it rewrites the header and gives the slot a
field-less shape, but leaves the old body untouched. Ruby-level access is
blocked by method_missing, yet C code that still holds the object from before
the move keeps reading it with its old type.
The visible case is $~ holding a MatchData that is then moved away:
r = Ractor.new { Ractor.receive }
"abc123xyz".match(/([a-z]+)(\\d+)/)
r.send($~, move: true)
"qqq777".match(/([a-z]+)(\\d+)/).frozen? #=> true
match_alloc_or_reuse() decides on MATCH_BUSY and RMATCH(existing)->capa, and
the husk satisfies both: rewriting the header cleared MATCH_BUSY, and capa
still reads the old capacity. RMatch fields are then written into a frozen
T_OBJECT whose shape says it has no fields, and once the GC walks it as a
ROBJECT the heap is corrupted: [BUG] try to mark T_NONE, out-of-heap,
use-after-poison in RSTRING_LEN, or silently wrong captures.
That the reuse is the cause is easy to confirm: if the following match needs
more registers than the husk's stale capa, no reuse happens and the same
program is clean.
Check the type in match_alloc_or_reuse() before trusting the body, and zero
the body when hollowing an object out, so that any other stale typed read
sees an empty object instead of the old internals (an Enumerator whose
Array#each is in progress, for instance, reads a length of 0 and stops
instead of walking off the end).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A frozen IO used to pass Ractor.make_shareable, but the result is not
useful and not sound:
* Almost nothing works on it. read, gets, seek, eof? and sync= all
raise FrozenError because they update the buffered state in rb_io;
only fileno and close remain.
* rb_io keeps VALUE members (pathv, tied_io_for_writing, the writeconv
fields, encs.ecopts, write_lock, timeout) in a C struct that the
make_shareable traversal never visits, so a shareable IO can still
reference unshareable objects. Keeping that invariant would mean
traversing all of them, which in turn drags in freezing the tied IO
and a Mutex that is not shareable itself.
Since a shareable IO can do nothing an unshareable one cannot, reject
it instead: Ractor.make_shareable(io) now raises Ractor::Error.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
make_shareable_check_shareable() assumes allow_frozen_shareable_p() can
only fail for T_DATA and dereferences RTYPEDDATA_TYPE() right after the
check. The VM_ASSERT documenting that assumption is compiled out in a
release build, so making any other type non-shareable segfaults instead
of raising. Raise Ractor::Error for the non-T_DATA case.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An UnboundMethod does not refer to any mutable children, so it should be
shareable when frozen. The documentation for RUBY_TYPED_FROZEN_SHAREABLE_NO_REC
states:
Similar to RUBY_TYPED_FROZEN_SHAREABLE, but doesn't make shareable
reachable objects from this T_DATA object on the Ractor.make_shareable.
Therefore, Ractor.shareable?(BasicObject.instance_method(:equal?).freeze)
should return true since it doesn't refer to any shareable children. However,
it doesn't do that right now so frozen UnboundMethod cannot be shared
between Ractors.
Moving an unshareable object graph with duplicate references failed to
preserve aliasing because the traversal checked the `rec` table too late.
On a move path, `move_leave` replaces the original object with a
`Ractor::MovedObject` tombstone. Because `move_enter` classifies this
tombstone as shareable, the second visit returned `traverse_skip` before
the visited `rec` lookup was reached.
Fix this by performing the `st_lookup` on `data->rec` at the top of
`obj_traverse_replace_i`, before calling `enter_func`. We guard the lookup
with a NULL check on `data->rec` to avoid allocating `rec_hash` for
simple graphs.
When an object is moved to another Ractor with `Ractor.send(obj, move: true)`,
the moved-out source is neutralized into a RactorMovedObject husk by move_leave.
The husk stays in the object's original, possibly larger, slot, but its shape
was reset to 0, losing the capacity. Shape 0 implies the smallest slot, so the
husk ended up with rb_obj_shape_slot_size(husk) != rb_gc_obj_slot_size(husk).
Since re-embeddable T_OBJECT fields landed, compaction requires every live
object to satisfy rb_gc_obj_slot_size == rb_obj_shape_slot_size (asserted in
gc_ref_update_object under RGENGC_CHECK_MODE). A GC.compact in the receiving
Ractor walks the husk and trips the assertion; in a release build it updates
references with a mismatched slot width and can corrupt the slot.
Give the husk an empty, frozen, RObject-layout shape while preserving its
SHAPE_ID_CAPACITY_MASK, so the slot size still matches and no field is exposed.
Co-Authored-By: Jean Boussier <jean.boussier@gmail.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For T_DATA objects which use declarative marking or have a NULL dmark
function, we can avoid taking the VM lock to iterate their children.
This is probably most significant for objects like Time.
Objects registered with rb_gc_register_mark_object were pinned in a single
vm->mark_object_ary. Give each Ractor its own (rb_ractor_t.mark_object_ary,
created lazily) and mark it from ractor_mark; the main Ractor's list is also
marked from rb_vm_mark so early-boot registrations are safe before the main
Ractor joins vm->ractor.set (e.g. under RUBY_DEBUG=gc_stress). When a Ractor
terminates, hand its objects to the main Ractor so these process-lifetime pins
stay alive.
global_object_list (rb_gc_register_address) stays VM-global: a registered
address is a slot whose value changes over time, so it has no single owner and
every root scan must keep seeing it. Behaviour is unchanged with a single
Ractor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Redesign the teardown of M:N (coroutine) threads around the natural call
structure nt_start -> co_start -> thread_start_func_2: when the thread body
returns, its epilogue (coroutine_thread_terminated) finishes all rb_thread_t /
rb_ractor_t business while th is still valid, and co_start then makes exactly
one final transfer back to the native thread's own context (nt_start's loop),
where the nt reclaims the dead coroutine's context. The resume itself proves
the final transfer's register save completed, so the reclaim needs no list,
no handshake and no extra locking.
This deletes the zombie-threads machinery (VM-global list, sched.finished
publish, the GC mark/reap pass) and closes its UAF window: publishing
`finished` before the terminal transfer let the GC free the Thread wrapper --
whose coroutine context the transfer still saves into -- mid-teardown.
* struct rb_thread_context bundles a coroutine thread's execution-owned
resources: the coroutine_context (kept first, so th->sched.context points
into the block) plus its machine stack, the stashed final-transfer target
(nt), and a dead flag. co_start's epilogue sets the flag and transfers via
the stashed nt, touching tctx alone -- th may already be collected by then.
The nt loop passes every coroutine it resumes from to
thread_sched_reclaim_from(), which frees dead ones (stack back to the
pool). A live coroutine cannot be resumed elsewhere, die and be freed
while the nt loop still inspects it: a live yield arrives with the sched
lock handed off, and no other nt can schedule that thread again until the
nt loop's own unlock.
* coroutine_thread_terminated (run from thread_start_func_2, where the
upstream non-coroutine teardown also runs, while th is still valid and
listed) does: wake the next thread as upstream's thread_sched_to_dead
does; stash th->nt into tctx and detach th->sched.context; leave the
living set; clear the current ec; and, for a shared-nt non-last thread,
enqueue the Ractor's next runnable thread (not earlier: waking it from
to_dead would let another thread of the same Ractor execute while this
teardown still runs). Everything that reads th happens before the
living-set removal, after which the GC may collect th (and, for a
Ractor's main thread, the Ractor).
* interrupt_lock is now destroyed in thread_free instead of during teardown
(thread_cleanup_func): while th is in its Ractor's living set, anyone --
terminate_all on the Ractor's terminating main thread, Thread#kill /
#raise -- may lock it, and destroying it mid-teardown leaves a window
where a concurrent interrupter locks a destroyed mutex (EINVAL; the
upstream non-coroutine path has the same, smaller, window). At thread_free
time the wrapper is unreachable AND off the living set (listed threads
are marked), so no interrupter can exist. The atfork cleanup reinitializes
the copied lock, which may have been held at the instant of fork.
* The Thread wrapper's dfree only reclaims the context of a thread that
never started; a terminated one cleared th->sched.context in its epilogue.
VM shutdown needs no extra synchronization: past the living-set removal the
epilogue touches only its own context block, the TLS and nt->nt_context,
none of which VM destruct frees (SNT native threads and the machine-stack
pool are intentionally not torn down at exit, as before).
Verified: bootstraptest 2050 PASS; test_thread / test_ractor / test_fiber /
test_gc / test_gc_compact / thread_queue 282 tests 0F0E; Ractor x Thread x
IO teardown churn 150/150 + 60/60 (exercises DNT promotion, terminate_all
racing dying threads, and the double-enqueue fixed in the previous commit);
fork with live Ractors + child GC 20/20; ASAN clean including unreferenced
Thread/Ractor collection (the old zombie window) and mixed stress;
RGENGC_CHECK_MODE=2 btest 2050 + churn; TSan warning profile identical to
master (the new functions appear in no race report); YJIT
move-under-GC.stress; kill/join races; Thread object retention across
Ractor death; exit-while-winding.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
`rb_gc_obj_slot_size` has to check the page metadata to
get the slot size, which likely cause cache misses.
`rb_obj_shape_slot_size` does simple arithmetic using
only a few shape bits.
```
compare-ruby: ruby 4.1.0dev (2026-07-08T03:48:16Z master 01c5b2e890) +PRISM [arm64-darwin25]
built-ruby: ruby 4.1.0dev (2026-07-08T10:17:41Z gc-changed-slot-size 08029b0f7c) +PRISM [arm64-darwin25]
warming up.
```
| |compare-ruby|built-ruby|
|:-----------------------|-----------:|---------:|
|binary_concat_embedded | 5.262M| 5.707M|
| | -| 1.08x|
Set RUBY_TYPED_THREAD_SAFE_FREE on TypedData types whose dfree function
is trivially safe, and only frees its own memory
Types not yet marked as THREAD_SAFE_FREE:
- id2ref_tbl_type: sets the process-global id2ref_tbl to NULL.
- mutex_data_type: unlinks itself from another thread's keeping_mutexes list.
- autoload_data_type: deletes nodes from a shared intrusive list.
- autoload_const_type: deletes itself from a shared list.
- rb_cont_data_type: mutates the shared fiber pool and the global first_jit_cont list.
- rb_fiber_data_type: delegates to cont_free.
- FiberPoolDataType: mutates shared fiber pool free-lists.
- ractor_data_type: reconfigures global VM event-hook flags and call caches.
- exported_object_registry: frees a global table under the VM lock.
- rb_box_data_type (Box::Entry): unlinks classext from other live class and module objects.
- box_ext_cleanup_type: dereferences another String object and calls unlink.
- monitor_data_type: uses the default free, so the flag has no effect.
Co-authored-by: Luke Gruber <luke.gruber@shopify.com>
* Reserve 2 bits for expressing object layout
We would like to make instance variable reads in the JIT compiler faster
(as well as simplify the JIT implementation). Currently, in order to
read an instance variable, we have to:
1. Test for heap object
2. Load object to a 64 bit register
3. Mask the object header
4. Bit test against the masked header
5. JNE
6. Load field
We would like to:
1. Test for heap object
2. Load object shape to a 32 bit register
3. Bit test against the shape
4. JNE
5. Load field
The way we fetch instance variables is not consistent across objects.
In order to realize our goal, we need to encode object layout inside the
shape. If we encode object layout inside the shape, then the shape
itself will guarantee that the access pattern generated by the JIT
compiler is correct.
We should encode the following load patterns into the shape tag bits.
This way we can share shapes on transitions, but be able to
differentiate the access patterns for the JIT compiler. In other words,
two objects can have an `@a -> @b -> @c` transition and share the same
shape, but the tag bits can differentiate the access pattern so that the
JIT compiler can be confident that the machine code is correct.
Here are the patterns:
1. Embedded/Extended T_OBJECT Instance Variables
Objects with direct references to instance variables or via malloc
buffer
2. Objects with fields_objects fields
These are Data and TypedData objects. They have an associated axillary
imemo/fields object that stores the instance variables. The access
pattern is `object[2] + 2`. The fields object is the 3rd field, and the
instance variables start at +2 inside the fields object. The fields
object itself is a Ruby object, so it contains the usual header bits +
class headers.
3. Non Boxable Classes / Modules
This is similar to Objects with fields_objects, but the fields object is
stored at a different offset. We’re differentiating this from boxable
classes and modules because those are harder to support.
4. Other
"Other" pattern is for objects that are rare, or have
difficult-to-implement access patterns. This includes:
* Boxable classes and modules
* Structs (for now)
* Objects that use the geniv table
Proposed shape bit layout:
```
Current shape_id_t is 32 bits:
31 28 27 26 25 24 23 22 19 18 0
+-----------+--+--+--+--+--+------------+----------------------------+
| unused |L1|L0|OI|FR|CX| heap index | shape tree offset |
+-----------+--+--+--+--+--+------------+----------------------------+
| | | | | | |
| | | | | | +-- bits 0-18: SHAPE_ID_OFFSET_MASK
| | | | | +--------------- bits 19-22: SHAPE_ID_HEAP_INDEX_MASK
| | | | +------------------ bit 23: SHAPE_ID_FL_COMPLEX
| | | +--------------------- bit 24: SHAPE_ID_FL_FROZEN
| | +------------------------ bit 25: SHAPE_ID_FL_HAS_OBJECT_ID
+--+--------------------------- bits 26-27: SHAPE_ID_LAYOUT_MASK
```
The important part about these layout patterns is that they do not
reflect the _type_ of object, only how the object is laid out in memory.
For example, we currently treat structs as "other", but we can refactor
them to have the same layout as "Objects with fields_objects", and when
we do that they should get a different bit in the shape header.
This commit only reserves the two bits, it doesn't use them in the JIT
compiler yet.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Co-Authored-By: Max Bernstein <tekknolagi@gmail.com>
* Update gc.c
Co-authored-by: Nobuyoshi Nakada <nobu.nakada@gmail.com>
* Update shape.h
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
* fix function name
* Update shape.c
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
* fix function name
* Revert "Update shape.c"
This reverts commit 900711defc6c541a93f3393a350819ae88cf87f1.
* add comment
---------
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: Max Bernstein <tekknolagi@gmail.com>
Co-authored-by: Nobuyoshi Nakada <nobu.nakada@gmail.com>
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
