In single-ractor mode, we don't want foreign_object_p checks to actually
run (there can be no foreign objspaces). This foreign_object_p check return
false if there was never a (non-main) Ractor created. Previously,
rb_gc_ever_multi_ractor_p() was needlessly complex. This change should
speed up marking and WB implementations in single-ractor mode.
Since e631daf17b, gc_sweep_finish calls
gc_malloc_counters_snapshot_free_at_last_gc on both oldcounters and
counters.
That is correct for counters, because gc_reset_malloc_info takes the
baselines at every GC. However for oldcounters, baselines are only taken
at a major GC, so every minor sweep forgot the frees it had just made
but keeps the allocation and oldmalloc_increase grows with allocation
volume.
This means a program churning short-lived malloc'd buffers hit the
oldmalloc limit every few minors.
This commit goes back to having a single oldmalloc_increase counter,
clamped at 0, reset by a major and checked at the end of a minor sweep.
I used the following script to measure the impact of this change.
```ruby
def now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
GC.start
before = GC.stat
start = now
2_000_000.times { "x" * 4096 }
after = GC.stat
elapsed = now - start
majors = after[:major_gc_count] - before[:major_gc_count]
minors = after[:minor_gc_count] - before[:minor_gc_count]
puts [
format("%-60s", RUBY_DESCRIPTION[0, 60]),
"majors=#{majors}",
"minors=#{minors}",
"oldmalloc_inc=#{after[:oldmalloc_increase_bytes]}",
"limit=#{after[:oldmalloc_increase_bytes_limit]}",
format("%.2fs", elapsed),
].join(" ")
```
And observed the following results:
4.0.7:
majors=0 minors=614 oldmalloc_inc=3343152 limit=16777216 0.46s
Previous master:
majors=53 minors=191 oldmalloc_inc=63220807 limit=134217728 0.65s
This commit:
majors=1 minors=243 oldmalloc_inc=20829148 limit=16777216 0.64s
The size_t atomics had add, sub, cas, exchange and fetch_add but no
load, so the GC read its atomic size_t counters through the VALUE load
with a cast.
This commit adds RUBY_ATOMIC_SIZE_LOAD and rbimpl_atomic_size_load,
following rbimpl_atomic_load, and uses it for the deferred free count.
The zombie_objspaces table in gc.c kept a page count for each entry and
a running total. gc.c asked the GC for the counts through
rb_gc_impl_zombie_heap_page_count, and the default GC read the total
back through rb_gc_vm_zombie_total_pages and
rb_gc_vm_refresh_zombie_pages. Only the default GC uses the numbers, so
they crossed the GC API twice for nothing.
In this commit we replace both page count functions with one hook,
rb_gc_impl_objspace_zombify, which gc.c calls when it parks an objspace
in the zombie table. The default GC stores the floored page count in the
objspace and keeps the total in the global objspace. objspace_absorb
subtracts it again, and each global cycle re-measures the zombies in its
snapshot. A nonzero zombie_pages also marks an objspace as a zombie, so
the verifier no longer needs rb_gc_vm_zombie_objspace_p.
The writers keep the same locks as before: the VM lock for zombify and
absorb, the barrier for the refresh. With RGENGC_CHECK_MODE we assert at
each global snapshot that zombie_total_pages equals the sum of
zombie_pages over the enumerated objspaces.
Recently we added some keys to GC.stat in order to measure the GC
performance of Ractor local GC and the associated global GC.
This commit refactors some of those terms: "global" was used in two
places with different meanings which could get confusing.
When it was passed to GC.stat as scope: :global it meant "Give me
aggregate stats for all the GCs running in this process".
Whereas inside the output of GC.stat, global referred to the number of
"global" GCs that are run. That is: stop-the-world GCs that run on
every ractor heap in the process.
Also the global_gc_count that GC.stat reports for a Ractor ignored
global GCs that other Ractors started, even though they swept its heap.
In this commit we fix this confusing behaviour, and make the GC.stat
output more consistent.
We've replaced the scope: parameter to GC.stat with the boolean
process: keyword arg. The default is process: false which means
GC.stat will return stats for the Ractor that executes the code (the
main Ractor if no other Ractors have been created). process: true will
return aggregate stats over all the Ractors in the process (including
dead ones).
Inside GC.stat(process: false), count and global_gc_count now
include every global GC that has swept that Ractor's heap. So Ractor
local gc count == count - global_gc_count.
heap_marked_slots now counts only the marked objects in that Ractor's
own heap. Before, the Ractor that started a global GC counted the
marked objects of every heap, and the other Ractors reported 0.
We also provide a new key global_gc_invocation_count that is the
number of global GCs that have been triggered by the Ractor in question.
We've also moved some keys related to the global page pool out of the
default GC.stat output and into GC.stat(process: true), because they
are process-wide and not Ractor specific.
There was a condition that only a heap with > 1 page can move an empty
page from free_pages to empty_pages. This was even true during a retire
GC (Ractor#join), so running a retire GC couldn't release these actually
empty pages back to the page pool. This meant that when a terminated Ractor
got absorbed, 768KB (12 heap pages) came with it, even if those pages were
empty (this assumes the Ractor had a page for every heap).
Since Ractors can now have 0 pages and the main global GC trigger is
zombie pages (and not retired Ractors), we need to count retired Ractors
with 0 pages towards this zombie pages limit. For now, the floor is
HEAP_COUNT (12 pages), but this can be changed if it's too high.
A Ractor's zombies with finalizers are poisoned and moved to the main
objspace when the Ractor terminates. If the process exits before they
are finalized, rb_gc_impl_shutdown_call_finalizer_i touches them and
ASAN reports use-after-poison.
```
8.times.map do
Ractor.new do
10.times do
4.times { ObjectSpace.define_finalizer(Object.new, proc { sleep 0.0005 }) }
GC.start
sleep 0.0001
end
end
end.each(&:join)
#=> ERROR: AddressSanitizer: use-after-poison (sometimes)
```
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`zjit_during_incremental_marking` was one process-wide global, written
by `gc_set_incremental_marking` on every transition of any objspace.
Since 24be314b45, two objspaces can run incremental marks at the same
time, so when one finished its mark it wrote zero while another was
still marking.
`gen_write_barrier` skips `rb_gc_writebarrier` when the receiver is young,
not shareable, and the word reads zero, so JIT code in the still-marking
objspace skipped the barrier.
A white object stored into a young black receiver was never marked, the
sweep freed it, and the next major crashed with
```
[BUG] try to mark T_NONE object (parent: T_ARRAY/Array).
```
This bug can be reproduced in a few seconds with
```
RUBY_GC_HEAP_OLDOBJECT_LIMIT_FACTOR=0.5 ./ruby --zjit test.rb, where test.rb is:
```
```
Ractor.new {
loop {
GC.start(full_mark: true, immediate_mark: false, global: false)
GC.start(global: false)
}
}
loop {
GC.start(full_mark: true, immediate_mark: false, global: false)
Array.new(2_000) { [nil] }.each { |b|
200.times { Object.new }
sleep 0.0002
b[0] = Object.new
}
}
```
Limiting the oldobject limit factor ensures we keep running majors which
makes this bug easier to trigger, and the array is pre-sized because the
inlined ArrayAset side-exits on an empty array.
This commit moves zjit_incremental_marking into rb_objspace_t and makes
it a bool, so each Ractor can signal to zjit when it's incrementally
marking. Because this is now directly addressable per objspace we've
also removed the incremental marking flag.
Compared to the default of 225, this gives noticeable performance
improvements.
For the following benchmark:
objs = []
2_000_000.times { objs << Object.new }
We can see this branch is about 10% faster:
Benchmark 1: branch
Time (mean ± σ): 193.0 ms ± 3.3 ms [User: 207.0 ms, System: 79.2 ms]
Range (min … max): 189.1 ms … 203.1 ms 15 runs
Benchmark 2: master
Time (mean ± σ): 214.9 ms ± 4.9 ms [User: 245.2 ms, System: 79.5 ms]
Range (min … max): 206.4 ms … 220.4 ms 13 runs
Summary
branch ran
1.11 ± 0.03 times faster than master
https://github.com/ruby/mmtk/commit/b736a1f92b
- Implement the gc_impl.h functions added since wbcheck was last updated
(single-objspace stubs, as in mmtk)
- Create the object table in objspace_init, since st_init_numtable needs
an objspace to allocate from
- Report during_global_gc_p outside the mutator phase: snapshots and GCs
run behind rb_gc_vm_barrier()
- Drop the internal header includes (only RCLASS_IS_ROOT was used), so
wbcheck builds like the other modular GCs without the generated id.h
A recent update in mmtk-core put non-moving roots behind a cargo feature
"non_moving_root". CRuby has been using non-moving roots for
conservative stack scanning. We explicitly enable this feature in
Cargo.toml.
https://github.com/ruby/mmtk/commit/ee8c7d2323
This commit removes RubyUpcalls and instead directly calls functions in
the C binding from Rust. This also allows LTO to inline C calls from Rust.
https://github.com/ruby/mmtk/commit/0f5a049a2d
The courier's hash size is an entry count rather than a String or Array length, so it stays `long`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Once rb_len_t widens this changes the `long *` output parameters of rb_range_beg_len(), rb_str_subpos(), rb_arithmetic_sequence_beg_len_step(), ruby_enc_find_basename() and ruby_enc_find_extname(), so extensions passing `long *` to them will need to be updated. The extension guides list these signatures verbatim and follow along.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
For `WriteBarrier a, b`, inline the fast-path of the write barrier that covers the majority of cases and doesn't require any actual barrier:
* `a` isn't old
* `a` isn't shareable
* not incremental marking
Removes the vast majority of barrier C calls.
Lobsters before:
```
Top-20 calls to C functions from JIT code (69.2% of total 50,814,557):
rb_hash_aref: 5,770,278 (11.4%)
rb_vm_opt_send_without_block: 5,428,040 (10.7%)
rb_gc_writebarrier: 3,304,955 ( 6.5%)
...
```
After: completely gone from top-20. Total C calls drop from ~50.8MM to ~47.6MM.
On a microbenchmark such as `setivar_object` or `setivar_young`, it takes per-iteration time from ~41ms to ~36ms (roughly -12%).
```c
void
rb_gc_impl_writebarrier(void *objspace_ptr, VALUE a, VALUE b)
{
rb_objspace_t *objspace = objspace_ptr;
#if RGENGC_CHECK_MODE
if (SPECIAL_CONST_P(a)) rb_bug("rb_gc_writebarrier: a is special const: %"PRIxVALUE, a);
if (SPECIAL_CONST_P(b)) rb_bug("rb_gc_writebarrier: b is special const: %"PRIxVALUE, b);
#else
RBIMPL_ASSERT_OR_ASSUME(!SPECIAL_CONST_P(a));
RBIMPL_ASSERT_OR_ASSUME(!SPECIAL_CONST_P(b));
#endif
GC_ASSERT(!during_gc);
GC_ASSERT(RB_BUILTIN_TYPE(a) != T_NONE);
GC_ASSERT(RB_BUILTIN_TYPE(a) != T_MOVED);
GC_ASSERT(RB_BUILTIN_TYPE(a) != T_ZOMBIE);
/* A shareable object now references an unshareable one: record b as a shref so its
* owner's local GC roots it (the parent may live in another objspace, untraversed
* there). Only b's owner stores this, on its own page: a plain store suffices. */
if (RB_UNLIKELY(RB_FL_TEST_RAW(a, RUBY_FL_SHAREABLE)) &&
!RB_FL_TEST_RAW(b, RUBY_FL_SHAREABLE)) {
struct heap_page *bpage = GET_HEAP_PAGE(b);
if (!_MARKED_IN_BITMAP(bpage->shref_bits, bpage, b)) {
_MARK_IN_BITMAP(bpage->shref_bits, bpage, b);
bpage->flags.has_shref_objects = TRUE;
}
}
retry:
if (!is_incremental_marking(objspace)) {
/* The generational barrier covers old->young edges within one objspace only; a
* foreign a or b has age bits another objspace mutates, unsafe to read, so check
* locality first when multi-Ractor (a foreign a is shareable and the shref above
* already keeps b alive). With a single Ractor nothing is foreign. */
if ((rb_gc_multi_ractor_p() &&
(GET_HEAP_OBJSPACE(a) != objspace || GET_HEAP_OBJSPACE(b) != objspace)) ||
!RVALUE_OLD_P(objspace, a) || RVALUE_OLD_P(objspace, b)) {
// do nothing
}
else {
gc_writebarrier_generational(a, b, objspace);
}
}
else {
/* Slow path, no lock: incremental marking only runs while the process has a single
* objspace, so the owning Ractor's GVL already serializes this barrier against its
* own GC. */
if (is_incremental_marking(objspace)) {
gc_writebarrier_incremental(a, b, objspace);
}
else {
goto retry;
}
}
return;
}
```
Ractor.make_shareable could previously take the VM lock and then call
rb_funcall(id_freeze), which is not correct to do. Create a new
rb_objspace_reachable_objects_from_local() function that works the same as the
non-local version except doesn't lock. The callback given to it must skip foreign
objects (classes/modules, etc.) as running mark_children could traverse data
structures that could be mutated out from under it.
The reachable_objects_from_local callback must skip foreign objects altogether
instead of checking their shareability because foreign objects that aren't
shareable could be collected or have their page unmapped due to RLGC. Of course,
foreign edges should be shareable anyway. This was a bug that existed in the
locking version as well. Unfortunately, that means we can't check that T_DATA
objects don't refer to unshareables across object spaces while making them
shareable.
The alternative is to check and risk a crash, and raise if we see these illegal
edges.
GC.stat(:global_gc_count) now reports the stop-the-world global cycles
the current Ractor started, instead of mirroring the process-wide count,
so the values can be summed across Ractors. GC.stat(:global_gc_count,
scope: :global) gives the process total, including Ractors that have
since exited.
A global cycle no longer counts as a major GC on the Ractor that started
it, so count == minor_gc_count + major_gc_count + global_gc_count.
WB-unprotected objects are always young. This invariant is maintained throughout the code base.
When WB-unprotecting an object, the object gets de-aged.
sigprocmask() is unspecified in multithreaded processes; on macOS it
left SIGSEGV/SIGBUS blocked on the faulting thread, so a second
read-barrier fault was never delivered and the process hung on the
faulting instruction. [Bug #22389]
Match the per-objspace ordering of gc_compact_finish (unprotect, then
update) across objspaces, so the update pass never faults on a foreign
protected page and abandons that page's moves.
There was a bug in mmtk-core that caused `is_live()` and
`is_reachable()` to return wrong results for objects in the large object
space. The function `object_survives_current_gc` was introduced to
workaround the bug.
The bug has been fixed in mmtk-core. We bump the mmtk-core dependency
revision, and remove the workaround. We now simply use
`object.is_reachable()` to test if an object will survive the current
GC.
See: https://github.com/mmtk/mmtk-core/pull/1594https://github.com/ruby/mmtk/commit/8676718e7d
Currently, if GC is disabled, postmortem GC is not ran when a Ractor dies.
This can cause the object space to have issues because we need to run the
postmortem GC. This commit will guarantee that it runs.
Now that registrations are Ractor-scoped, a store of another Ractor's
unshareable object through a registered address is a use-after-free
waiting for the owner's next GC.
This commit introduces a method to catch it in debug and ASAN builds by
recording each VALUE at registration time so that when
gc_verify_internal_consistency walks the registration list it can flag
live, non-shareable values that are owned by a Ractor other than the
registrant.
The registered_globals list made every Ractor's GC scan every slot under
a shared lock. This commit splits it into per-Ractor lists.
We also maintain a global registry that references every Ractor that has
ever registered so that unregistering from a Ractor other than the one
that registered can look up the owning list.
When a Ractor is joined, the dead Ractor's slots are moved to the
joining Ractor's heap before the objspace merge happens.
This change tightens up the requirements for values stored in a
registered address.
Previously, Every Ractor's local GC scanned the shared list. So if a
slot held an unshareable object living in a different Ractor's heap,
that object was always kept alive regardless.
After this change, only the registering Ractor's GC scans its list. So if a
slot holds another Ractor's unshareable object, the owning Ractor's
local GC cannot see that reference, so it can collect the object while
the registered slot still points at it.
Storing such an object was already a Ractor containment violation. It
was just hidden by the shared scan.
[Feature #22277]
Previously, major GC incremental marking was disabled (even on the main Ractor)
when more than 1 Ractor was running. By changing the WB implementation, we now
have it back.
We need to keep track of pending pages in RubyHeapTrigger otherwise a large
allocation may cause infinite number of GCs to be ran since it will always
appear that we have enough space in the heap (and thus the heap won't grow)
but still not have enough for the allocation.
https://github.com/ruby/mmtk/commit/dd8034050d
Sweeping unshareables generally doesn't modify global VM data structures. The exception
is rb_free_generic_ivar, which modifies the generic_fields_tbl. However, that table has
its own mutex protecting it.
Each objspace has its own finalizer list since RLGC landed so there aren't
concurrency issues here. This used to be a global table that all Ractors
shared. This affects only the default GC.
When running multiple Ractors, we can't call dfree of a non-thread-safe
T_DATA while other Ractors are running or doing GC work. If there are multiple
Ractors, we save the T_DATA object's `dfree` and `data` pointer in an entry. Once
the entry buffer is full per-Ractor, we publish it to the global list atomically.
Once the threshold for these deferred T_DATAs is reached (right now ~65K), the next
Ractor to run interrupts calls the `dfree` functions for these objects under the VM
barrier. It does not do any other GC work during this postponed job.
If a global GC runs, it also calls the `dfree` functions for all these objects (it drains
the deferred-free queue). When only 1 Ractor is running (`rb_gc_single_objspace_p()`),
these objects are freed normally and aren't added to any queue.
NOTE
----
Embedded T_DATA objects need to be special-cased because their `data` points to inside the
object's slot itself, which we reuse right away now. For non-thread-safe T_DATA that is
embeddable, we need to create it as unembedded - even without Ractors.
The initial heap size is a slot count target, bytes / slot_size for each
heap, so below one slot in the largest heap heap_prepare never forces that
heap's first page. RUBY_GC_RACTOR_HEAP_INIT_BYTES rejects such values;
RUBY_GC_HEAP_INIT_BYTES accepts anything above zero, and dies partway
through boot:
$ RUBY_GC_HEAP_INIT_BYTES=512 ./ruby -e 'puts "ok"'
encdb.so: [BUG] cannot create a new page after major GC
Give it the same bound. Too small a value is now ignored with a warning
under -w, as one that overflows or does not parse already is:
RUBY_GC_HEAP_INIT_BYTES=512 (default value: 2621440) is ignored
because it must be greater than 1023.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gc_marks_finish keeps at least heap_free_slots * r_mul slots free, where
r_mul was the number of Ractors allocating from this objspace, capped at
8. Per-Ractor GC replaced that count with the VM-wide one, so every
Ractor's objspace now asks for free slots in proportion to how many
Ractors the whole VM has, although only its own Ractor allocates from it.
The floor can then exceed the heap it applies to. A Ractor started with
the smallest initial heap holds about 7,200 slots; with one other Ractor
in the VM the floor is 4096 * 2, so no sweep can ever leave enough behind
and every collection escalates to a major one:
1,000,000 allocations on one Ractor, all garbage
collections major reason
before 409 102 nofree
after 617 1 oldgen
Lowering RUBY_GC_HEAP_FREE_SLOTS to 1024 has the same effect, which is
what identified the floor. The major collections free nothing: they cost
8% on a Ractor that allocates nothing but garbage and nothing once one
holds a live set, and the heap does not shrink. Memory is unchanged
either way, at 298 KB per Ractor.
The main objspace keeps the VM-wide count it has used since before
per-Ractor GC.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A page with nothing live on it leaves its heap for the global empty pages,
and may be returned to the system from there. When it is the heap's only
page the heap is left with none, and nothing grows a heap in that state:
gc_sweep_finish_heap's growth path asks for total_slots > 0, and
heap_prepare forces a first page only when it is not already sweeping. An
allocation in that heap then reaches "cannot create a new page after GC".
It takes a heap whose page empties completely while another heap is being
swept, so a Ractor allocating a mix of sizes finds it and one allocating a
single size does not:
Warning[:experimental] = false
r = Ractor.new {
20_000.times { |i| [Array.new(i % 300), "x" * (i % 2000), {a: 1}] }
}
r.value
Keep the page instead. A heap that has been used once holds at least one,
which is what the rest of the code already assumes. Recovering afterwards
in heap_prepare was measured too and holds the same 224-231 KB per Ractor,
but this keeps the invariant rather than repairing it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
heap_init_bytes is a slot count target applied per objspace, bytes divided
by each heap's slot size: heap_prepare fills a heap up to it before the
first collection, gc_sweep_finish_heap puts a floor under the free slots a
sweep should leave, and gc_marks_finish sums it across the heaps as the cap
on how much may stay free, which is what keeps empty pages around. With
one shared objspace this 2.5 MB was paid once. With per-Ractor GC every
Ractor's own objspace pays it again, and that is the steady heap a Ractor
holds: a Ractor that allocates 40,000 objects and discards them keeps
4.3 MB, where Ruby 4.0 kept 71 KB.
Add ractor_heap_init_bytes (RUBY_GC_RACTOR_HEAP_INIT_BYTES) and use it for
every objspace but the main one. Its default, 0, is resolved at boot to
the smallest size that works: one slot in the largest heap. Below that,
heap_prepare never forces a heap's first page and allocating there fails
with "cannot create a new page after GC", so the environment variable
rejects smaller values the way the other size parameters reject theirs.
Measured with 64 Ractors, each allocating 40,000 objects and then
discarding them, reported after a GC so that it is the heap floor rather
than live data:
ruby kept per Ractor GCs/Ractor wall
4.0.6 70.9 KB 147.1 271 ms
master 4,310.3 KB 1.2 264 ms
this commit 302.2 KB 24.5 195 ms
Nothing between the minimum and one page's worth is worth choosing: from
1 KB to 16 KB the GC behaves identically (409 collections, 102 of them
major) and only the memory grows, 302 KB to 857 KB. The 102 major
collections are all MAJOR_BY_NOFREE, and they stop at 64 KB, which is
HEAP_PAGE_SIZE: below one page's worth of slots, a sweep can never leave
enough free slots behind. Removing them costs three times the memory
(910 KB) and buys 8% on a Ractor that allocates nothing but garbage, and
nothing at all once a Ractor holds a live set: with 200,000 objects live
the minimum runs 56 collections against 45 and takes 1% longer.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
setup_gc_stat_symbols(), setup_gc_stat_heap_symbols() and the symbols
gc_info_decode() decodes were each filled on first use and guarded by
their own first element. That element is also the first one written, so
a second ractor arriving while the first was still filling saw the guard
already set and skipped the setup, leaving the later entries at 0:
Warning[:experimental] = false
port = Ractor::Port.new
256.times { Ractor.new(port) { |p|
begin
GC.stat(:heap_allocated_pages); p << :ok
rescue ArgumentError => e
p << e.message
end
nil
} }
p 256.times.map { port.receive }.reject { _1 == :ok }
#=> ["unknown key: heap_allocated_pages"]
Reproduces about once per ten runs here; calling GC.stat once on the main
ractor first makes it go away, which is what pointed at the guard.
Fill all three tables from rb_gc_impl_init(), where no other ractor
exists yet, and drop the guards. gc_info_decode()'s function-local
statics move to file scope so the new setup function can reach them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Peter Zhu