Don't let GC's own frees pay for the next cycle's malloc budget

Since 45fd5d58f1 both malloc counter baselines are taken at the start of a
GC, so every xfree performed by sweeping lands in `free_delta` and is
subtracted from the next cycle's `malloc_delta`.  A program that churns
malloc-backed buffers has to allocate what the sweep freed all over again
before `malloc_increase` reaches `malloc_limit`, and each cycle frees more
than the last, so the interval grows without bound.

Before the monotonic counters this could not happen: the counter was
decremented on free and clamped at zero, so a burst of frees could cancel
the value it had accumulated but never create a debt.

Take the free baseline again at the end of sweeping.  It cannot be done at
the point of free instead: while a lazy sweep is in progress its frees must
keep reducing `malloc_increase`, because the malloc trigger sweeps and
retries on exactly that (objspace_malloc_increase_body), which is the
problem 45fd5d58f1 fixed.  The malloc baseline stays where 45fd5d58f1 put
it, so allocations made during lazy sweeping are still counted.  (Frees a
thread still holds in its malloc_increase_local cache slip past the
snapshot, bounded by GC_MALLOC_INCREASE_LOCAL_THRESHOLD per thread.)

A global GC needed one more piece: it sweeps every objspace but ran
gc_reset_malloc_info only on the driver, so advancing the others' free
baseline would overstate their malloc_increase by everything the global
sweep freed.  Give every objspace the same reset at global GC start.

Dropping 200,000 64 KiB buffers one at a time, MiB allocated between
consecutive GCs:

    200_000.times { s = String.new(capacity: 65536); s << "x" }

    before   66 GCs   14 36 31 31 32 31 ... 479 511 572 575   RSS 582 MB
    after   391 GCs   14 22 31 32 32 32 ...  32  32  32  32   RSS  10 MB

The live set is flat throughout, so nothing here should be making the
limit grow.  glibc does not hand a grown arena back, so the RSS stays for
the life of the process.  The loop itself costs 3.6s before and 4.0s
after.  The same loop split across two Ractors: 603 MB before, 44 MB
after.

puma with 16 workers serving a static response, total PSS across the
process tree at the plateau of a 30s run, median of three: 3239 MB before,
1348 MB after, throughput within run-to-run spread (59.1k and 57.1k req/s).

The case 45fd5d58f1 was written for is unaffected: its script still reports
count=196 minor=153 major=43, because sweeping frees almost nothing when
the objects stay live.  Workloads that allocate only heap slots are
unchanged down to the GC count.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Koichi SasadaandClaude Opus 5 committed 2026-08-14 09:09:24 +09:00
1 parent a9c656ebd2
commit e631daf17b
1 file changed
+35 -3
Notes: git 2026-08-14 00:09:55 +00:00
+35 -3
View File
@@ -563,7 +563,9 @@ struct gc_malloc_bytes {
gc_counter_t malloc;
gc_counter_t free;
/* Snapshots of `malloc` / `free` taken at the end of the last GC */
/* Baselines the increase is measured from: malloc snapshotted at GC start
* (gc_reset_malloc_info), free re-snapshotted at gc_sweep_finish so the
* sweep's own frees never count. */
gc_counter_t malloc_at_last_gc;
gc_counter_t free_at_last_gc;
};
@@ -1261,6 +1263,17 @@ gc_malloc_counters_increase_unsigned(rb_objspace_t *objspace, const struct gc_ma
return (size_t)inc;
}
/* Frees done while sweeping are the GC's own work, not the mutator's: advance
* free_at_last_gc past them so they cannot pay for the next cycle's allocation.
* malloc_at_last_gc stays at gc_reset_malloc_info's snapshot (GC start). */
static inline void
gc_malloc_counters_snapshot_free_at_last_gc(rb_objspace_t *objspace, struct gc_malloc_bytes *c)
{
MALLOC_COUNTERS_LOCK(objspace);
gc_counter_store_release(&c->free_at_last_gc, gc_counter_load_relaxed(&c->free));
MALLOC_COUNTERS_UNLOCK(objspace);
}
static inline void
gc_malloc_counters_snapshot(rb_objspace_t *objspace, struct gc_malloc_bytes *c)
{
@@ -4860,6 +4873,13 @@ gc_sweep_finish(rb_objspace_t *objspace)
}
}
/* Not before: while sweeping is in progress its frees must keep reducing
* malloc_increase (objspace_malloc_increase_body sweeps and retries on it). */
gc_malloc_counters_snapshot_free_at_last_gc(objspace, &objspace->malloc_counters.counters);
#if RGENGC_ESTIMATE_OLDMALLOC
gc_malloc_counters_snapshot_free_at_last_gc(objspace, &objspace->malloc_counters.oldcounters);
#endif
gc_event_hook(objspace, RUBY_INTERNAL_EVENT_GC_END_SWEEP);
gc_mode_transition(objspace, gc_mode_none);
}
@@ -7919,8 +7939,9 @@ gc_reset_malloc_info(rb_objspace_t *objspace, bool full_mark)
/* reset oldmalloc info */
#if RGENGC_ESTIMATE_OLDMALLOC
if (!full_mark) {
/* Don't snapshot on minor GC: oldmalloc_increase is meant to
* accumulate across minor GCs and only reset at major GC. */
/* No full snapshot on minor GC: oldmalloc_increase accumulates across
* minors and resets at major GC. (gc_sweep_finish still advances the
* free baseline after every sweep.) */
int64_t oldmalloc_increase = gc_malloc_counters_increase(objspace, &objspace->malloc_counters.oldcounters);
if (oldmalloc_increase > 0 &&
(uint64_t)oldmalloc_increase > objspace->rgengc.oldmalloc_increase_limit) {
@@ -8776,6 +8797,17 @@ gc_start_global(rb_objspace_t *driver, unsigned int reason, bool compact, bool a
global_objspace->global_gc.objspaces[i]->flags.during_global_gc = TRUE;
}
/* A global GC collects every objspace, so each needs the malloc-counter reset the
* driver got in gc_start_record; without it, gc_sweep_finish advancing free_at_last_gc
* (step 9) would leave their malloc_increase overstated by everything swept here. */
for (size_t i = 0; i < global_objspace->global_gc.n_objspaces; i++) {
rb_objspace_t *const os = global_objspace->global_gc.objspaces[i];
if (os != driver) {
os->profile.latest_gc_info = reason;
gc_reset_malloc_info(os, true);
}
}
/* step 3: settle every lazy sweep so the mark bits' meaning is fixed before the clear
* below. (during_gc is a macro over the local "objspace".) rb_gc_get_ec() resolves
* through objspace->vm_context during a GC, so initialize it for all: the driver