For valid power-of-two alignments, use `lg_floor()` to encode the
alignment flags in the C++ aligned sized delete helper. This avoids the
public macro's split-word calculation when alignment is a runtime
argument.
prof_recent_alloc_reset() called prof_sys_thread_name_fetch() before the
reentrancy check in prof_tctx_create(), and prof_tdata_get(tsd, true)
asserts the level is 0. A free of a recorded allocation from within a
prof_sample hook aborted debug builds. Guard the fetch the same way.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Add experimental_hpa_sec_trim, disabled by default, to trim SEC during
background HPA maintenance. Return half of each bin's minimum observed
inventory, rounded up to whole extents, selecting cold extents first.
Keep the original SEC overflow policy and leave PAC SEC policy unchanged.
The dump reads other threads' tdata->thread_name with no lock in common
with the owner's write. buf_writer_cb() does strlen() then memcpy(), so a
shrink in between embeds a NUL and prof_dump_flush() drops the rest of
the buffer.
Writers take tdata->lock; the heap dump reads under it (tdatas_mtx ->
tdata->lock order already exists). prof_try_log() already holds it.
prof_recent copies names into the node at record time instead, since
prof_recent_dump_mtx ranks above tdata->lock.
Otherwise it may lead to a broken profile output:
$ grep -C3 -r VFSRead2c | cat -A
clickhouse.jemalloc.704.4.m4.heap- t454: 1: 1301078 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap- t455: 0: 0 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap- t456: 1: 1301078 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap: t457: 1: 14316 [0: 0] VFSRead2c 0xff2c2361b79c$ # <-- this one
clickhouse.jemalloc.704.4.m4.heap- t*: 2: 128 [0: 0]$
clickhouse.jemalloc.704.4.m4.heap- t313: 2: 128 [0: 0]$
clickhouse.jemalloc.704.4.m4.heap- f: 28564753249 56 64 4 3 313$
Retain 49 bits in compact x86_64 edata pointers when LG_VADDR is 48,
while keeping radix-tree indexing and geometry at the configured width.
This preserves both native sign-extended 48-bit pointers and
zero-extended 48-bit pointers supplied to x86_64 binaries under QEMU
user-mode emulation on ARM64 hosts.
hpa_purge_threshold_ratio limits partial purging to avoid splitting
occupied hugepages. Applying the same threshold to non-hugified
pageslabs can strand dirty memory above the shard's hpa_dirty_mult
budget. The threshold is based on the full hugepage size, so sparsely
populated slabs can retain substantial dirty memory relative to their
active allocations.
Apply the threshold only to pageslabs HPA considers huge, allowing dirty
pages in non-hugified pageslabs to be reclaimed under the existing dirty
budget and purge timing controls.
hpa_purge_threshold_ratio limits partial purging of occupied hugepages, since purging part of a hugepage can cause the kernel to split it. Splitting can be costly, and restoring hugepage backing is not guaranteed. Applying the same threshold to pageslabs HPA considers non-huge can strand dirty memory above hpa_dirty_mult without protecting a known hugepage. The threshold is based on the full hugepage size, so this is particularly costly for sparse slabs with mostly untouched pages.
This PR changes the purging condition to have the threshold only apply to pageslabs HPA considers huge. Non-hugified dirty pages can then be reclaimed under the existing hpa_dirty_mult budget and timing controls.
Reclaiming those pages can also unblock hugification of other pageslabs in the shard. Before hugifying a candidate, HPA checks that the shard's dirty pages plus the candidate's retained pages do not exceed hpa_dirty_mult times the shard's active pages. Dirty memory stranded in another pageslab can therefore block an otherwise eligible candidate; purging it reduces the shard's dirty count and can allow hugification to proceed.
CPU migration can switch the arena used for tcache refills, leaving partially consumed slabs in multiple arenas. Slab starts can therefore outnumber slab ends before the guarded small allocation test has observed enough of both.
Bound the recorded starts and wait for at least the requested number of ends while retaining guard and slab separation checks. Also free exactly the pointers allocated by each invocation to avoid freeing a stale entry from an earlier invocation.
Use a manual arena as the baseline for the thread.arena invalid-output test so automatic per-CPU selection cannot change it between calls. Retain all partial-output and rejected-migration checks, and restore the original thread arena afterward.
Non-tree batch allocation restarts its bitmap search at group zero after
each group. Pass a lower bound to bitmap_ffu_group() so subsequent
iterations skip the exhausted prefix.
Keep the cursor local to each batch and assert that a group is exhausted
before continuing. Tree bitmaps retain their summary-based search.
timer_scale multiplies the tcache GC interval, dirty/muzzy decay
epochs, HPA min purge interval and hugify delay, and the background
thread minimum sleep. Set it with MALLOC_CONF=timer_scale:N (1..1000,
default 1; opt.timer_scale) or at run time via the "timer_scale"
mallctl. Useful when the whole process runs uniformly slower, e.g.
under heavy profiling or instrumentation.
Decay samples timer_scale once per epoch; reported decay_ms values are
unscaled.
bin_slab_reg_alloc_batch allocated each region with bitmap_sfu when the
bitmap was a tree, requiring a root-to-leaf walk and fullness
propagation for every region.
Find the first non-full bottom-level group once, take regions from it
with cfs_lu, then store and propagate once per group. The same loop
replaces the separate non-tree popcount path.
With large size classes disabled, multiplying curlextents by the size
class upper bound overstates allocated bytes in malloc_stats_print.
Read the existing active_bytes from the ctl snapshot for the allocated
column and accumulate those bytes when merging arena snapshots.
Exercise page rounding through the existing JSON output buffer and check
that large detail rows sum to large.allocated.
Starting with MALLOC_CONF=tcache:false and reading or writing
thread.tcache.max before first enabling the cache dereferences NULL.
The tcache_t embedded in TSD always exists, but tcache->tcache_slow is
assigned only by tcache_init(), which is skipped in this case. The
tcache_slow settings embedded in TSD are initialized independently.
Disabling an already initialized cache does not encounter this missing
link, since tcache_init() has already run.
Read and update the embedded settings through tsd_tcache_slowp_get().
This allows configuring the limit before the first enable without
initializing the cache as a side effect.
Add a fresh-thread regression covering the initial read, update while
disabled, and preservation of the limit on first enable. A standalone
public-API program also reproduces the original crash with tcache:false
against a release library.
PAC SEC bypasses the ecache deallocation path that clears the zeroed flag. Clear it before caching a used extent so large allocations with MALLOCX_ZERO are zeroed when reusing SEC entries below the madvise threshold.
Add a regression that writes to an extent, verifies a subsequent zeroed allocation hits SEC, and checks the returned bytes.
The shortcut produced stale arena choices for 10% and 37% of
selections in two production workloads. The current CPU lookup took at
most 7 cycles in a Linux microbenchmark.
Always read the current CPU and test migration between CPUs.
With profiling active, xallocx may decline to grow an allocation that is
not page-aligned. The profiling path can require sampled xallocx growth
to use page-aligned pointers; otherwise it can return the old usable
size without committing the purged tail.
Observed on Darwin, where decommit/commit are real VM state changes.
The shrink/purge half succeeds and records a decommit, then the
profiling-limited grow skips the matching commit.
This is not Darwin-specific. Linux no-overcommit should expose the same
mismatch. Usual Linux overcommit behavior papers it over by making
commit/decommit hooks report unsupported, so both sides of the old
equality check stay false.
Relax the default-path assertion only while profiling is active and we
can prove xallocx hasn't reached hooks. Add a page-aligned case that
always requires xallocx shrink and grow to succeed, preserving
commit/merge coverage.
Assisted-by: Codex gpt-5.5 xhigh
percpu_arena_choose() returned an unclamped sched_getcpu(), so a process
confined to a subset of CPUs got arena indices past the end of the auto
range. Such arenas are invisible to the stats and ctl paths, which
iterate only to narenas_total_get(), and with a small narenas the index
could instead collide with huge_arena_ind and route ordinary
allocations into the oversize arena. Choose the arena through a table
built once at boot: CPUs the boot affinity mask allows are ranked so
each gets its own arena and every index stays inside the auto range.
Under LTO `&tsd_tls` is hoisted out of the inlined malloc/free and
survives swapcontext in a callee-saved register: a fiber resumed on
another OS thread races on the old thread's tcache.
Fix it by taking the address in a noinline volatile-asm barrier accessor.
It is done under option (--enable-experimental-fiber-safe-tls), because
it make the fastpath slower, on a microbench (that is very heavy depends
on the most optimal case) the overhead is 35%, but this is unlikely the
real world scenario.
Anyway here are some numbers for microbench.c compiled with LTO:
type |vanilla (ns)|`--enable-experimental-fiber-safe-tls` (ns)|optimized (ns) [1]
--------|------------|-------------------------------------------|------------------
malloc |6.51 |8.77 **(+34%)** |6.79 (4%)
mallocx |8.48 |10.62 **(25%)** |8.76 (3%)
free |6.46 |8.68 **(34%)** |6.76 (5%)
dallocx |8.190 |10.97 **(33%)** |8.74 (7%)
[1]: https://github.com/jemalloc/jemalloc/pull/2944
This patch also adds a separate lane for CI with LTO
This prevents touch static buffer as well as providing better tailored
format detections. Also change the return type of pid namespace from
long to uint64_t to be more robust.
Replace divisor-based small-bin refill control and fixed GC flushing
with per-bin fill and retention targets based on use between GC events.
Keep the bounds valid for all configured capacities, and ensure a
full-bin flush always frees a slot.
Simplify the configuration space by removing and deprecating
experimental_tcache_gc, lg_tcache_nslots_mul,
tcache_nslots_small_min, tcache_nslots_small_max, tcache_nslots_large,
tcache_gc_delay_bytes, lg_tcache_flush_small_div, and
lg_tcache_flush_large_div. Per-size-class cache capacity remains
configurable through tcache_ncached_max.
Under heavy cpu load the background thread starves and
it doesn't purge in the required time interval. The fix is
to wake it up early so that it can make some progress.
Due to an incorrect condition, a prof-promoted small allocation
was incorrectly wrapped in large-extent guard pages. This consumed
the large-guard budget, so the next genuinely large allocation that
should have been guarded got skipped.
src/stats.c fused mallctl reads with dual-format (text/JSON) emission inside a
few very large, bottom-up functions, behind dense macros, with the public entry
point at the very bottom. Rework it to be output-preserving but much easier to
read and extend:
- Separate gathering from emission where it is worth it: most sections become a
small gather (stats_gather_*, the mallctl reads into a typed struct) plus an
emit that renders it, with the plumbing macros (CTL_*, COL*) and gather
structs moved to internal header (stats_internal.h). This is a balance,
not a rule: the per-size-class tables stay O(1) streaming (per-row gather+emit,
no large stack buffers, avoiding heap buffering), the mutex rows keep reading
straight into emitter columns, and the already-readable config/opt probes are
left as they are.
- Read like the output: open with a table-of-contents comment and the public
stats_print(), then the section functions top-down, keeping the
interval/boot/fork plumbing at the bottom. The ToC names the function that
prints each section, and its JSON sub-path.
- Keep one emission engine: the emitter remains the single dual-format renderer,
so each section has one rendering path rather than parallel renderers (the two
formats are not field-identical). Move some table-only concerns into it --
gap-collapsing ("---") and a dual columnar row op (emitter_row) -- so section
code carries less table-format detail.
Output-preserving: text is byte-identical and JSON keys/values are unchanged.
The frameptr-based unwinder was an interim solution while issues with
libunwind were resolved. It's better if we remove it,
so we don't have to support it moving forward.
New tests cover:
- background thread states after arena_reset
- background thread stats
- toggle background thread on and off during parallel stress
allocs/dallocs
- background thread fork behavior
Add background-thread fork unit test
test_fork_background_thread (in fork.c): with the background thread enabled,
fork and assert the child comes up with it disabled yet usable (re-enable +
alloc round-trips), while the parent keeps its threads. Placed in fork.c to
reuse its wait_for_child_exit helper and fork/WIN32 handling instead of
duplicating them in a separate file.
Add unit tests to cover public interface of pac decay & deferral
relevant functions. To align with the decopuling between arena and pac,
rename arena_decay to pac_decay here.
Currently there are deferral-relevant constants in arena.c and
background_thread.c. This commit moves them out into corresponding PAC
and HPA headers. Those commonly needed by both are put into a new header
deferral.h where the contract of deferral are clearly stated.
* Allow resuming per-CPU arena selection via thread.arena
With percpu_arena enabled, thread.arena control is one-directional. A
thread can be bound to a manually created arena (an index at or above the
per-CPU auto range) to route a bounded region of work to a dedicated,
long-lived arena, but there is no way back: thread_arena_ctl returns EPERM
for any index within the auto range, and arena_choose_impl only re-selects
a per-CPU arena for threads whose current arena is already in that range.
So once a thread is bound to a manual arena it stays pinned there forever,
and its later allocations land there instead of following the CPU.
Treat setting thread.arena to an index within the per-CPU range as a
request to resume automatic per-CPU selection: hand the thread back to
percpu management (rebinding it to the current CPU's arena via
percpu_arena_update) and return 0 instead of EPERM. Binding to a manual
arena is unchanged. The requested index is advisory; under percpu the
thread is governed by its current CPU, so it resumes on the current CPU's
arena regardless of the value passed.
Add test/unit/percpu_arena_resume covering the manual-arena to resume
round trip with allocs & dallocs happening meanwhile, and update
test_thread_arena, which asserted the old EPERM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This only changes behavior for the tsd_generic path, where
pthread_getspecific() can report no TSD after the pthread key destructor
has finished. Other TSD backends keep tsd_teardown_done() as a constant
false, so the added checks compile out there.
Avoid recreating TSD for late deallocations. Preserve existing
reincarnation behavior for late allocations and nonzero reallocations.
Add Linux CI coverage for force_tls=0, with and without --enable-debug,
to exercise the Android-equivalent generic TSD path.
Slobodan Predolac