Under LTO `&tsd_tls` is hoisted out of the inlined malloc/free and
survives swapcontext in a callee-saved register: a fiber resumed on
another OS thread races on the old thread's tcache.
Fix it by taking the address in a noinline volatile-asm barrier accessor.
It is done under option (--enable-experimental-fiber-safe-tls), because
it make the fastpath slower, on a microbench (that is very heavy depends
on the most optimal case) the overhead is 35%, but this is unlikely the
real world scenario.
Anyway here are some numbers for microbench.c compiled with LTO:
type |vanilla (ns)|`--enable-experimental-fiber-safe-tls` (ns)|optimized (ns) [1]
--------|------------|-------------------------------------------|------------------
malloc |6.51 |8.77 **(+34%)** |6.79 (4%)
mallocx |8.48 |10.62 **(25%)** |8.76 (3%)
free |6.46 |8.68 **(34%)** |6.76 (5%)
dallocx |8.190 |10.97 **(33%)** |8.74 (7%)
[1]: https://github.com/jemalloc/jemalloc/pull/2944
This patch also adds a separate lane for CI with LTO
This prevents touch static buffer as well as providing better tailored
format detections. Also change the return type of pid namespace from
long to uint64_t to be more robust.
configure.ac force-disables JEMALLOC_LAZY_LOCK on every Windows build
already, so there is no need to do another _WIN32 check here after the
JEMALLOC_LAZY_LOCK check. Same for JEMALLOC_MAPS_COLESCE and
JEMALLOC_PURGE_MADVISE_DONTNEED_ZEROS.
Replace divisor-based small-bin refill control and fixed GC flushing
with per-bin fill and retention targets based on use between GC events.
Keep the bounds valid for all configured capacities, and ensure a
full-bin flush always frees a slot.
Simplify the configuration space by removing and deprecating
experimental_tcache_gc, lg_tcache_nslots_mul,
tcache_nslots_small_min, tcache_nslots_small_max, tcache_nslots_large,
tcache_gc_delay_bytes, lg_tcache_flush_small_div, and
lg_tcache_flush_large_div. Per-size-class cache capacity remains
configurable through tcache_ncached_max.
Due to an incorrect condition, a prof-promoted small allocation
was incorrectly wrapped in large-extent guard pages. This consumed
the large-guard budget, so the next genuinely large allocation that
should have been guarded got skipped.
Migrates the mutex backend selection out of mutex.h/mutex.c, and the
background thread's condition-variable and signal-masking code out of
background_thread.c, onto the OS layer dispatcher pattern.
Introduce the `os.h` dispatcher pattern under
`include/jemalloc/internal/os/`, with no modules wired in yet.
- os.h - documents the os/<module>.h dispatcher / os/posix/
default / os/<os>/ override pattern that every
facility will follow.
- os/detect.h - JEMALLOC_OS_POSIX platform detection, shared by every
module dispatcher.
src/stats.c fused mallctl reads with dual-format (text/JSON) emission inside a
few very large, bottom-up functions, behind dense macros, with the public entry
point at the very bottom. Rework it to be output-preserving but much easier to
read and extend:
- Separate gathering from emission where it is worth it: most sections become a
small gather (stats_gather_*, the mallctl reads into a typed struct) plus an
emit that renders it, with the plumbing macros (CTL_*, COL*) and gather
structs moved to internal header (stats_internal.h). This is a balance,
not a rule: the per-size-class tables stay O(1) streaming (per-row gather+emit,
no large stack buffers, avoiding heap buffering), the mutex rows keep reading
straight into emitter columns, and the already-readable config/opt probes are
left as they are.
- Read like the output: open with a table-of-contents comment and the public
stats_print(), then the section functions top-down, keeping the
interval/boot/fork plumbing at the bottom. The ToC names the function that
prints each section, and its JSON sub-path.
- Keep one emission engine: the emitter remains the single dual-format renderer,
so each section has one rendering path rather than parallel renderers (the two
formats are not field-identical). Move some table-only concerns into it --
gap-collapsing ("---") and a dual columnar row op (emitter_row) -- so section
code carries less table-format detail.
Output-preserving: text is byte-identical and JSON keys/values are unchanged.
The frameptr-based unwinder was an interim solution while issues with
libunwind were resolved. It's better if we remove it,
so we don't have to support it moving forward.
Currently there are deferral-relevant constants in arena.c and
background_thread.c. This commit moves them out into corresponding PAC
and HPA headers. Those commonly needed by both are put into a new header
deferral.h where the contract of deferral are clearly stated.
This only changes behavior for the tsd_generic path, where
pthread_getspecific() can report no TSD after the pthread key destructor
has finished. Other TSD backends keep tsd_teardown_done() as a constant
false, so the added checks compile out there.
Avoid recreating TSD for late deallocations. Preserve existing
reincarnation behavior for late allocations and nonzero reallocations.
Add Linux CI coverage for force_tls=0, with and without --enable-debug,
to exercise the Android-equivalent generic TSD path.
Reformat the malloc_getcpu Apple comment to the block style and note that the
tpidr_el0/sidt scheme requires macOS 12+: macOS 11 and earlier kept the CPU
number in tpidrro_el0's low 3 bits (now retired). Cite xnu's
__TPIDR_CPU_NUM_MASK / MACHDEP_TPIDR_CPUNUM_MASK, kept in sync with
_os_cpu_number, as the source of truth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Apple branch read the CPU number from the low 3 bits of `tpidrro_el0`.
That layout no longer holds on Apple Silicon: those bits read back as 0, so
`malloc_getcpu` returned 0 for every thread. `percpu_arena` then funneled all
allocations into arena 0, or was disabled outright at init.
Read the CPU number the same way as libplatform's `_os_cpu_number`: the low 12
bits of `tpidr_el0` on arm64, or of the IDT base (via `sidt`) on x86. Also move
the Apple branch ahead of the `rdtscp` one so Apple x86 uses `sidt` rather than
`rdtscp`, whose `ecx` is not the CPU id under XNU.
https://github.com/apple-oss-distributions/xnu/blob/main/libsyscall/os/tsd.h
Verified on a 12-CPU M2 Pro: with `percpu_arena:percpu`, `opt.percpu_arena` now
stays `percpu` and `thread.arena` spreads across all 12 arenas {0..11}; the old
code collapsed every thread onto arena 0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
It is unused and no longer needed: hpa_dirty_mult, hpa_purge_threshold,
hpa_min_purge_interval_ms, and hpa_min_purge_delay_ms now provide finer
control over the purge rate.
9c1a484e embedded sec_t in pac_t (+~48B). Since pac sits before the hot
hpa/edata_cache fields in pa_shard_s, the larger pac_t shifts them and
regresses the free/extent paths ~2% on the fill-flush microbenchmark --
even though SEC is disabled by default and no SEC code runs; it is purely
a struct-layout effect.
Move pac to the last field so its growth shifts only the trailing
all_bins[] flex array. Pure reorder, no functional change.
Validated with the fill-flush microbenchmark (500 threads, 256 B,
MALLOC_CONF=tcache:true,prof:true), turbo pinned, interleaved medians:
on AMD EPYC 9D64 (x86-64) bad regresses ~2% and the reorder restores the
baseline; on ARM Neoverse-V2 (aarch64) there is no measurable regression
and the reorder is neutral.
Add a small extent cache in front of the PAC ecaches. Allocs and dallocs
that fit are served from per-shard SEC bins without taking the ecache
mutex; overflow falls through to the backing ecaches, including
ecache_pinned for pinned extents.
The feature is gated behind experimental_pac_sec_nshards (default 0,
disabled). To support independent HPA and PAC SEC instances,
sec_alloc/sec_dalloc/sec_fill take an explicit shard argument, with HPA
and PAC using separate TSD shard slots.
Per-bin SEC stats (bytes_cur, nmisses, nhits, ndalloc_flush,
ndalloc_noflush, noverfills) become atomic_zu_t fields directly inside
sec_bin_t. Writers continue to hold bin->mtx; sec_stats_merge now reads
them lock-free with ATOMIC_RELAXED instead of acquiring bin->mtx.
This removes a lock-rank reversal that would otherwise occur whenever
stats aggregation runs while holding arena->stats.mtx.
Veracode flags CWE-190/191 on the `size + mask` round-ups in
sz_s2u_compute_using_delta, sz_s2u_compute and sz_sa2u. We believe these are
false positives: the arithmetic is unsigned size_t (overflow is well-defined in
C, not UB), `size` is already bounded by the SC_LARGE_MAXCLASS guard, and the
masks are tiny, so the additions cannot actually overflow.
CodeQL's security-extended suite (CWE-190/191) does not flag these. Its
uncontrolled-arithmetic query only reports *signed* overflow ("unsigned overflow
is well-defined"), and its tainted-arithmetic query credits the existing
upper-bound guard. Veracode instead treats the public-API size/alignment as
tainted and ignores guards placed above the expression, so piling on more
pre-checks does not satisfy it.
Effect on the malloc/free hot path is zero instructions as verified with
clang -O3 and ThinLTO objdump
The runtime option aborted on every OOM, breaking new(std::nothrow)
semantics. Replace with configure-time --enable-cxx-infallible-new
(default off): when on, throwing new aborts (size logged) and
nothrow returns null; when off, standard new_handler + bad_alloc /
null behavior is preserved. Under LTO the on-path lets the compiler
prove operator new is no-throw.
Tony Printezis