Starting with MALLOC_CONF=tcache:false and reading or writing
thread.tcache.max before first enabling the cache dereferences NULL.
The tcache_t embedded in TSD always exists, but tcache->tcache_slow is
assigned only by tcache_init(), which is skipped in this case. The
tcache_slow settings embedded in TSD are initialized independently.
Disabling an already initialized cache does not encounter this missing
link, since tcache_init() has already run.
Read and update the embedded settings through tsd_tcache_slowp_get().
This allows configuring the limit before the first enable without
initializing the cache as a side effect.
Add a fresh-thread regression covering the initial read, update while
disabled, and preservation of the limit on first enable. A standalone
public-API program also reproduces the original crash with tcache:false
against a release library.
percpu_arena_choose() returned an unclamped sched_getcpu(), so a process
confined to a subset of CPUs got arena indices past the end of the auto
range. Such arenas are invisible to the stats and ctl paths, which
iterate only to narenas_total_get(), and with a small narenas the index
could instead collide with huge_arena_ind and route ordinary
allocations into the oversize arena. Choose the arena through a table
built once at boot: CPUs the boot affinity mask allows are ranked so
each gets its own arena and every index stays inside the auto range.
Under LTO `&tsd_tls` is hoisted out of the inlined malloc/free and
survives swapcontext in a callee-saved register: a fiber resumed on
another OS thread races on the old thread's tcache.
Fix it by taking the address in a noinline volatile-asm barrier accessor.
It is done under option (--enable-experimental-fiber-safe-tls), because
it make the fastpath slower, on a microbench (that is very heavy depends
on the most optimal case) the overhead is 35%, but this is unlikely the
real world scenario.
Anyway here are some numbers for microbench.c compiled with LTO:
type |vanilla (ns)|`--enable-experimental-fiber-safe-tls` (ns)|optimized (ns) [1]
--------|------------|-------------------------------------------|------------------
malloc |6.51 |8.77 **(+34%)** |6.79 (4%)
mallocx |8.48 |10.62 **(25%)** |8.76 (3%)
free |6.46 |8.68 **(34%)** |6.76 (5%)
dallocx |8.190 |10.97 **(33%)** |8.74 (7%)
[1]: https://github.com/jemalloc/jemalloc/pull/2944
This patch also adds a separate lane for CI with LTO
Replace divisor-based small-bin refill control and fixed GC flushing
with per-bin fill and retention targets based on use between GC events.
Keep the bounds valid for all configured capacities, and ensure a
full-bin flush always frees a slot.
Simplify the configuration space by removing and deprecating
experimental_tcache_gc, lg_tcache_nslots_mul,
tcache_nslots_small_min, tcache_nslots_small_max, tcache_nslots_large,
tcache_gc_delay_bytes, lg_tcache_flush_small_div, and
lg_tcache_flush_large_div. Per-size-class cache capacity remains
configurable through tcache_ncached_max.
Due to an incorrect condition, a prof-promoted small allocation
was incorrectly wrapped in large-extent guard pages. This consumed
the large-guard budget, so the next genuinely large allocation that
should have been guarded got skipped.
The frameptr-based unwinder was an interim solution while issues with
libunwind were resolved. It's better if we remove it,
so we don't have to support it moving forward.
Add unit tests to cover public interface of pac decay & deferral
relevant functions. To align with the decopuling between arena and pac,
rename arena_decay to pac_decay here.
* Allow resuming per-CPU arena selection via thread.arena
With percpu_arena enabled, thread.arena control is one-directional. A
thread can be bound to a manually created arena (an index at or above the
per-CPU auto range) to route a bounded region of work to a dedicated,
long-lived arena, but there is no way back: thread_arena_ctl returns EPERM
for any index within the auto range, and arena_choose_impl only re-selects
a per-CPU arena for threads whose current arena is already in that range.
So once a thread is bound to a manual arena it stays pinned there forever,
and its later allocations land there instead of following the CPU.
Treat setting thread.arena to an index within the per-CPU range as a
request to resume automatic per-CPU selection: hand the thread back to
percpu management (rebinding it to the current CPU's arena via
percpu_arena_update) and return 0 instead of EPERM. Binding to a manual
arena is unchanged. The requested index is advisory; under percpu the
thread is governed by its current CPU, so it resumes on the current CPU's
arena regardless of the value passed.
Add test/unit/percpu_arena_resume covering the manual-arena to resume
round trip with allocs & dallocs happening meanwhile, and update
test_thread_arena, which asserted the old EPERM.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
free_sized() and free_aligned_sized() forward straight to sdallocx(), which
expects a non-NULL pointer and asserts on it in debug builds. C23 says both
should accept NULL and do nothing, like free(NULL) does, so a NULL argument
either trips that assert or feeds NULL into the dealloc path in release builds.
It is not hard to hit. glibc 2.41 ships free_sized()/free_aligned_sized(), and
a C++ sized delete of a null pointer compiles down to a free_sized() call. Once
jemalloc is preloaded its versions take over, and that NULL call takes down the
process. I ran into it with GTK4/GLib apps under LD_PRELOAD.
Check for NULL first, the way free() already does, and add an integration test
covering the NULL case for both functions.
While here, give free_aligned_sized() its own core.free_aligned_sized.entry
and .exit logging and call je_sdallocx_impl() directly rather than the
je_sdallocx() wrapper, so it mirrors free_sized() and no longer logs under
sdallocx. The C++ sized-delete paths (sizedDeleteImpl, alignedSizedDeleteImpl)
get the same treatment: log entry/exit unconditionally and guard the call with
likely(ptr != nullptr).
Add a small extent cache in front of the PAC ecaches. Allocs and dallocs
that fit are served from per-shard SEC bins without taking the ecache
mutex; overflow falls through to the backing ecaches, including
ecache_pinned for pinned extents.
The feature is gated behind experimental_pac_sec_nshards (default 0,
disabled). To support independent HPA and PAC SEC instances,
sec_alloc/sec_dalloc/sec_fill take an explicit shard argument, with HPA
and PAC using separate TSD shard slots.
The runtime option aborted on every OOM, breaking new(std::nothrow)
semantics. Replace with configure-time --enable-cxx-infallible-new
(default off): when on, throwing new aborts (size logged) and
nothrow returns null; when off, standard new_handler + bad_alloc /
null behavior is preserved. Under LTO the on-path lets the compiler
prove operator new is no-throw.
Pull the tcache-aware routing helpers out of arena into a layer that
sits directly below the public malloc interface:
arena_malloc -> malloc_dispatch_malloc
arena_palloc -> malloc_dispatch_palloc
arena_ralloc -> malloc_dispatch_ralloc
arena_dalloc* -> malloc_dispatch_dalloc*
arena_sdalloc* -> malloc_dispatch_sdalloc*
arena_dalloc_promoted -> malloc_dispatch_dalloc_promoted
The new module (malloc_dispatch.h, malloc_dispatch_inlines.h,
src/malloc_dispatch.c) owns the tcache-vs-fall-through decision; the
only consumer is jemalloc_internal_inlines_c.h. arena keeps a narrower
arena_prof_demote() for the sampled-allocation demotion path.
Some pages (e.g., hugetlb pages) cannot be purged, and should be
prioritized for reuse. A custom extent_alloc hook signals this by
OR'ing EXTENT_ALLOC_FLAG_PINNED into the low bits of the returned
pointer; jemalloc strips the flag bits and caches pinned extents in
a dedicated ecache_pinned, separate from the dirty/muzzy decay
pipeline.
Pinned extents do not coalesce eagerly, except for ones larger than
SC_LARGE_MINCLASS. A prefer-small policy reuses the smallest fitting
pinned extent, to avoid unnecessary split/fragmentation.
* Replace std::__throw_bad_alloc call with standard C++
Since December of 2025, std::__throw_bad_alloc is no longer visible
through #include <new> causing jemalloc build failures with gcc 16.
As far as I can tell, all std::__throw_bad_alloc did was arrange to
raise a std::bad_alloc exception if exceptions are enabled. I am not
sure whether its usage was truly meaningful in jemalloc since the call
is wrapped in a try catch and any usage of try catch is considered an
error when compiling with -fno-exceptions on gcc, at least.
This change adds a check to configure.ac that determines whether
exceptions are enabled by compiling a simple try catch that raises a
std::bad_alloc exception. If that test succeeds, the macro
JEMALLOC_HAVE_CXX_EXCEPTIONS is defined, and jemalloc will raise an
exception. Otherwise, we call std::terminate() to abort.
This was tested on FreeBSD with the gcc16 port with and without exceptions
enabled.
* Replace std::set_new_handler calls with std::get_new_handler
Previously, std::set_new_handler was used as a workaround for
compilers with only partial support for C++11. Now that C++14 is a
requirement to enable C++ support, we can assume std::get_new_handler
is available.
The second expansion attempt in large_ralloc_no_move omitted the !
before large_ralloc_no_move_expand(), inverting the return value.
On expansion failure, the function falsely reported success, making
callers believe the allocation was expanded in-place when it was not.
On expansion success, the function falsely reported failure, causing
callers to unnecessarily allocate, copy, and free.
Add unit test that verifies the return value matches actual size change.
tsd_tcache_data_init() returns true on failure but its callers ignore
this return value, leaving the per-thread tcache in an uninitialized
state after a failure.
This change disables the tcache on an initialization failure and logs
an error message. If opt_abort is true, it will also abort.
New unit tests have been added to test tcache initialization failures.
YANG XIN