For valid power-of-two alignments, use `lg_floor()` to encode the
alignment flags in the C++ aligned sized delete helper. This avoids the
public macro's split-word calculation when alignment is a runtime
argument.
prof_recent_alloc_reset() called prof_sys_thread_name_fetch() before the
reentrancy check in prof_tctx_create(), and prof_tdata_get(tsd, true)
asserts the level is 0. A free of a recorded allocation from within a
prof_sample hook aborted debug builds. Guard the fetch the same way.
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Add experimental_hpa_sec_trim, disabled by default, to trim SEC during
background HPA maintenance. Return half of each bin's minimum observed
inventory, rounded up to whole extents, selecting cold extents first.
Keep the original SEC overflow policy and leave PAC SEC policy unchanged.
The dump reads other threads' tdata->thread_name with no lock in common
with the owner's write. buf_writer_cb() does strlen() then memcpy(), so a
shrink in between embeds a NUL and prof_dump_flush() drops the rest of
the buffer.
Writers take tdata->lock; the heap dump reads under it (tdatas_mtx ->
tdata->lock order already exists). prof_try_log() already holds it.
prof_recent copies names into the node at record time instead, since
prof_recent_dump_mtx ranks above tdata->lock.
Otherwise it may lead to a broken profile output:
$ grep -C3 -r VFSRead2c | cat -A
clickhouse.jemalloc.704.4.m4.heap- t454: 1: 1301078 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap- t455: 0: 0 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap- t456: 1: 1301078 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap: t457: 1: 14316 [0: 0] VFSRead2c 0xff2c2361b79c$ # <-- this one
clickhouse.jemalloc.704.4.m4.heap- t*: 2: 128 [0: 0]$
clickhouse.jemalloc.704.4.m4.heap- t313: 2: 128 [0: 0]$
clickhouse.jemalloc.704.4.m4.heap- f: 28564753249 56 64 4 3 313$
Retain 49 bits in compact x86_64 edata pointers when LG_VADDR is 48,
while keeping radix-tree indexing and geometry at the configured width.
This preserves both native sign-extended 48-bit pointers and
zero-extended 48-bit pointers supplied to x86_64 binaries under QEMU
user-mode emulation on ARM64 hosts.
hpa_purge_threshold_ratio limits partial purging to avoid splitting
occupied hugepages. Applying the same threshold to non-hugified
pageslabs can strand dirty memory above the shard's hpa_dirty_mult
budget. The threshold is based on the full hugepage size, so sparsely
populated slabs can retain substantial dirty memory relative to their
active allocations.
Apply the threshold only to pageslabs HPA considers huge, allowing dirty
pages in non-hugified pageslabs to be reclaimed under the existing dirty
budget and purge timing controls.
hpa_purge_threshold_ratio limits partial purging of occupied hugepages, since purging part of a hugepage can cause the kernel to split it. Splitting can be costly, and restoring hugepage backing is not guaranteed. Applying the same threshold to pageslabs HPA considers non-huge can strand dirty memory above hpa_dirty_mult without protecting a known hugepage. The threshold is based on the full hugepage size, so this is particularly costly for sparse slabs with mostly untouched pages.
This PR changes the purging condition to have the threshold only apply to pageslabs HPA considers huge. Non-hugified dirty pages can then be reclaimed under the existing hpa_dirty_mult budget and timing controls.
Reclaiming those pages can also unblock hugification of other pageslabs in the shard. Before hugifying a candidate, HPA checks that the shard's dirty pages plus the candidate's retained pages do not exceed hpa_dirty_mult times the shard's active pages. Dirty memory stranded in another pageslab can therefore block an otherwise eligible candidate; purging it reduces the shard's dirty count and can allow hugification to proceed.
CPU migration can switch the arena used for tcache refills, leaving partially consumed slabs in multiple arenas. Slab starts can therefore outnumber slab ends before the guarded small allocation test has observed enough of both.
Bound the recorded starts and wait for at least the requested number of ends while retaining guard and slab separation checks. Also free exactly the pointers allocated by each invocation to avoid freeing a stale entry from an earlier invocation.
Use a manual arena as the baseline for the thread.arena invalid-output test so automatic per-CPU selection cannot change it between calls. Retain all partial-output and rejected-migration checks, and restore the original thread arena afterward.
When the first slab supplies nfill_min objects after clipping, place the
batch at the end of the cache stack and update the pointer array length.
This lets cache_bin_finish_fill skip memmove while preserving the existing
fill-count policy and multi-slab fallback.
Keep the optimization within the existing clipping branch.
Expand the github test jobs to test the dynamic page size feature.
Linux x64 will have the best coverage (+42 jobs).
Linux arm64, FreeBSD, MacOS will have basic coverage (+2 or +4 jobs).
Non-tree batch allocation restarts its bitmap search at group zero after
each group. Pass a lower bound to bitmap_ffu_group() so subsequent
iterations skip the exhausted prefix.
Keep the cursor local to each batch and assert that a group is exhausted
before continuing. Tree bitmaps retain their summary-based search.
timer_scale multiplies the tcache GC interval, dirty/muzzy decay
epochs, HPA min purge interval and hugify delay, and the background
thread minimum sleep. Set it with MALLOC_CONF=timer_scale:N (1..1000,
default 1; opt.timer_scale) or at run time via the "timer_scale"
mallctl. Useful when the whole process runs uniformly slower, e.g.
under heavy profiling or instrumentation.
Decay samples timer_scale once per epoch; reported decay_ms values are
unscaled.
bin_slab_reg_alloc_batch allocated each region with bitmap_sfu when the
bitmap was a tree, requiring a root-to-leaf walk and fullness
propagation for every region.
Find the first non-full bottom-level group once, take regions from it
with cfs_lu, then store and propagate once per group. The same loop
replaces the separate non-tree popcount path.
Extract helpers for finding the first non-full bottom-level group and
for storing an updated group while propagating fullness up the tree.
Leave bitmap_sfu unchanged to preserve the single-region allocation
path.
cfs_lu cleared the bit it found with x ^= 1 << ffs(x), so each
step of a loop that drains a bitmap waited on the ffs result.
x & (x - 1) clears the same bit without depending on ffs, cutting
the loop-carried chain in bin_slab_reg_alloc_batch's non-tree path
from tzcnt + btc to a single blsr on x86 (sub + and on arm64).
je_dallocx() kept the flags != 0 path inline, so the whole function got
one stack frame and register assignment sized for the slow path, and the
flags == 0 fast path paid for it: callee-saved register spills, and with
clang after the emap_alloc_ctx_lookup() L1 inlining, an extra register
move.
Move the flags != 0 body into a noinline dallocx_default(), matching
free_default(), sdallocx_default() and mallocx_default().
dallocx(ptr, 0) now has the same fast path as free(): no frame setup and
no spills. flags != 0 callers pay one extra call.
Instructions retired per mallocx() + dallocx(ptr, 0) pair on aarch64:
gcc -O3, libjemalloc.so: 103.6 -> 98.6
clang ThinLTO, libjemalloc.so: 117.5 -> 105.5
clang ThinLTO, statically linked: 106.5 -> 82.5
On x86-64 with clang ThinLTO, the dallocx(ptr, 0) path goes from 57 to
37 instructions.
With large size classes disabled, multiplying curlextents by the size
class upper bound overstates allocated bytes in malloc_stats_print.
Read the existing active_bytes from the ctl snapshot for the allocated
column and accumulate those bytes when merging arena snapshots.
Exercise page rounding through the existing JSON output buffer and check
that large detail rows sum to large.allocated.
- Inline the L1 cache hit in emap_alloc_ctx_lookup() and move the slowpath
out of line to rtree_read_slow().
- Avoid dereferencing edata for slab allocations in emap_alloc_ctx_lookup().
- Optimize je_sdallocx_impl() by checking flags == 0 for fastpath.
Starting with MALLOC_CONF=tcache:false and reading or writing
thread.tcache.max before first enabling the cache dereferences NULL.
The tcache_t embedded in TSD always exists, but tcache->tcache_slow is
assigned only by tcache_init(), which is skipped in this case. The
tcache_slow settings embedded in TSD are initialized independently.
Disabling an already initialized cache does not encounter this missing
link, since tcache_init() has already run.
Read and update the embedded settings through tsd_tcache_slowp_get().
This allows configuring the limit before the first enable without
initializing the cache as a side effect.
Add a fresh-thread regression covering the initial read, update while
disabled, and preservation of the limit on first enable. A standalone
public-API program also reproduces the original crash with tcache:false
against a release library.
The single-line '#if !defined(__has_include) || __has_include(...)'
is rejected by compilers that predate the operator (GCC < 5): the
preprocessor must parse the whole expression, the unknown identifier
is replaced with 0, and '0 (<header>)' is a syntax error. Nest the
#ifs so __has_include only appears where it is known to be defined.
Fixes#3022.
PAC SEC bypasses the ecache deallocation path that clears the zeroed flag. Clear it before caching a used extent so large allocations with MALLOCX_ZERO are zeroed when reusing SEC entries below the madvise threshold.
Add a regression that writes to an extent, verifies a subsequent zeroed allocation hits SEC, and checks the returned bytes.
Slobodan Predolac