Add experimental_hpa_sec_trim, disabled by default, to trim SEC during
background HPA maintenance. Return half of each bin's minimum observed
inventory, rounded up to whole extents, selecting cold extents first.
Keep the original SEC overflow policy and leave PAC SEC policy unchanged.
The dump reads other threads' tdata->thread_name with no lock in common
with the owner's write. buf_writer_cb() does strlen() then memcpy(), so a
shrink in between embeds a NUL and prof_dump_flush() drops the rest of
the buffer.
Writers take tdata->lock; the heap dump reads under it (tdatas_mtx ->
tdata->lock order already exists). prof_try_log() already holds it.
prof_recent copies names into the node at record time instead, since
prof_recent_dump_mtx ranks above tdata->lock.
Otherwise it may lead to a broken profile output:
$ grep -C3 -r VFSRead2c | cat -A
clickhouse.jemalloc.704.4.m4.heap- t454: 1: 1301078 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap- t455: 0: 0 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap- t456: 1: 1301078 [0: 0] ThreadPool$
clickhouse.jemalloc.704.4.m4.heap: t457: 1: 14316 [0: 0] VFSRead2c 0xff2c2361b79c$ # <-- this one
clickhouse.jemalloc.704.4.m4.heap- t*: 2: 128 [0: 0]$
clickhouse.jemalloc.704.4.m4.heap- t313: 2: 128 [0: 0]$
clickhouse.jemalloc.704.4.m4.heap- f: 28564753249 56 64 4 3 313$
Retain 49 bits in compact x86_64 edata pointers when LG_VADDR is 48,
while keeping radix-tree indexing and geometry at the configured width.
This preserves both native sign-extended 48-bit pointers and
zero-extended 48-bit pointers supplied to x86_64 binaries under QEMU
user-mode emulation on ARM64 hosts.
hpa_purge_threshold_ratio limits partial purging to avoid splitting
occupied hugepages. Applying the same threshold to non-hugified
pageslabs can strand dirty memory above the shard's hpa_dirty_mult
budget. The threshold is based on the full hugepage size, so sparsely
populated slabs can retain substantial dirty memory relative to their
active allocations.
Apply the threshold only to pageslabs HPA considers huge, allowing dirty
pages in non-hugified pageslabs to be reclaimed under the existing dirty
budget and purge timing controls.
hpa_purge_threshold_ratio limits partial purging of occupied hugepages, since purging part of a hugepage can cause the kernel to split it. Splitting can be costly, and restoring hugepage backing is not guaranteed. Applying the same threshold to pageslabs HPA considers non-huge can strand dirty memory above hpa_dirty_mult without protecting a known hugepage. The threshold is based on the full hugepage size, so this is particularly costly for sparse slabs with mostly untouched pages.
This PR changes the purging condition to have the threshold only apply to pageslabs HPA considers huge. Non-hugified dirty pages can then be reclaimed under the existing hpa_dirty_mult budget and timing controls.
Reclaiming those pages can also unblock hugification of other pageslabs in the shard. Before hugifying a candidate, HPA checks that the shard's dirty pages plus the candidate's retained pages do not exceed hpa_dirty_mult times the shard's active pages. Dirty memory stranded in another pageslab can therefore block an otherwise eligible candidate; purging it reduces the shard's dirty count and can allow hugification to proceed.
Non-tree batch allocation restarts its bitmap search at group zero after
each group. Pass a lower bound to bitmap_ffu_group() so subsequent
iterations skip the exhausted prefix.
Keep the cursor local to each batch and assert that a group is exhausted
before continuing. Tree bitmaps retain their summary-based search.
timer_scale multiplies the tcache GC interval, dirty/muzzy decay
epochs, HPA min purge interval and hugify delay, and the background
thread minimum sleep. Set it with MALLOC_CONF=timer_scale:N (1..1000,
default 1; opt.timer_scale) or at run time via the "timer_scale"
mallctl. Useful when the whole process runs uniformly slower, e.g.
under heavy profiling or instrumentation.
Decay samples timer_scale once per epoch; reported decay_ms values are
unscaled.
Extract helpers for finding the first non-full bottom-level group and
for storing an updated group while propagating fullness up the tree.
Leave bitmap_sfu unchanged to preserve the single-region allocation
path.
cfs_lu cleared the bit it found with x ^= 1 << ffs(x), so each
step of a loop that drains a bitmap waited on the ffs result.
x & (x - 1) clears the same bit without depending on ffs, cutting
the loop-carried chain in bin_slab_reg_alloc_batch's non-tree path
from tzcnt + btc to a single blsr on x86 (sub + and on arm64).
With large size classes disabled, multiplying curlextents by the size
class upper bound overstates allocated bytes in malloc_stats_print.
Read the existing active_bytes from the ctl snapshot for the allocated
column and accumulate those bytes when merging arena snapshots.
Exercise page rounding through the existing JSON output buffer and check
that large detail rows sum to large.allocated.
- Inline the L1 cache hit in emap_alloc_ctx_lookup() and move the slowpath
out of line to rtree_read_slow().
- Avoid dereferencing edata for slab allocations in emap_alloc_ctx_lookup().
- Optimize je_sdallocx_impl() by checking flags == 0 for fastpath.
The single-line '#if !defined(__has_include) || __has_include(...)'
is rejected by compilers that predate the operator (GCC < 5): the
preprocessor must parse the whole expression, the unknown identifier
is replaced with 0, and '0 (<header>)' is a syntax error. Nest the
#ifs so __has_include only appears where it is known to be defined.
Fixes#3022.
The current code does not define LG_QUANTUM for SH-3, only SH-3E.
Build tested on a SuperH-3 baseline sysroot.
Signed-off-by: Mingcong Bai <jeffbai@aosc.io>
The shortcut produced stale arena choices for 10% and 37% of
selections in two production workloads. The current CPU lookup took at
most 7 cycles in a Linux microbenchmark.
Always read the current CPU and test migration between CPUs.
percpu_arena_choose() returned an unclamped sched_getcpu(), so a process
confined to a subset of CPUs got arena indices past the end of the auto
range. Such arenas are invisible to the stats and ctl paths, which
iterate only to narenas_total_get(), and with a small narenas the index
could instead collide with huge_arena_ind and route ordinary
allocations into the oversize arena. Choose the arena through a table
built once at boot: CPUs the boot affinity mask allows are ranked so
each gets its own arena and every index stays inside the auto range.
Under LTO `&tsd_tls` is hoisted out of the inlined malloc/free and
survives swapcontext in a callee-saved register: a fiber resumed on
another OS thread races on the old thread's tcache.
Fix it by taking the address in a noinline volatile-asm barrier accessor.
It is done under option (--enable-experimental-fiber-safe-tls), because
it make the fastpath slower, on a microbench (that is very heavy depends
on the most optimal case) the overhead is 35%, but this is unlikely the
real world scenario.
Anyway here are some numbers for microbench.c compiled with LTO:
type |vanilla (ns)|`--enable-experimental-fiber-safe-tls` (ns)|optimized (ns) [1]
--------|------------|-------------------------------------------|------------------
malloc |6.51 |8.77 **(+34%)** |6.79 (4%)
mallocx |8.48 |10.62 **(25%)** |8.76 (3%)
free |6.46 |8.68 **(34%)** |6.76 (5%)
dallocx |8.190 |10.97 **(33%)** |8.74 (7%)
[1]: https://github.com/jemalloc/jemalloc/pull/2944
This patch also adds a separate lane for CI with LTO
This prevents touch static buffer as well as providing better tailored
format detections. Also change the return type of pid namespace from
long to uint64_t to be more robust.
configure.ac force-disables JEMALLOC_LAZY_LOCK on every Windows build
already, so there is no need to do another _WIN32 check here after the
JEMALLOC_LAZY_LOCK check. Same for JEMALLOC_MAPS_COLESCE and
JEMALLOC_PURGE_MADVISE_DONTNEED_ZEROS.
Replace divisor-based small-bin refill control and fixed GC flushing
with per-bin fill and retention targets based on use between GC events.
Keep the bounds valid for all configured capacities, and ensure a
full-bin flush always frees a slot.
Simplify the configuration space by removing and deprecating
experimental_tcache_gc, lg_tcache_nslots_mul,
tcache_nslots_small_min, tcache_nslots_small_max, tcache_nslots_large,
tcache_gc_delay_bytes, lg_tcache_flush_small_div, and
lg_tcache_flush_large_div. Per-size-class cache capacity remains
configurable through tcache_ncached_max.
Due to an incorrect condition, a prof-promoted small allocation
was incorrectly wrapped in large-extent guard pages. This consumed
the large-guard budget, so the next genuinely large allocation that
should have been guarded got skipped.
Tony Printezis