Add `GetStatsTest.Latency` to guard against regressions where `DumpStats` or `DumpStatsInPbtxt` blocks for seconds (such as slow `/proc` or `/sys` reads under `pageheap_lock`).
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1008685385
batch to `freelist()` when `to_return == 0` instead of re-acquiring `lock_`
only to unlock and exit. In `ShardedTransferCacheManager::Plunder`, skip size
classes where `!should_use(size_class)`.
Preserves legacy behavior under `-DTCMALLOC_INTERNAL_LEGACY_LOCKING2`.
PiperOrigin-RevId: 1008616070
When `use_bitmap_for_small = true`, free objects are tracked in an outline
bitmap at the end of the span rather than an inline linked list, so neither
popping nor pushing touches object memory unless explicitly checked.
- Poison backing span memory with `0xA5` before `BuildFreelist` in `State` so
uninitialized outline bitmap words surface immediately.
- Track `allocated_idx` in `State` to verify that popped and pushed objects are
aligned, lie within `[0, objects_per_span * object_size)` (not overlapping
the outline tail bitmap), and are never allocated twice concurrently.
- Write per-object canary bytes on pop and verify them on push.
PiperOrigin-RevId: 1008456250
When contributing a hugepage tracker, the filler steps a 48-bit linear congruential generator via `ExponentialBiased::NextRandom` and samples 1% of trackers for VMA tagging. Taking the remainder of the raw generator state modulo 100 depends on its low two bits, which cycle with period 4 because 100 is a multiple of 4. Starting from a zero seed, only contributions at index 3 modulo 4 could ever be sampled, which excluded three quarters of contributed hugepages.
Extract high-order bits via `ExponentialBiased::GetRandom` before taking the remainder modulo 100 so all modulo-4 contribution indices are sampled uniformly, and initialize the generator state to 42.
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1008455755
Root Cause: ExtractStats, DumpStats, and DumpStatsInPbtxt called
UsePerCpuCache(tc_globals), which registers rseq on the calling thread and calls
MallocExtension_Internal_MarkThreadIdle() (acquiring pageheap_lock while
ExtractStats already holds pageheap_lock), and ExtractStats and
GetNumericProperty read tc_globals before calling tc_globals.InitIfNecessary().
PiperOrigin-RevId: 1008451909
When compiled with `-DTCMALLOC_INTERNAL_LEGACY_LOCKING2`, `MADV_REMOVE` is attempted before `MADV_DONTNEED` on all mappings.
PiperOrigin-RevId: 1008399804
`FuzzSpan` previously pushed freed objects back to the span one at a time
(`batch.size() == 1`), leaving the multi-object coalescing loop in
`Span::DynamicBitmapPush` unexercised by `FuzzSpan`.
- Push objects in batches up to `num_to_move` in `FuzzSpan` and verify
`span->Allocated()` after each non-terminal push.
PiperOrigin-RevId: 1008371700
Replaces repetitive hand-written `Scoped*` classes in `testutil.h` with a two-argument template `ScopedParameter<Getter, Setter>` that records the current value on construction and restores it on destruction. Adds `ScopedMaxPerCpuCacheSize` and `ScopedMaxTotalThreadCacheBytes`, and converts manual parameter restore logic in `profile_test.cc`, `threadcachesize_test.cc`, and `hooks_test.cc`.
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1008370660
In DumpStats() and DumpStatsInPbtxt(), limit(kSoft), limit(kHard), limit_hits(kSoft), limit_hits(kHard), successful_shrinks_after_limit_hit(kSoft), and successful_shrinks_after_limit_hit(kHard) were queried sequentially, acquiring and releasing pageheap_lock 6 separate times.
Add PageAllocatorImpl::GetLimitStats() (ABSL_EXCLUSIVE_LOCKS_REQUIRED(pageheap_lock)) and snapshot LimitStats inside ExtractStats() alongside GetReleaseStats() under the existing PageHeapSpinLockHolder scope, eliminating 6 pageheap_lock acquisitions per DumpStats / DumpStatsInPbtxt call.
PiperOrigin-RevId: 1008281099
The `using_hpaa` guard looked for "HugePageAwareAllocator", which the stats
never print, so its checks never ran (one searched the text output for a pbtxt
field). HPAA is the only page allocator, so check unconditionally. Sampled
and cold allocators always subrelease, so the enabled case checks that no
allocator reports it disabled.
PiperOrigin-RevId: 1008278009
In `Dealloc::Perform`, `Span::FreelistPushBatch` returns `false` without
pushing objects or decrementing `allocated_` when all live objects in the span
are freed (`n == state.live_ptrs.size()`). Previously, `Dealloc::Perform`
ignored this return value and cleared `state.live_ptrs`, desynchronizing
`state.span->Allocated()` from `state.live_ptrs.size()` and permanently losing
those objects across subsequent instructions.
- Check the return value of `FreelistPushBatch` in `Dealloc::Perform` and
rebuild the span freelist when the span becomes completely free.
- Check `state.span->Allocated() == state.live_ptrs.size()` after each
instruction in `FuzzSpanInstructions`.
- Add `FuzzSpanInstructionsFullDealloc` regression test.
PiperOrigin-RevId: 1008265820
`UseWiderSlabs()` only enabled 512 KiB per-CPU slabs when `active_partitions() == 1`. With light security heap partitioning enabled by default in builds with alloc-token support (`active_partitions() == 2`), `UseWiderSlabs()` is no longer active.
PiperOrigin-RevId: 1008221735
Releasing from HugeRegion without pageheap_lock lets other operations run
while an unback is in flight. Strengthen the fuzzer's reentrant unback
callback and invariants so it catches a hugepage being unbacked twice or
stats drifting out of agreement:
- Assert that an in-flight unback range does not overlap another in-flight
range and contains no already-released page, before and after the reentrant
subprogram runs.
- Check `AddSpanStats` output against `free_pages()`/`unmapped_pages()` and
the mock's released-page set in `CheckInvariants`.
- Track released state at whole-hugepage granularity on allocation and check
`from_released` in both directions.
- Add `HugeRegionTest.UnbackPendingCandidateDuringRelease`: a candidate that
Release selected up front is unbacked by a concurrent Release or by
Allocate + Deallocate(release=true) while an earlier range is in flight.
PiperOrigin-RevId: 1008202931
A PTE mapping the shared zero page reads PM_PRESENT with neither
PM_MMAP_EXCLUSIVE nor PM_FILE set. The kernel does not count it in Rss, but
ResidencyPageMap::Update() did, overstating resident bytes. Only count present
pages that also have PM_MMAP_EXCLUSIVE or PM_FILE set.
Anonymous pages shared after fork() and KSM pages are no longer counted
either; an exact zero-page check would need PFNs, which require
CAP_SYS_ADMIN. Huge zero page PMD mappings report PM_FILE and are still
counted. GetUnbackedAndSwappedBitmaps() is unchanged.
New tests cross-check ResidencyPageMap against /proc/self/smaps Rss for real
mappings: touched and read-faulted anonymous memory, a THP before and after a
split by MADV_COLD, and a file mapped twice.
PiperOrigin-RevId: 1008179989
Large sparse bitmaps are slow, since we need to scan lots of empty words.
The worst case is 8b objects in 256K page config with 512 uint64_t words for bitmap.
Introduce two-level bitmap tracking for Span dynamic bitmaps across five specializations:
- `<= 64` objects (`kInline64`): single-level bitmap in `Span::raw_bitmap_[0]`.
- `65..256` objects (`kInline256`): 1st-level bitmap in `Span::raw_bitmap_[0..3]`
and a 4-bit 2nd-level summary bitmap in a new `level2_bitmap_ : kRawBitmapSize`
bitfield placed before `first_page_` (preserving `sizeof(Span)`).
- `> 256` objects: 1st-level bitmap at the end of the 1-page span and 2nd-level
summary bitmap in `Span::raw_bitmap_[0..3]`, split into:
- `kOutlineL2_1`: 1st-level bitmap has `<= 64` words (1 L2 word).
- `kOutlineL2_4`: 1st-level bitmap has `65..256` words (up to 4 L2 words,
compiled only when `kPageSize == 256 KiB`).
- `kOutlineL2_8`: 1st-level bitmap has `257..512` words (8-byte objects in
the 256 KiB page configuration, where each L2 bit summarizes a pair of L1
words; compiled only when `kPageSize == 256 KiB`).
Template `DynamicBitmapPushImpl`, `DynamicBitmapPopWord`, and
`DynamicBitmapPopBitmap` on `Span::BitmapMode` so inner loops are branch-free
with respect to the configuration.
This prevents steep drop in single-threaded performance esp for 256K page config.
PiperOrigin-RevId: 1008101179
Index dynamic Span bitmap backwards. This allows to locate the bitmap without offset.
Bitmap at the end of the span is used only for spans with 1 page, so we can locate
"beginning" of the bitmap by adding kPageSize to span start, and then index backwards.
This is needed in preparation for the next change that adds 2-level bitmaps,
and multiple "modes". Without this change we would need to pass both
bitmap_offset and bitmap_mode around through all methods, which increases
register pressure.
If we index the bitmap backwards, we only need to pass bitmap_mode around.
PiperOrigin-RevId: 1008055986
`Static::SlowInitIfNecessary` activates `cpu_cache_` after dropping
`pageheap_lock`, still holding `init_lock_`, when
`Parameters::per_cpu_caches()` and `subtle::percpu::IsFast()` hold. `inited_`
is published only after activation, so a thread racing to make its first
allocation waits on `init_lock_` instead of starting on a per-thread cache.
Setting `TCMALLOC_DEFER_PER_CPU_CACHES=1` defers activation as before; the
`defer_per_cpu_caches` test variant keeps that path covered.
PiperOrigin-RevId: 1007839812
Nothing defines or calls the weak
`TCMalloc_Internal_{Get,Set}PrioritizeSpansEnabled` and
`TCMalloc_Internal_{Get,Set}ReleaseFreeSwapped` accessors.
Drop `Parameters` friend declarations for
`TCMalloc_Internal_SetLifetimeAllocatorOptions` (never defined),
`TCMalloc_Internal_SetReleaseFreeSwapped`, and four setters that don't touch
`Parameters` internals (`SetHPAASubrelease`, `SetMadvise`,
`SetMaxPerCpuCacheSize`, `SetPerCpuCachesEnabled`).
PiperOrigin-RevId: 1007828919
Root Cause: IsNormalMemory and kNormalOrBadDeallocationMask masked only bit 0
of the pointer tag, accepting kMetadata (0x3) in IsNormalMemory and unassigned
tags (0x5, 0x7) in both IsNormalMemory and kNormalOrBadDeallocationMask.
PiperOrigin-RevId: 1007828882
These weak shims have had no definitions since the subrelease interval split into short and long intervals, and have no callers.
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1007827280
Per the ELF specification, p_align of 0 or 1 specifies default 4-byte note alignment rather than 0- or 1-byte alignment. Normalize p_align <= 1 to alignof(ElfW(Nhdr)) and require power-of-2 alignment >= alignof(ElfW(Nhdr)) before dereferencing note headers.
PiperOrigin-RevId: 1007825808
`Parameters::release_drained_slab_metadata()` checked a `LowLevelCallOnce`
flag on every read to apply `TCMALLOC_RELEASE_DRAINED_METADATA_ABLATION`.
Apply the experiment in `Parameters::InitializeFromExperiments()` with the
other experiment-defaulted parameters, and make the getter a plain load.
`TCMalloc_Internal_{Get,Set}ReleaseDrainedSlabMetadata` call
`InitIfNecessary()` first, like the other parameters initialized there, so an
explicit setting still wins over the experiment.
PiperOrigin-RevId: 1007825324
`TcmallocTest` realloc tests allocate buffers at several starting sizes and check that small positive and negative size deltas stay in place without reallocating. Under the 256 KiB page flexible size-class tables, a 9984-byte size class sits between 9968 and 10000, so shrinking a 10000-byte allocation by 32 bytes crosses into the smaller size class and forces `do_realloc` to allocate a new buffer.
Use 9900 instead of 10000 as the starting size so the 9868-to-9916 byte range stays within a single size class across all page sizes and size-class tables.
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1007817945
(annotated `ABSL_EXCLUSIVE_LOCKS_REQUIRED(pageheap_lock)` and
`ABSL_ATTRIBUTE_LIFETIME_BOUND`) to avoid copying the 112-byte
`HugePageResidencyState` (including three 32-byte `PageBitmap`s) by value
when `SelectEligibleTrackers` and `UsageInfo::Record` inspect tracker
metadata under `pageheap_lock`.
Preserves legacy behavior under `-DTCMALLOC_INTERNAL_LEGACY_LOCKING2`.
PiperOrigin-RevId: 1007801966
AllocCpuCache only padded its mapping when the requested alignment exceeded
EXEC_PAGESIZE, assuming mmap returns EXEC_PAGESIZE-aligned memory. On aarch64,
EXEC_PAGESIZE is 64 KiB, but kernels commonly use 4 KiB pages, so slabs
requesting up to 64 KiB alignment came back 4 KiB-aligned and larger alignments
were under-padded. Pad relative to GetPageSize() instead.
PiperOrigin-RevId: 1007780831
Root Cause: The guarded sampling interval was saved in a `bool`, so the test
restored an interval of 1, and `min_hot_access_hint` was never restored after
being set to 3.
Use the existing `ScopedGuardedSamplingInterval` and
`ScopedSkipSubrelease{Short,Long}Interval` helpers, and restore
`min_hot_access_hint`.
PiperOrigin-RevId: 1007780601
In `SampleRecorder::PushDead`, acquire `sample->lock` before `graveyard_.lock` (returning early if `sample->dead != nullptr`) so `graveyard_.lock` is held only across the two-pointer dead-list splice (`Register` already pops `graveyard_.dead` before acquiring `sample->lock`), and delegate `UnregisterAll` to `PushDead(sample)` so both paths share the `sample->lock` -> `graveyard_.lock` acquisition order (`ABSL_ACQUIRED_AFTER(T::lock)`).
PiperOrigin-RevId: 1007780388
Actuate the backed-pages filler parameter from the experiment and the `--tcmalloc_prefer_backed_filler_pages` flag, and export it in allocator statistics and test variants.
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1007657956
When allocating spans from a partially released hugepage, standard best-fit selection chooses the shortest free range across all free pages regardless of whether the pages are currently backed or released to the OS. When an unbacked free hole happens to be a tighter size match than an already-backed free hole on the same hugepage, allocating the unbacked hole faults in previously unbacked pages while leaving resident free pages unused.
Gate backed-first allocation within a hugepage behind `Parameters::prefer_backed_filler_pages()`. When enabled on a tracker with released pages, `PageTracker::Get()` scans the combined allocated and released page bitmaps to pick the shortest backed free range that fits the request, and falls back to unbacked holes only when no backed range is large enough.
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1007593493
cpu_cache_test exercises each CpuCache path in isolation. The new FuzzTest
harness drives a fuzzer-chosen sequence of Allocate, Deallocate, Churn,
Drain, Reclaim, ReleaseDrainedMetadata, TryDrainingCaches, ShuffleCpuCaches,
ResizeSizeClasses, ResizeSizeClassMaxCapacities, ResizeSlabIfNeeded,
SetCacheLimit, Print, PrintInPbtxt, and stats reads against a single-threaded
CpuCache<TestStaticForwarder> from up to four faked CPUs, under every
combination of dynamic slab, drained-slab release, real or three-class SizeMap,
sharded transfer cache backing (none, large size classes only, or generic),
and six per-CPU cache limits. After each step it checks
that Allocated + Unallocated == Capacity per CPU, that total capacity is
conserved across shuffles and stealing and follows SetCacheLimit (growing adds
the difference to every CPU, shrinking resets every CPU to the limit), that
TotalUsedBytes agrees with both the per-CPU and per-size-class views, that
size-class capacities stay within GetMaxCapacity for the current shift, that
unpopulated CPUs hold no capacity, and that ResizeSlabIfNeeded lands on the
shift predicted from the interval miss counters.
Reclaim and ReleaseDrainedMetadata (CpuCachePeer::Reclaim and
CpuCachePeer::ReleaseDrainedSlabMetadata) unpopulate CPUs and release their
slab metadata without waiting for two idle drain intervals, so Populate runs
again on CPUs whose slab was madvised away and ResizeSlabs copies around
unpopulated CPUs. The sharded transfer cache mode is part of the fuzzed
configuration rather than an instruction: ShardedTransferCacheManagerBase::Init
latches it once at startup and nothing changes it afterwards. The harness owns
that manager so that it is backed by FakeCentralFreeList, whose objects can be
dereferenced and leak-checked, and carves its metadata from a rewindable arena
because the manager never frees its shards.
In production every remote slab operation stops and fences the CPUs it
touches, and the kernel's rseq notify-resume invalidates each thread's cached
slab pointer. With rseq unregistered by ScopedFakeCpuId that never happens,
so the harness re-registers rseq after each such instruction; otherwise the
thread would keep using a slab that had been drained, unpopulated, or
madvised away. TestStaticForwarder::AllocReportedImpending now allocates
through AllocCpuCache so that resized slabs carry the alignment
ReleaseSlabMetadataForDrainedCpus checks for.
The harness records every mapping handed out through the forwarder and
unmaps the survivors at teardown, since CpuCache retains one slab per shift
for reuse and only Deactivate() returns the current one.
PiperOrigin-RevId: 1005924977
Ensure `PageAllocatorImpl::ShrinkToUsageLimitSlow` preserves `over_limit_ = backed > limits_[kSoft]` before returning early when `limits_[kHard]` is configured alongside `limits_[kSoft]`, so subsequent `Finalize(..., /*may_have_grown=*/false)` calls still enforce `limits_[kSoft]` after `AllocAndContribute` adds the new hugepage to `filler_`. Also accumulate `last_may_have_grown_ |= may_have_grown` in `FakeHugePageStaticForwarder::ShrinkToUsageLimit` across calls within a single allocation.
When compiled with `-DTCMALLOC_INTERNAL_LEGACY_LOCKING2`, `*from_released` is passed to `Finalize` as before.
PiperOrigin-RevId: 1005872181
cpu_cache_fuzz will drive CpuCache<TestStaticForwarder> with the same
forwarder and white-box accessors as cpu_cache_test, so hoist CpuCachePeer,
DynamicSlab, and TestStaticForwarder verbatim from cpu_cache_test.cc into a
testonly mock_cpu_cache library.
PiperOrigin-RevId: 1005826410
The first allocation in a process looks up its size class before `SizeMap` is
initialized, so `fast_alloc` passes `size_class == 0`. Today the
`!UsePerCpuCache()` check routes it to
`alloc_small_sampled_hooks_or_perthread`, which recomputes the size class.
Have `Sampler::RecordAllocationSlow` return `-1` when initializing the sampler
for an unsampled allocation so `weight != 0` in `slow_alloc_small` routes it to
`alloc_small_sampled_hooks_or_perthread` without adding a branch once
initialization activates per-CPU caches.
PiperOrigin-RevId: 1005680372
Give `SampleRecorder::graveyard_` its own `SpinLock` and `dead` member declarations (via `struct Graveyard : public T`) annotated with `ABSL_ACQUIRED_BEFORE(T::lock)` and `ABSL_GUARDED_BY(lock)`, and add `ABSL_LOCKS_EXCLUDED` annotations to `SampleRecorder` methods so Clang `-Wthread-safety-analysis` distinguishes `graveyard_.lock` from per-sample `sample->lock` and statically enforces their acquisition order.
PiperOrigin-RevId: 1005677913
Concurrent `TCMalloc_Internal_ForceCpuCacheActivation` calls could each see
`!CpuCacheActive()` and call `CpuCache::Activate` twice.
`Static::ActivateCpuCacheIfNecessary` rechecks under `init_lock_`, which
`SlowInitIfNecessary` also holds, so initialization can later activate the
cache after dropping `pageheap_lock`.
PiperOrigin-RevId: 1005667748
Replace the three duplicate
want_disable_{huge_region_more_often,cfl_subbucket_prioritization,subrelease_unbacked_hugepages}_test{,_helper}
pairs with want_disable_test and a shared want_disable_test_helper that writes
MallocExtension::GetStats() to stdout, and exclude the helper source from
open-source exports.
PiperOrigin-RevId: 1005598420
Assert reported stats against Parameters getters instead of re-checking
IsExperimentActive in get_stats_test and drop the experiment dependency.
PiperOrigin-RevId: 1005591448
Over-aligned allocations with size <= kMaxSize have sizeclass == 0 in PageMap. Checking only actual_new_size != old_size returned old_ptr in-place when new_size mapped to a small size class of the same byte size, failing CorrectSize and routing subsequent free_sized calls to FreeSmall on a PageHeap large page.
PiperOrigin-RevId: 1005244547
Scheduled runs in the GitHub Actions CI workflow pass `--cache_test_results=no` across all 12 matrix jobs to re-run the full test suite without cached test results. Running that uncached 12-job wave every hour creates recurring contention on the remote execution pool.
Change the `schedule` cron expression from `0 * * * *` to `17 */4 * * *` so scheduled test runs execute every 4 hours, and offset the trigger to minute 17 to avoid top-of-the-hour cron traffic spikes.
Co-authored-by: Chris Kennelly <ckennelly@google.com>
PiperOrigin-RevId: 1005214566
Each forwarder owned an Arena, and Arena never returns its blocks: the first allocation takes a hugepage from the system allocator and keeps it for the life of the process. huge_page_aware_allocator_fuzz builds a HugePageAwareAllocator, and so a forwarder, per input, leaking about a hugepage of resident memory per execution. Under Centipede's persistent runner that reached 11 GiB after a minute of fuzzing and the run was killed.
Hand out one process-wide Arena instead. Metadata an allocator took from it stays allocated once the allocator is gone, which is a few KiB per instance.
PiperOrigin-RevId: 1005182717
