The out-of-line definitions in central_freelist.cc become inline members of
the template, so central_freelist.cc is deleted. central_freelist.h forward
declares StaticForwarder, since static_forwarder.h depends on it via
parameters.h.
The StaticForwarderSpanTest cases, previously moved to
static_forwarder_test.cc, are retargeted to the production forwarder. The
small-but-slow TransferCacheManager is templated on its forwarder, as the
default one is, so that CentralFreeList's members are only instantiated where
used.
PiperOrigin-RevId: 998314669
Clang 11 (which added asm goto support) was released almost
six years ago, and GCC has supported this since about 2010.
Clean it up so that the code is easier to read.
Unfortunately, GCC has grown a new regression in the meantime
(supposedly wrong code from GCC 14, ICE from GCC 16), so add
a smaller workaround until it is hopefully fixed in 16.3.
PiperOrigin-RevId: 970561435
cpu_cache.h calls this Reclaim, but TcmallocSlab and most people
seem to call it Drain. Rename so that we use only one term
for what is the same operation.
PiperOrigin-RevId: 967665500
The current text implies that all sampled objects are placed on the sampled
heap and not hugepage'd, but this only applies to objects smaller than
`kMaxSize` (typically 256KB).
PiperOrigin-RevId: 949562209
This adds a new build mode to TCMalloc that can be enabled by setting
TCMALLOC_INTERNAL_LATENCY_INJECTION. This mode introduces delay
injection at 4 critical points in the allocator:
1. Page Heap SpinLock acquisition
2. Central FreeList SpinLock acquisition
3. per-CPU Cache UpdateMaxCapacities
4. per-CPU Cache ResizeSlabs
These latency injections are inspired by Coz (https://arxiv.org/abs/1608.03676).
PiperOrigin-RevId: 893785128
Change-Id: Iea28741fd89a0eedcae15b7a4091f28a367f01b7
this is terrible. GitHub processes markdown before LaTeX, so it does things
like & -> &, * -> _, and \! -> ! transforms before giving it to the LaTeX
parser, which also just completely ignores \boxed and fatals on \phantom. The
only cross-platform way to get this to work is ASCII art.
PiperOrigin-RevId: 871476854
Change-Id: Iffbd32c747290b985391dc3bb7c310865726ea7c
When enabled, HugeCache uses the recent demand history to guide its release decisions: it would reduce the target if those free hugepages will likely be reallocated soon. Note, it would only try to optimize for the background non-urgent release requests.
PiperOrigin-RevId: 655755775
Change-Id: I8380ea044e28756066d6feb7f180647af4b8c163
A recent change fixed the names of the sampling "rate" to "interval": the
profile sampling interval denotes the number of bytes between allocations.
Update the wording in documentation to use the correct terminology.
PiperOrigin-RevId: 643331112
Change-Id: Ic4f7cc733de9d6ca0d9d45d830bcd2e6a996b941
Currently the slab header is 8 bytes and holds [begin, current, end] indices.
This change switches to the new scheme where the header is 4 bytes and
holds only [current, end] indices. Begin indices are stored in a separate array
and are used only in slower operations.
In the new scheme we mark the stub element stored before the size class range
(the one that we use for prefetching when we pop the last element) with
low bit set. This allows to easily understand when we reached the begin
of the range during pop.
Currently Pop check looks as follows:
movzwq (%[scratch], %[size_class], 8), %[current]
cmpw 2(%[scratch], %[size_class], 8), %w[current]
jbe %l[underflow_path]
movq -8(%[scratch], %[current], 8), %[result]
In the new scheme Pop check looks as follows:
movzwq (%[scratch], %[size_class], 4), %[current]
movq -8(%[scratch], %[current], 8), %[result]
testb $1, %b[result]
jnz %l[underflow_path]
Effectively we trade:
5167ca: 66 45 3b 4c d0 02 cmpw 0x2(%r8,%rdx,8), %r9w
for:
516654: a8 01 testb $0x1, %al
This has several advantages: -1 load, -4 instruction bytes, but more importantly
fewer hot cache lines in L1D$ (~5 instead of 10).
Note: this required to move TcmallocSlab initialization a bit earlier in CpuCache,
so that we don't try to pop from uninitialized slab.
Microbenchmark results with -no-pie:
│ sec/op │ sec/op vs base │
BM_new_delete/8 9.091n ± 0% 8.851n ± 0% -2.65% (p=0.000 n=20)
BM_new_delete/64 9.111n ± 0% 8.873n ± 0% -2.62% (p=0.000 n=20)
BM_new_sized_delete/8 3.585n ± 1% 3.565n ± 0% -0.57% (p=0.000 n=20)
BM_new_sized_delete/64 4.559n ± 0% 4.544n ± 2% ~ (p=0.298 n=20)
BM_size_returning_new_delete/8 3.814n ± 0% 3.741n ± 1% -1.91% (p=0.000 n=20)
BM_size_returning_new_delete/64 3.840n ± 1% 3.761n ± 1% -2.04% (p=0.000 n=20)
BM_aligned_new/8/8 5.440n ± 1% 5.434n ± 1% ~ (p=0.255 n=20)
BM_aligned_new/64/64 5.929n ± 2% 5.462n ± 0% -7.88% (p=0.000 n=20)
Without -fpie:
│ sec/op │ sec/op vs base │
BM_new_delete/8 9.216n ± 0% 8.785n ± 0% -4.68% (p=0.000 n=20)
BM_new_delete/64 9.236n ± 0% 8.811n ± 0% -4.60% (p=0.000 n=20)
BM_new_sized_delete/8 4.454n ± 13% 4.419n ± 1% ~ (p=0.071 n=20)
BM_new_sized_delete/64 4.465n ± 0% 4.468n ± 0% ~ (p=0.846 n=20)
BM_size_returning_new_delete/8 4.438n ± 0% 4.416n ± 2% ~ (p=0.057 n=20)
BM_size_returning_new_delete/64 4.489n ± 2% 4.892n ± 5% ~ (p=0.107 n=20)
BM_aligned_new/8/8 4.625n ± 0% 4.623n ± 0% ~ (p=0.587 n=20)
BM_aligned_new/64/64 6.365n ± 0% 6.472n ± 9% ~ (p=0.080 n=20)
PiperOrigin-RevId: 617565606
Change-Id: I4291813fe540e6c0cecfb30c8893950c33c21dae
Currently the slab header is 8 bytes and holds [begin, current, end] indices.
This change switches to the new scheme where the header is 4 bytes and
holds only [current, end] indices. Begin indices are stored in a separate array
and are used only in slower operations.
In the new scheme we mark the stub element stored before the size class range
(the one that we use for prefetching when we pop the last element) with
low bit set.
PiperOrigin-RevId: 609982618
Change-Id: I7cff2c196c6a52f5fb7b08f1548c3e72a5ae2ad9
Currently the slab header is 8 bytes and holds [begin, current, end] indices.
This change switches to the new scheme where the header is 4 bytes and
holds only [current, end] indices. Begin indices are stored in a separate array
and are used only in slower operations.
In the new scheme we mark the stub element stored before the size class range
(the one that we use for prefetching when we pop the last element) with
low bit set. This allows to easily understand when we reached the begin
of the range during pop.
Currently Pop check looks as follows:
movzwq (%[scratch], %[size_class], 8), %[current]
cmpw 2(%[scratch], %[size_class], 8), %w[current]
jbe %l[underflow_path]
movq -8(%[scratch], %[current], 8), %[result]
In the new scheme Pop check looks as follows:
movzwq (%[scratch], %[size_class], 4), %[current]
movq -8(%[scratch], %[current], 8), %[result]
testb $1, %b[result]
jnz %l[underflow_path]
Effectively we trade:
5167ca: 66 45 3b 4c d0 02 cmpw 0x2(%r8,%rdx,8), %r9w
for:
516654: a8 01 testb $0x1, %al
This has several advantages: -1 load, -4 instruction bytes, but more importantly
fewer hot cache lines in L1D$ (~5 instead of 10).
Note: this required to move TcmallocSlab initialization a bit earlier in CpuCache,
so that we don't try to pop from uninitialized slab.
Microbenchmark results with -no-pie:
│ sec/op │ sec/op vs base │
BM_new_delete/8 9.091n ± 0% 8.851n ± 0% -2.65% (p=0.000 n=20)
BM_new_delete/64 9.111n ± 0% 8.873n ± 0% -2.62% (p=0.000 n=20)
BM_new_sized_delete/8 3.585n ± 1% 3.565n ± 0% -0.57% (p=0.000 n=20)
BM_new_sized_delete/64 4.559n ± 0% 4.544n ± 2% ~ (p=0.298 n=20)
BM_size_returning_new_delete/8 3.814n ± 0% 3.741n ± 1% -1.91% (p=0.000 n=20)
BM_size_returning_new_delete/64 3.840n ± 1% 3.761n ± 1% -2.04% (p=0.000 n=20)
BM_aligned_new/8/8 5.440n ± 1% 5.434n ± 1% ~ (p=0.255 n=20)
BM_aligned_new/64/64 5.929n ± 2% 5.462n ± 0% -7.88% (p=0.000 n=20)
Without -fpie:
│ sec/op │ sec/op vs base │
BM_new_delete/8 9.216n ± 0% 8.785n ± 0% -4.68% (p=0.000 n=20)
BM_new_delete/64 9.236n ± 0% 8.811n ± 0% -4.60% (p=0.000 n=20)
BM_new_sized_delete/8 4.454n ± 13% 4.419n ± 1% ~ (p=0.071 n=20)
BM_new_sized_delete/64 4.465n ± 0% 4.468n ± 0% ~ (p=0.846 n=20)
BM_size_returning_new_delete/8 4.438n ± 0% 4.416n ± 2% ~ (p=0.057 n=20)
BM_size_returning_new_delete/64 4.489n ± 2% 4.892n ± 5% ~ (p=0.107 n=20)
BM_aligned_new/8/8 4.625n ± 0% 4.623n ± 0% ~ (p=0.587 n=20)
BM_aligned_new/64/64 6.365n ± 0% 6.472n ± 9% ~ (p=0.080 n=20)
PiperOrigin-RevId: 608636032
Change-Id: I78547cf4535c84a0be2b3345bcff5e800ba6d1f6
Currently the slab header is 8 bytes and holds [begin, current, end] indices.
This change switches to the new scheme where the header is 4 bytes and
holds only [current, end] indices. Begin indices are stored in a separate array
and are used only in slower operations.
In the new scheme we mark the stub element stored before the size class range
(the one that we use for prefetching when we pop the last element) with
low bit set. T
PiperOrigin-RevId: 604897488
Change-Id: I6e5ee68239c3784fe388b9973c00b10b23d8bb43
Currently the slab header is 8 bytes and holds [begin, current, end] indices.
This change switches to the new scheme where the header is 4 bytes and
holds only [current, end] indices. Begin indices are stored in a separate array
and are used only in slower operations.
In the new scheme we mark the stub element stored before the size class range
(the one that we use for prefetching when we pop the last element) with
low bit set. This allows to easily understand when we reached the begin
of the range during pop.
Currently Pop check looks as follows:
movzwq (%[scratch], %[size_class], 8), %[current]
cmpw 2(%[scratch], %[size_class], 8), %w[current]
jbe %l[underflow_path]
movq -8(%[scratch], %[current], 8), %[result]
In the new scheme Pop check looks as follows:
movzwq (%[scratch], %[size_class], 4), %[current]
movq -8(%[scratch], %[current], 8), %[result]
testb $1, %b[result]
jnz %l[underflow_path]
Effectively we trade:
5167ca: 66 45 3b 4c d0 02 cmpw 0x2(%r8,%rdx,8), %r9w
for:
516654: a8 01 testb $0x1, %al
This has several advantages: -1 load, -4 instruction bytes, but more importantly
fewer hot cache lines in L1D$ (~5 instead of 10).
Note: this required to move TcmallocSlab initialization a bit earlier in CpuCache,
so that we don't try to pop from uninitialized slab.
Microbenchmark results with -no-pie:
│ sec/op │ sec/op vs base │
BM_new_delete/8 9.091n ± 0% 8.851n ± 0% -2.65% (p=0.000 n=20)
BM_new_delete/64 9.111n ± 0% 8.873n ± 0% -2.62% (p=0.000 n=20)
BM_new_sized_delete/8 3.585n ± 1% 3.565n ± 0% -0.57% (p=0.000 n=20)
BM_new_sized_delete/64 4.559n ± 0% 4.544n ± 2% ~ (p=0.298 n=20)
BM_size_returning_new_delete/8 3.814n ± 0% 3.741n ± 1% -1.91% (p=0.000 n=20)
BM_size_returning_new_delete/64 3.840n ± 1% 3.761n ± 1% -2.04% (p=0.000 n=20)
BM_aligned_new/8/8 5.440n ± 1% 5.434n ± 1% ~ (p=0.255 n=20)
BM_aligned_new/64/64 5.929n ± 2% 5.462n ± 0% -7.88% (p=0.000 n=20)
Without -fpie:
│ sec/op │ sec/op vs base │
BM_new_delete/8 9.216n ± 0% 8.785n ± 0% -4.68% (p=0.000 n=20)
BM_new_delete/64 9.236n ± 0% 8.811n ± 0% -4.60% (p=0.000 n=20)
BM_new_sized_delete/8 4.454n ± 13% 4.419n ± 1% ~ (p=0.071 n=20)
BM_new_sized_delete/64 4.465n ± 0% 4.468n ± 0% ~ (p=0.846 n=20)
BM_size_returning_new_delete/8 4.438n ± 0% 4.416n ± 2% ~ (p=0.057 n=20)
BM_size_returning_new_delete/64 4.489n ± 2% 4.892n ± 5% ~ (p=0.107 n=20)
BM_aligned_new/8/8 4.625n ± 0% 4.623n ± 0% ~ (p=0.587 n=20)
BM_aligned_new/64/64 6.365n ± 0% 6.472n ± 9% ~ (p=0.080 n=20)
PiperOrigin-RevId: 604575758
Change-Id: I79d1d556f2308536d92e944d303bb6c8c105f80a
Currently we have 2 mechanisms to synchronize remote operations:
1. Coarse-gained used by Resize (sets resizing_ to prevent slab caching).
2. Fine-gained used by all other operations (uses complex header locking).
Switch all operations to the coarse-grained scheme.
Besides removing a chunk of the most complex and tricky code,
this reduces number of fences we need per remote operation
(now it's always 1 in the beginning, and no fences at the end),
and also unlocks other nice things in future changes:
- combining fences for series of remote grow/shrink operations
- arbitrary changes to the slab Header (now it can be any number
of bytes changed in any way)
PiperOrigin-RevId: 603321290
Change-Id: Ib69d220fcd1d6188dabecbc023bc103339d96732
Introduced a pair of parameters, filler_skip_subrelease_short_interval() and filler_skip_subrelease_long_interval(), for calculating the recent demand requirement. The former finds the recent short-term demand fluctuation peak and the latter finds the recent long-term demand trend. The feature is enabled by configuring either one of the two intervals. The demand requirement is the sum of the two (fluctuation peak and demand trend) if both are configured.
Note, we recommend configuring both intervals, and the short-interval should be (significantly) shorter or equal to the long-interval to avoid realized fragmentation caused by non-recent (short-term) demand spikes.
The feature is not enabled by default: HugePageFiller prefers to use recent demand peak for making skip-subrelease decisions if the existing parameter filler_skip_subrelease_interval() is configured.
PiperOrigin-RevId: 509328699
Change-Id: I734c1f5bd7b8352fadd0fc739cd144e969a4adc3
Lifetime profiling reports a list of object lifetimes as pairs of allocation and deallocation records. Profiling is initiated by calling MallocExtension::StartLifetimeProfiling(). Profiling continues until `Stop` is invoked on the token. Lifetimes are only reported for objects where allocation *and* deallocation are observed while profiling is active.
The design and prototype of the deallocation oriented, lifetime profiler came out of an internship by Mohammad Mahdi Javanmard. Martin Maas brought the prototype to production where it has been used since mid-2019.
PiperOrigin-RevId: 477807613
Change-Id: I36a35a4e5ef4e90aec6c83940f28b0ae45909ee4
This reports the number of objects currently occupied by the freelists in the transfer cache. It also reports the current capacity of the freelists, and the maximum capacity that they can grow to. We also make CapacityNeeded in RingBufferTransferCache public as it is called in the tests.
PiperOrigin-RevId: 449242106
Change-Id: I1584f1f9fbcb9f5a3800d800bc37531a75964faf
This approximates "Adaptive Huge-Page Subrelease for Non-moving Memory
Allocators in Warehouse-Scale Computers"'s estimate of realized fragmentation
as T_{schedule} -> infinity. It captures memory going unused at a peak in
OS-visible demand due to metadata requirements and binpacking losses.
PiperOrigin-RevId: 439611199
Change-Id: Iccc056d7439a0ccaa0e9b92df615f000b8d84ed6
Motivation: we're considering using these ratios in the slab resizing decision.
PiperOrigin-RevId: 427761329
Change-Id: Ia8e8a71df511a22d233ab07f316b5371fe8fa0d8
Chris Kennelly CA