mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-10-26 10:57:38 +08:00
Linus Torvalds
c5fd5e67c7
Pull MM updates from Andrew Morton:
- "selftests/mm: improve khugepaged coverage" (Kiryl Shutsemau):
Give khugepaged a proper workout. Extend the tests to all supported
mTHP orders, check exactly which windows collapsed, and make scan
completion deterministic. Race collapse against faults, pinning, fork
and remap; verify that the contents survive.
Add coverage for whole-page-table zaps and memory pressure too.
- "selftests/mm: separate GUP microbenchmarking from functional testing"
(Sarthak Sharma):
Untangle GUP benchmarking from correctness testing. Move the
benchmark into tools/mm, add a standalone harness-based GUP selftest,
and move shared helpers into tools/lib/mm. Give each tool one job
instead of mixing timing measurements with correctness checks.
Cover the mapping, pinning and batch-size combinations explicitly.
- "mm: Switch device DAX to section-based vmemmap optimization" (Muchun
Song):
Apply section-based vmemmap optimization to device DAX. Rework the
backing metadata layout to save memory while preserving the handling
of device memory hotplug and its different alignment and lifetime
requirements.
- "mm: Introduce section-based vmemmap optimization for HugeTLB"
(Muchun Song):
Cut the memory consumed by HugeTLB's struct page array with
section-based vmemmap optimization. Arrange the metadata so large
huge pages need fewer backing pages, while keeping initialization,
teardown and the various hugepage configurations straight.
- "mm/huge_memory: clean up and decouple the anon and file split
helpers" (Kairui Song):
Separate anonymous and file-backed folio splitting, which have rather
different locking and bookkeeping requirements. Move the work into
separate helpers, drop dead checks and tidy swap-cache handling. Fix
error paths, including one that could leave a folio frozen.
Keep the swap cache cases separate from file-cache splitting.
- "mm: Unconditional per-VMA locks and cleanups" (Dave Hansen):
Make per-VMA locking unconditional and clean up the old conditional
code around it. This removes configuration-dependent plumbing and
lets callers depend on one locking model rather than carrying both
cases along.
- "memcg: remove the v1 soft limit" (Shakeel Butt):
Remove the old cgroup v1 memory soft-limit machinery. This has long
since been superseded by the newer memory-control interfaces; hauling
around a separate reclaim scheme for it no longer buys us much.
- "read proc/pid/smaps_rollup under per-vma lock" (Suren Baghdasaryan):
Read /proc/PID/smaps_rollup using per-VMA locks where possible,
rather than holding mmap_lock across the entire walk. Retain the
contention fallback and add tearing tests. This considerably reduces
latency when another thread is busy changing its mappings.
- "mm/damon: introduce data access-as-a-data attribute" (SJ Park):
Teach DAMON to expose memory access observations as data attributes,
making them available to the probe and filtering machinery. Extend
the interfaces, documentation and tests so the new information can be
used rather than merely collected.
- "Remove PG_private by using page/folio->private checks instead" (Zi
Yan):
Get rid of PG_private and use the folio's private pointer to say
whether private data is attached. Convert the filesystem users,
including erofs, f2fs and md, and clean up the old accessors and
flags. There are a few special cases, notably swapcache and hugetlb,
which need explicit care.
- "mm: make userland page table freeing RCU-safe" (Lorenzo Stoakes):
Make freeing userspace page tables safe against RCU readers. Sort out
the page-table lifetime and locking rules so walkers cannot stumble
over freed tables while mappings are being torn down.
- "make unused huge shrinker memcg aware" (Qi Zheng):
Make shmem's unused-huge-folio shrinker aware of memory cgroups, so
reclaim can find those folios under the right memcg. Wire up the
per-memcg tracking and its lifetime rules, and give the shrinklist
its own lockdep class to avoid spurious dependency reports.
- "mm: Drop pxd_ERROR()" (Anshuman Khandual):
Remove the old pxd_ERROR() page-table checks and their
architecture-specific baggage. Simplify the common page-table
interfaces and callers now that these checks no longer serve a
useful purpose.
- "memcg: group struct fields by access pattern" (Shakeel Butt):
Rearrange memcg's hot and cold fields according to how they're
actually accessed. Keep frequently updated counters and their
neighboring state from fighting over cachelines and reduce needless
cacheline traffic.
- "mm/damon/core: fix the size charged for a filter-trimmed region"
(Donggeun Yoo):
Fix DAMOS quota accounting when a filter trims a monitoring region.
Charge only the part left after filtering, not the original region,
and add KUnit coverage. Previously the excluded memory could eat into
the quota and make schemes run more slowly than requested.
- "mm/damon: add kunit and selftests for probes and probe weights" (SJ
Park):
Add more tests for DAMON probes and their weights, including
validation and the results reported to userspace. This is mostly
about catching regressions in the newer probe infrastructure before
they become harder to diagnose.
- "mm: memcontrol: constify the read side of the read side of the memcg
API" (Tal Zussman):
Mark the read-only parts of the memcg interface const, from folio and
lruvec accessors to counters and swap helpers. Most of this is
mechanical, but it makes the contracts clearer and lets the compiler
catch accidental writes.
- "mm/damon/vaddr: support {prep,apply}_probes" (SJ Park):
Support the new prepare/apply probe operations in DAMON's
virtual-address backend. This brings vaddr monitoring into line with
the probe infrastructure and makes the added probe types useful there
too.
- "mm/mglru: clean up folio counters and flag usage" (Kairui Song):
Simplify MGLRU's folio counters and flag handling. Remove redundant
accounting paths and make the state transitions easier to follow
without changing the intended reclaim policy.
- "arch, mm/execmem: resolve confusion about set_direct_map_valid_noflush()"
(Mike Rapoport (Microsoft)):
Clarify and straighten out the set_direct_map_valid_noflush()
interface across architectures and execmem. Make the requested
mapping changes and the callers' expectations agree, rather than
relying on subtly different architecture behavior.
- "mm: zswap: free cold writeback folios promptly" (Alexandre Ghiti):
Free zswap writeback folios as soon as their disk write completes.
They are cold temporary buffers, so putting them on the LRU only
makes reclaim rediscover them later. Extend swap-cache dropbehind and
let zswap keep these folios off the LRU altogether.
- "mm: gup: cleanup gup_fast call chain" (Aristeu Rozanski):
Clean up the GUP-fast call chain and simplify how the fast-path
helpers pass their arguments and results around. Keep the page-pin
behavior the same while making the call sequence easier to maintain.
- "mm: optimize zone-device memmap initialization" (Li Zhe):
Speed up initialization of zone-device struct pages. Avoid redundant
work when setting up large device-memory ranges, where touching every
descriptor can get expensive.
- "mm/zswap: shrink zswap_entry via a pool id" (Jianyue Wu):
Reduce the size of zswap_entry by storing a compact pool identifier
instead of a full pointer. Keep the pool lookup and lifetime handling
correct while saving per-entry metadata.
- "mm/damon: support access monitoring of hugetlb-backed memory"
(Krishna Iyer):
Let DAMON monitor hugetlb-backed memory too. Extend the relevant
access checks and add coverage so those mappings no longer fall
through a hole in the monitoring machinery.
- "mm/execmem: fixes and cleanups for the ROX cache" (Mike Rapoport):
Tidy execmem's read-only-executable cache and fix its handling of the
awkward allocation and release cases. The aim is to make the cache's
ownership and permission transitions more predictable.
- "zsmallc: remove old object read API" (Sergey Senozhatsky):
Drop zsmalloc's old object-read API and move its remaining users to
the supported access mechanism. This removes a duplicate interface
which we no longer need.
- "mm/mempolicy: stop copying state in the interleave paths" (Gregory
Price):
Stop copying mempolicy state in the interleave paths when a reference
to the existing policy will do. This avoids extra work and makes the
ownership rules less obscure.
- "mm/mglru: speed up inc_min_seq() and fix cold/hot inversions" (Barry
Song):
Make MGLRU's generation advancement cheaper and fix cases where cold
and hot folios end up in the wrong order. Adjust the counter and list
handling so reclaim doesn't keep passing over the better eviction
candidates.
- "mm: Fix MF_DELAYED handling on memory failure" (Lisa Wang):
Treat MF_DELAYED as a partially completed memory-failure operation,
not an outright failure. Propagate that result correctly through
shmem and the generic handler, fix the extra-reference accounting,
and add tests for the resulting SIGBUS behavior.
- "tools/mm/page_owner_sort: fix --sort, add module filter, improve
usage" (Ye Liu):
Fix page_owner_sort's --sort handling, add module-based filtering and
make the usage output more helpful. This makes the tool less
frustrating when tracking down who is holding on to pages.
- "mm/damon: move damos filter range arguments validation to core" (SJ
Park):
Move DAMOS filter range validation into the core, where all callers
can share it. Also close the sysfs read-twice races which could pass
inconsistent ranges, and add tests for invalid input.
- "mm/damon: misc cleanups" (Zenghui Yu):
Clean up a collection of small DAMON rough edges in the core,
interfaces and tests. No large behavior change here, just simpler
code and fewer opportunities for confusion.
- "mm/damon: cleanup code, add test cases, and update guidances in
docs" (SJ Park):
Clean up DAMON's core and interfaces, add missing test cases and
bring the documentation into line with the code. These changes should
make the newer facilities easier to understand and maintain.
- "mm/damon: introduce hugepage_size probe filter" (SJ Park):
Add a DAMON probe filter for hugepage size. Let users select a range
of folio sizes through sysfs, validate the range and document the new
interface, so monitoring can concentrate on memory backed by the
desired hugepage sizes.
- "selftests/mm: pagemap_ioctl test fixes and cleanups" (Zenghui Yu):
Fix some pagemap ioctl test assumptions and clean up their setup and
reporting. Make failures point to the operation that went wrong
rather than to test harness bookkeeping.
- "mm: avoid large folio splits when swap is unavailable" (Johannes
Weiner):
Avoid splitting large folios merely to attempt swap when no swap
space is available. There is little point paying for an expensive
split when reclaim cannot actually swap the resulting pages out.
- "mm: workingset: fix the shadow node budget under MGLRU" (Hui Zhu):
Fix the workingset shadow-node budget with MGLRU enabled. Keep the
shadow entries within the intended limits so refault tracking does
not quietly consume more metadata than it should.
- "selftests/mm: TAP output and global-state fixes" (Song Hu):
Fix selftests which produce misleading TAP output or fail to restore
global settings after a run. A test ought to leave the system as it
found it, particularly when a whole suite follows behind.
- "mm: remove page_swap_entry()" (Tal Zussman):
Remove page_swap_entry() and convert its remaining users to the folio
and swap-entry helpers. Lose an old page-level interface and make the
swap paths a little more consistent.
- "mm: refactor zonelist constructors and iterators" (Gregory Price):
Refactor the zonelist construction and walk helpers to reduce
duplicated logic. Keep the allocation ordering rules intact while
making it clearer how zones are selected.
- "mm: kmemleak: batch the struct page scan" (Breno Leitao):
Scan adjacent struct pages in batches instead of taking kmemleak_lock
once for every single page. This preserves scan coverage and the
lock-hold bound, while cutting overhead substantially on debug
kernels.
- "mm/damon: misc improvements in tests and documents" (Zenghui Yu):
Fix some DAMON selftests which leave kdamond running or fail under
Python safe-path mode, and improve the KUnit coverage and
documentation. Nothing especially glamorous, but these are worth
doing.
- "mm/damon: introduce probe_hits_wsum DAMOS core filter" (SJ Park):
Add a DAMOS core filter using weighted probe-hit counts. This gives
schemes another way to select regions based on what the monitoring
probes observed.
- "mm/mglru: fix ineffective memory protection for non-kswapd reclaim"
(Ridong Chen):
Fix MGLRU's memory protection handling for reclaim outside kswapd.
The protection settings weren't having the intended effect in these
paths, leaving protected memory more exposed than requested.
- "mm/damon: improve readability, clarity and test coverage" (SJ Park):
Simplify a handful of DAMON core helpers, clarify their behavior and
expand test coverage. There are no intended userspace-visible changes
in this batch.
- "mm: replace PF_KCOMPACTD/PF_KSWAPD with kthread_func()" (Kefeng
Wang):
Stop identifying reclaim and compaction workers through the PF_KSWAPD
and PF_KCOMPACTD task flags. Use their kthread functions instead and
remove the redundant flag plumbing.
- "mm/hugetlb: fix surplus accounting and availability checks during
demotion" (Longlong Xia):
Fix HugeTLB surplus accounting and availability checks when demoting
huge pages. Keep the counters and reservation decisions consistent as
one huge page turns into several smaller ones.
- "mm/damon: add kunit tests for probe_hits handling and probe params
validation" (Jason Angelov):
Add KUnit tests for DAMON probe-hit handling and invalid probe
parameters. These cover edge cases that were too easy to miss in
ordinary monitoring tests.
- "mm/damon: fix misc bugs in kunit, quota goals and sysfs refresh_ms"
(SJ Park):
Fix a few DAMON bugs in KUnit tests, quota-goal handling and the
sysfs refresh interval. Small fixes, but the incorrect results and
control behavior were real.
- "mm: zswap: optimize zswap invalidate and store" (Kefeng Wang):
Simplify zswap's invalidate and store paths to avoid avoidable
lookups and work. Keep the entry lifetime rules intact while making
those frequent operations cheaper.
- "mm: replace custom ratelimiting logic" (Pedro Falcato):
Replace hand-rolled MM rate limiting with the common kernel helpers.
There is no need to maintain another local version of the same
mechanism.
- "mm/vmalloc: minor cleanups" (Ye Liu):
Tidy some vmalloc initialization and reporting code, and pull
duplicated vmap-area insertion work into a helper. No functional
changes are intended.
- "mm/mglru: clean up isolate_folios for readability and clarity"
(Ridong Chen):
Clean up MGLRU's isolate_folios() so the decisions are easier to read
and verify. Keep reclaim behavior unchanged while removing
unnecessary twists in the control flow.
- "Catch automatic storage in IDA and Maple Tree definitions" (Yury
Norov):
Catch IDA and maple-tree definitions placed in automatic storage
where the lifetime rules are wrong or easy to get wrong. Add checks
so such mistakes are harder to slip through unnoticed.
- "mm, swap: some random fixes and cleanups" (Kemeng Shi):
Fix and simplify a few unrelated swap paths, removing redundant work
and tidying up error handling. A small collection of things which
were better fixed than left lying around.
- "mm/page_io: folio conversion cleanups" (Tal Zussman):
Finish some page-to-folio conversions in swap I/O, removing obsolete
wrappers and using folio operations consistently. The I/O behavior is
intended to remain the same.
- "mm/swap: skip empty clusters in the swapoff scan" (Youngjun Park):
Skip empty swap clusters during swapoff instead of walking slots
which have nothing to release. This avoids pointless work on sparsely
used swap devices.
- "mm: trace: decode architecture-specific VMA flags" (Meijing Zhao):
Teach the VMA trace output to decode architecture-specific flags,
rather than leaving them as unexplained bits. That should make
tracing a good deal more useful when debugging unusual mappings.
- "mm, swap: don't spin on a bad swap entry" (Breno Leitao):
Stop repeatedly retrying a swap entry which is already known to be
bad. Make the error path terminate cleanly instead of wasting CPU
chasing an entry that cannot be used.
- "mm/page_isolation: fix UBSAN shift-out-of-bounds in
isolate_single_pageblock" (Qi Xi):
Fix an out-of-range shift in isolate_single_pageblock() and tighten
its associated boundary handling. UBSAN found the issue; the result
should be more robust pageblock isolation.
- "mm: khugepaged: fix tracepoint UAF" (Vernon Yang):
Fix a use-after-free in khugepaged tracepoint handling. Make sure the
information recorded by tracing stays valid for as long as the trace
event needs it.
- "mm/damon: fix DAMOS bugs in core, paddr and vaddr" (SJ Park):
Fix several DAMOS problems across the core and physical- and
virtual-address backends. The changes close error cases and bring the
implementations back into agreement.
- "mm/hugetlb_cgroup: move the per-node usage along with the folio"
(Hongfu Li):
Move HugeTLB per-node usage accounting when folios migrate between
nodes or are reparented to another cgroup. Otherwise numa_stat can
retain stale charges and even underflow when the folio is freed.
- "docs/core-api: memory-allocation: add k[mz]alloc_obj() and clarify
kmalloc" (Mike Rapoport):
Document the typed k[mz]alloc_obj() allocation helpers and clear up
some kmalloc guidance. Make it easier for callers to choose the right
interface.
- "selftests/mm: Validate selections and scope memfd_secret setup"
(Tianyi Chen):
Check selftest selections before running and confine memfd_secret
setup to the cases that actually need it. Avoid confusing failures
caused by the test harness itself.
- "mm/damon: move zero damos quota target_value handling to the core
layer" (SJ Park):
Handle a zero DAMOS quota goal target in the core rather than making
each user of the interface invent its own rule. Keep quota decisions
consistent across callers.
- "mm/damon: Introduce a huge page collapsing mechanism using auto
tuning" (Asier Gutierrez):
Add an auto-tuned DAMON mechanism for collapsing huge pages. Use
monitoring feedback to decide when collapse is worthwhile rather than
forcing one fixed policy onto every workload.
- "kmemleak: fix stale documentation and raise the verbose default"
(Breno Leitao):
Bring the kmemleak documentation up to date and require a third
confirming scan before verbose kernels print a suspected leak. The
extra scan helps suppress transient false positives in noisy debug
builds.
- "mm/memory: improve insert_pages() error handling" (Avi Weiss):
Tighten insert_pages() error handling so partial failures are
reported and cleaned up consistently. The caller should not have to
guess how far the operation got.
- "mm/damon: allow NULL or unstarted damon_ctx parameter for
damon_call()" (SJ Park):
Make damon_call() handle NULL and not-yet-started contexts safely.
Simplify the rules for callers and avoid an indefinite wait on an
invalid state.
- "mm/damon: introduce pgidle_set probe filter type" (SJ Park):
Add a pgidle_set DAMON probe filter, giving monitoring users another
way to select pages by their idle-tracking state. Wire it through the
filter interfaces and tests.
- "mm/damon/core: remove unused helper functions" (Cheng-Han Wu):
Remove DAMON core helpers which no longer have callers. Keep the API
surface smaller rather than retaining dead code just in case.
- "mm: stop calling pmd_folio() on special PMDs" (Gregory Price):
Avoid calling pmd_folio() on special PMD mappings that have no struct
page. The old assumption could fault in procfs walkers; use the
normal-folio checks and handle those mappings properly.
- "mm: remove three unused helpers from mm.h" (Tal Zussman):
Remove three unused helpers from mm.h. Fewer obsolete interfaces for
the next person to stumble over.
- "mm/damon: improvements in efficiency, error handling, documents"
(Liew Rui Yan):
Skip needless DAMOS quota work once the quota is full, propagate a
missing sysfs error and tidy a couple of comments. The ignored error
could make a monitoring-start request appear successful when it
wasn't.
- "docs/mm/damon/design: add explanation of nr_snapshots" (Liew Rui
Yan):
Explain DAMON's nr_snapshots accounting in the design documentation
and straighten out the associated terminology. Make it clearer what
users should expect from that counter.
- "selftests/cgroup: fixes for test_zswap on single core VM" (Wilson
Felipe Pereira):
Fix two test_zswap failures on single-CPU VMs. Wait for the exiting
child's cgroup to become unpopulated before reconfiguring it, and
avoid another test assumption which breaks under constrained
scheduling.
- "docs/ksm: fix advisor documentation and comment" (Jinjiang Tu):
Correct the KSM advisor documentation and a misleading code comment.
Keep the description aligned with how the advisor actually works.
- "Fix bugs in HugeTLB allocation when mem_cgroup_charge_hugetlb()
fails" (Ackerley Tng):
Fix HugeTLB allocation cleanup when the memcg charge fails. Make sure
failure exits undo what was allocated rather than leaving stale
accounting or resources behind.
- "mm, memcg: fix memory.peak reset clobbering other fds' watermark"
(Ridong Chen):
Fix memory.peak reset handling so resetting one file descriptor
doesn't overwrite the watermark kept for another. Keep those per-open
measurements independent as intended.
- "mm: reject zone device folios in more folio walkers" (Gregory
Price):
Reject zone-device folios in page walkers that cannot safely handle
them. Extend the existing checks rather than assuming every folio has
ordinary system-memory backing.
- "percpu: remove code with no effect" (Sang-Heon Jeon):
Remove a few pieces of percpu code which no longer do anything. There
is no reason to keep maintaining dead paths.
* tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (539 commits)
mm: shmem: assign a lock class to the unused huge shrinklist
mm: vmscan: put rotation-missed folios at the LRU tail
mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches
mm: kmemleak: move the struct page scan into a helper
mm/damon: document that a zero sample_interval is accepted
mm/damon: fix typos in comments
mm/damon/sysfs: propagate damon_call() error in turn_damon_on
mm/damon/core: skip quota score setup when the quota is full
mm/damon/tests/core-kunit: test the size charged for a filter-trimmed region
mm/damon/core: charge only the part of a region the filter left
mm: disallow raw PFN mappings of huge/shared zeropage
selftests/mm: zap whole PTE tables in the khugepaged race harness
selftests/mm: add memory-pressure threads to the khugepaged race harness
selftests/mm: race the collapse of windows with holes
selftests/mm: add khugepaged race harness
selftests/mm: check that one khugepaged pass collapses one window
selftests/mm: run every supported collapse order by default
selftests/mm: cover a shared-source collapse write race
selftests/mm: parameterize the mixed-source collapse case by source order
selftests/mm: add order-parameterized khugepaged collapse cases
...
296 lines
8.9 KiB
C
296 lines
8.9 KiB
C
/* SPDX-License-Identifier: GPL-2.0 */
|
|
#ifndef __CGROUP_INTERNAL_H
|
|
#define __CGROUP_INTERNAL_H
|
|
|
|
#include <linux/cgroup.h>
|
|
#include <linux/kernfs.h>
|
|
#include <linux/workqueue.h>
|
|
#include <linux/list.h>
|
|
#include <linux/refcount.h>
|
|
#include <linux/fs_parser.h>
|
|
|
|
#define TRACE_CGROUP_PATH_LEN 1024
|
|
extern spinlock_t trace_cgroup_path_lock;
|
|
extern char trace_cgroup_path[TRACE_CGROUP_PATH_LEN];
|
|
extern void __init enable_debug_cgroup(void);
|
|
|
|
/*
|
|
* cgroup_path() takes a spin lock. It is good practice not to take
|
|
* spin locks within trace point handlers, as they are mostly hidden
|
|
* from normal view. As cgroup_path() can take the kernfs_rename_lock
|
|
* spin lock, it is best to not call that function from the trace event
|
|
* handler.
|
|
*
|
|
* Note: trace_cgroup_##type##_enabled() is a static branch that will only
|
|
* be set when the trace event is enabled.
|
|
*/
|
|
#define TRACE_CGROUP_PATH(type, cgrp, ...) \
|
|
do { \
|
|
if (trace_cgroup_##type##_enabled()) { \
|
|
unsigned long flags; \
|
|
spin_lock_irqsave(&trace_cgroup_path_lock, \
|
|
flags); \
|
|
cgroup_path(cgrp, trace_cgroup_path, \
|
|
TRACE_CGROUP_PATH_LEN); \
|
|
trace_cgroup_##type(cgrp, trace_cgroup_path, \
|
|
##__VA_ARGS__); \
|
|
spin_unlock_irqrestore(&trace_cgroup_path_lock, \
|
|
flags); \
|
|
} \
|
|
} while (0)
|
|
|
|
/*
|
|
* The cgroup filesystem superblock creation/mount context.
|
|
*/
|
|
struct cgroup_fs_context {
|
|
struct kernfs_fs_context kfc;
|
|
struct cgroup_root *root;
|
|
struct cgroup_namespace *ns;
|
|
unsigned int flags; /* CGRP_ROOT_* flags */
|
|
|
|
/* cgroup1 bits */
|
|
bool cpuset_clone_children;
|
|
bool none; /* User explicitly requested empty subsystem */
|
|
bool all_ss; /* Seen 'all' option */
|
|
u32 subsys_mask; /* Selected subsystems */
|
|
char *name; /* Hierarchy name */
|
|
char *release_agent; /* Path for release notifications */
|
|
};
|
|
|
|
static inline struct cgroup_fs_context *cgroup_fc2context(struct fs_context *fc)
|
|
{
|
|
struct kernfs_fs_context *kfc = fc->fs_private;
|
|
|
|
return container_of(kfc, struct cgroup_fs_context, kfc);
|
|
}
|
|
|
|
struct cgroup_pidlist;
|
|
|
|
struct cgroup_file_ctx {
|
|
struct cgroup_namespace *ns;
|
|
|
|
struct {
|
|
void *trigger;
|
|
} psi;
|
|
|
|
struct {
|
|
bool started;
|
|
struct css_task_iter iter;
|
|
} procs;
|
|
|
|
struct {
|
|
struct cgroup_pidlist *pidlist;
|
|
} procs1;
|
|
|
|
struct cgroup_of_peak peak;
|
|
};
|
|
|
|
/*
|
|
* A cgroup can be associated with multiple css_sets as different tasks may
|
|
* belong to different cgroups on different hierarchies. In the other
|
|
* direction, a css_set is naturally associated with multiple cgroups.
|
|
* This M:N relationship is represented by the following link structure
|
|
* which exists for each association and allows traversing the associations
|
|
* from both sides.
|
|
*/
|
|
struct cgrp_cset_link {
|
|
/* the cgroup and css_set this link associates */
|
|
struct cgroup *cgrp;
|
|
struct css_set *cset;
|
|
|
|
/* list of cgrp_cset_links anchored at cgrp->cset_links */
|
|
struct list_head cset_link;
|
|
|
|
/* list of cgrp_cset_links anchored at css_set->cgrp_links */
|
|
struct list_head cgrp_link;
|
|
};
|
|
|
|
/* used to track tasks and csets during migration */
|
|
struct cgroup_taskset {
|
|
/* the src and dst cset list running through cset->mg_node */
|
|
struct list_head src_csets;
|
|
struct list_head dst_csets;
|
|
|
|
/* the number of tasks in the set */
|
|
int nr_tasks;
|
|
|
|
/* the subsys currently being processed */
|
|
int ssid;
|
|
|
|
/*
|
|
* Fields for cgroup_taskset_*() iteration.
|
|
*
|
|
* Before migration is committed, the target migration tasks are on
|
|
* ->mg_tasks of the csets on ->src_csets. After, on ->mg_tasks of
|
|
* the csets on ->dst_csets. ->csets point to either ->src_csets
|
|
* or ->dst_csets depending on whether migration is committed.
|
|
*
|
|
* ->cur_csets and ->cur_task point to the current task position
|
|
* during iteration.
|
|
*/
|
|
struct list_head *csets;
|
|
struct css_set *cur_cset;
|
|
struct task_struct *cur_task;
|
|
};
|
|
|
|
/* migration context also tracks preloading */
|
|
struct cgroup_mgctx {
|
|
/*
|
|
* Preloaded source and destination csets. Used to guarantee
|
|
* atomic success or failure on actual migration.
|
|
*/
|
|
struct list_head preloaded_src_csets;
|
|
struct list_head preloaded_dst_csets;
|
|
|
|
/* tasks and csets to migrate */
|
|
struct cgroup_taskset tset;
|
|
|
|
/* subsystems affected by migration */
|
|
u32 ss_mask;
|
|
};
|
|
|
|
#define CGROUP_TASKSET_INIT(tset) \
|
|
{ \
|
|
.src_csets = LIST_HEAD_INIT(tset.src_csets), \
|
|
.dst_csets = LIST_HEAD_INIT(tset.dst_csets), \
|
|
.csets = &tset.src_csets, \
|
|
}
|
|
|
|
#define CGROUP_MGCTX_INIT(name) \
|
|
{ \
|
|
LIST_HEAD_INIT(name.preloaded_src_csets), \
|
|
LIST_HEAD_INIT(name.preloaded_dst_csets), \
|
|
CGROUP_TASKSET_INIT(name.tset), \
|
|
}
|
|
|
|
#define DEFINE_CGROUP_MGCTX(name) \
|
|
struct cgroup_mgctx name = CGROUP_MGCTX_INIT(name)
|
|
|
|
extern struct cgroup_subsys *cgroup_subsys[];
|
|
extern struct list_head cgroup_roots;
|
|
extern bool cgrp_dfl_visible;
|
|
|
|
/* iterate across the hierarchies */
|
|
#define for_each_root(root) \
|
|
list_for_each_entry_rcu((root), &cgroup_roots, root_list, \
|
|
lockdep_is_held(&cgroup_mutex))
|
|
|
|
/**
|
|
* for_each_subsys - iterate all enabled cgroup subsystems
|
|
* @ss: the iteration cursor
|
|
* @ssid: the index of @ss, CGROUP_SUBSYS_COUNT after reaching the end
|
|
*/
|
|
#define for_each_subsys(ss, ssid) \
|
|
for ((ssid) = 0; (ssid) < CGROUP_SUBSYS_COUNT && \
|
|
(((ss) = cgroup_subsys[ssid]) || true); (ssid)++)
|
|
|
|
static inline bool notify_on_release(const struct cgroup *cgrp)
|
|
{
|
|
return test_bit(CGRP_NOTIFY_ON_RELEASE, &cgrp->flags);
|
|
}
|
|
|
|
void put_css_set_locked(struct css_set *cset);
|
|
|
|
static inline void put_css_set(struct css_set *cset)
|
|
{
|
|
unsigned long flags;
|
|
|
|
/*
|
|
* Ensure that the refcount doesn't hit zero while any readers
|
|
* can see it. Similar to atomic_dec_and_lock(), but for an
|
|
* rwlock
|
|
*/
|
|
if (refcount_dec_not_one(&cset->refcount))
|
|
return;
|
|
|
|
spin_lock_irqsave(&css_set_lock, flags);
|
|
put_css_set_locked(cset);
|
|
spin_unlock_irqrestore(&css_set_lock, flags);
|
|
}
|
|
|
|
/*
|
|
* refcounted get/put for css_set objects
|
|
*/
|
|
static inline void get_css_set(struct css_set *cset)
|
|
{
|
|
refcount_inc(&cset->refcount);
|
|
}
|
|
|
|
bool cgroup_ssid_enabled(int ssid);
|
|
|
|
struct cgroup_root *cgroup_root_from_kf(struct kernfs_root *kf_root);
|
|
struct cgroup *task_cgroup_from_root(struct task_struct *task,
|
|
struct cgroup_root *root);
|
|
struct cgroup *cgroup_kn_lock_live(struct kernfs_node *kn, bool drain_offline);
|
|
void cgroup_kn_unlock(struct kernfs_node *kn);
|
|
int cgroup_path_ns_locked(struct cgroup *cgrp, char *buf, size_t buflen,
|
|
struct cgroup_namespace *ns);
|
|
|
|
void cgroup_favor_dynmods(struct cgroup_root *root, bool favor);
|
|
void cgroup_free_root(struct cgroup_root *root);
|
|
void init_cgroup_root(struct cgroup_fs_context *ctx);
|
|
int cgroup_setup_root(struct cgroup_root *root, u32 ss_mask);
|
|
int rebind_subsystems(struct cgroup_root *dst_root, u32 ss_mask);
|
|
int cgroup_do_get_tree(struct fs_context *fc);
|
|
|
|
int cgroup_migrate_vet_dst(struct cgroup *dst_cgrp);
|
|
void cgroup_migrate_finish(struct cgroup_mgctx *mgctx);
|
|
void cgroup_migrate_add_src(struct css_set *src_cset, struct cgroup *dst_cgrp,
|
|
struct cgroup_mgctx *mgctx);
|
|
int cgroup_migrate_prepare_dst(struct cgroup_mgctx *mgctx);
|
|
int cgroup_migrate(struct task_struct *leader, bool threadgroup,
|
|
struct cgroup_mgctx *mgctx);
|
|
|
|
int cgroup_attach_task(struct cgroup *dst_cgrp, struct task_struct *leader,
|
|
bool threadgroup);
|
|
void cgroup_attach_lock(enum cgroup_attach_lock_mode lock_mode,
|
|
struct task_struct *tsk);
|
|
void cgroup_attach_unlock(enum cgroup_attach_lock_mode lock_mode,
|
|
struct task_struct *tsk);
|
|
struct task_struct *cgroup_procs_write_start(char *buf, bool threadgroup,
|
|
enum cgroup_attach_lock_mode *lock_mode)
|
|
__acquires(&cgroup_threadgroup_rwsem);
|
|
void cgroup_procs_write_finish(struct task_struct *task,
|
|
enum cgroup_attach_lock_mode lock_mode)
|
|
__releases(&cgroup_threadgroup_rwsem);
|
|
|
|
int cgroup_lock_and_drain_offline(struct cgroup *cgrp);
|
|
|
|
int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, umode_t mode);
|
|
int cgroup_rmdir(struct kernfs_node *kn);
|
|
int cgroup_show_path(struct seq_file *sf, struct kernfs_node *kf_node,
|
|
struct kernfs_root *kf_root);
|
|
|
|
int __cgroup_task_count(const struct cgroup *cgrp);
|
|
int cgroup_task_count(const struct cgroup *cgrp);
|
|
|
|
/*
|
|
* rstat.c
|
|
*/
|
|
int css_rstat_init(struct cgroup_subsys_state *css);
|
|
void css_rstat_exit(struct cgroup_subsys_state *css);
|
|
int ss_rstat_init(struct cgroup_subsys *ss);
|
|
void cgroup_base_stat_cputime_show(struct seq_file *seq);
|
|
|
|
/*
|
|
* namespace.c
|
|
*/
|
|
extern const struct proc_ns_operations cgroupns_operations;
|
|
|
|
/*
|
|
* cgroup-v1.c
|
|
*/
|
|
extern struct cftype cgroup1_base_files[];
|
|
extern struct kernfs_syscall_ops cgroup1_kf_syscall_ops;
|
|
extern const struct fs_parameter_spec cgroup1_fs_parameters[];
|
|
|
|
int proc_cgroupstats_show(struct seq_file *m, void *v);
|
|
void cgroup1_pidlist_destroy_all(struct cgroup *cgrp);
|
|
void cgroup1_release_agent(struct work_struct *work);
|
|
void cgroup1_check_for_release(struct cgroup *cgrp);
|
|
int cgroup1_parse_param(struct fs_context *fc, struct fs_parameter *param);
|
|
int cgroup1_get_tree(struct fs_context *fc);
|
|
int cgroup1_reconfigure(struct fs_context *ctx);
|
|
|
|
#endif /* __CGROUP_INTERNAL_H */
|