mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-10-26 10:57:38 +08:00
master
5195
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Linus Torvalds
|
55fbedcec6 |
Merge tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
Pull bpf updates from Alexei Starovoitov:
"Major changes:
- Introduce scalar evolution (SCEV) analysis in the verifier.
It infers how registers and stack slots change in a loop and lets
the verifier check bounded loops with induction variables as ranges
instead of walking every iteration. The number of processed insns
drops ten fold in programs with loops (Eduard Zingerman)
- Add 'callx' instruction: indirect call of a static BPF subprogram.
The address is taken with ld_imm64 or loaded from read-only data.
Supported by x86-64, arm64 and riscv64 JITs (Alexei Starovoitov, Pu
Lehui)
- Add unwind_goto and unwind_exit instructions, so that Rust panic or
C++ exception can run destructors of every frame it passes through.
The normal path is free of unwind code. Supported by x86-64 and
arm64 JITs (Alexei Starovoitov)
- Support programs compiled by rust-bpf: treat load and store through
a number as arena access with BPF_F_ARENA_SCALAR, accept rust names
in BTF (Alexei Starovoitov)
- Raise BPF program stack size from 512 bytes to 2 KiB on x86-64 and
arm64 (Kumar Kartikeya Dwivedi)
- Support struct, union and __int128 up to 16 bytes passed and
returned by value in BPF subprograms and kfuncs, and fix x86-64 and
arm64 trampolines for such arguments (Yonghong Song)
- Introduce a common way to attach struct_ops to a cgroup via
bpf_link. bpf_tcp_ops is the first user and mirrors TCP sock_ops
callbacks (Martin KaFai Lau, Amery Hung)
- Describe inlined functions in BTF. New kinds tell where the inline
sites are and how to fetch the arguments there.
CONFIG_DEBUG_INFO_BTF_INLINE keeps it apart from the main BTF in
.BTF.inline section or in btf_vmlinux_inline.ko (Alan Maguire)
- Fix many memory safety, lifetime and locking bugs in BPF core,
verifier, iterators, local storage, dynptr, devmap, cpumap and
seg6local (Josef Bacik, Alexei Starovoitov)
- Diff stat is dominated by selftests and by removal of
lib/test_bpf.c. Its 16k lines of instruction tests were converted
to BPF selftests that BPF CI runs on every architecture it covers
(Daniel Borkmann)
Other features and fixes:
- Add KASAN checks to memory accesses of JITed programs on x86-64
(Alexis Lothoré)
- Complete unification of helper, kfunc and subprog argument checks
in the verifier (Amery Hung)
- Add bpf_copy_from_user_mm[_str]() kfuncs (Anastasios Papagiannis)
- Various features and fixes (Daniel Borkmann):
- Add .bpf keyring for program signature verification, modelled
after dm-verity keyring, and support ML-DSA in signing tools
- Emit shorter and faster code in x86-64 JIT for patterns that
LLVM generates
- Fix various issues in gotox handling
- Let BPF LSM set xattrs at inode creation with bpf_inode_init_xattr()
(David Windsor, Daniel Borkmann)
- Use sleepable page allocation in sleepable arena paths. Fix races
between arena free, allocation and user page fault (Emil Tsalapatis)
- Switch BPF crypto from lskcipher to AES-CBC and AES-ECB libraries
(Eric Biggers)
- Add bpf_proactive_reclaim() kfunc (Hui Zhu)
- Render decl_tags in libbpf btf_dump and add tests for bpftool C
dump (Ihor Solodrai)
- Speed up tracing_multi attach on x86 (Jiri Olsa)
- Various features and fixes (Kumar Kartikeya Dwivedi):
- Add BPF_PROG_STREAM_OPEN command to read BPF streams through a
file descriptor with poll support
- Let kernel structs publish arena pointers with __arena type tag
- Fix __uninit kfunc output buffers and fastcall rewrite with
indirect stack accesses
- Refuse CO-RE relocations for signed programs
- Add enqueue_rcvq and dequeue_rcvq hooks to bpf_tcp_ops to adjust
sk_rcvlowat and avoid unnecessary wakeups (Kuniyuki Iwashima)
- Add bpf_ksock_sendto() kfunc and allow ksock connect and send from
bpf_wq (Mahe Tardy)
- Fix racy plt target update in arm64 bpf_arch_text_poke()
(Matthew Wood)
- Restructure BPF selftests Makefile as a layered build
(Mykola Lysenko)
- Speed up hash map lookups for u32 and u64 keys (Mykyta Yatsenko)
- Add signed div/mod, movsx, bswap and gotol to MIPS JITs, signed
div/mod to parisc JITs (Nicholas Dudar, Johan Almbladh)
- Fix stack overflow in build ID parser on 32-bit (Nickolai
Zeldovich)
- Add stack arguments, signed arena loads and gotox to riscv64 JIT.
Pu Lehui is now co-maintainer of riscv BPF JITs (Pu Lehui, Chen
Pei, Feng Jiang)
- Add bpf_call_rcu() and bpf_call_rcu_tasks_trace() kfuncs. Support
fetching and/or/xor atomics in arena on x86-64 (Puranjay Mohan)
- Fix flaky selftests, selftests with clang 23 and bpf-gcc
(Qiliang Yuan, KaFai Wan, Vineet Gupta)
- Fix builds broken by fatal resolve_btfids warnings, BTF generation
on ARC, and define arch_cmpxchg_local() on arc, hexagon and
openrisc (Sasha Levin)
- Reduce memory use of hash maps by up to 50% per element
(T.J. Mercier)
- Mark pointer fields of sched_ext ops arguments as trusted (Tejun
Heo)
- Remove legacy bpf_asm, bpf_dbg and bpf_jit_disasm tools (Xia Yan)
- Fix index overflow in queue and stack maps, arena range tree after
allocation failure, tail call loop after freplace detach, lost
signal in bpf_send_signal() (Yuan Chen)
- Many more fixes and cleanups"
* tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (660 commits)
selftests/bpf: Time tc_edt from its first packet with data
selftests/bpf: Test unwind_exit of an exception callback
selftests/bpf: Tests for 32-bit scalar evolution bounds computation
bpf: Reject unwind_goto and unwind_exit in offloaded progs
bpf: Support 32-bit linear scalar evolution
bpf: Make unwind_exit of an exception callback exit in JITs
selftests/bpf: Test forbidden bpf_ksock_sendto() LSM attach
selftests/bpf: Test ksock sendto()
bpf: Add bpf_ksock_sendto() kfunc
selftests/bpf: Test unwinding from R0: R2 subprog
selftests/bpf: Add a C++ prog with exceptions
selftests/bpf: Add tests for unwind_goto and unwind_exit
bpf, arm64: Support unwind_goto and unwind_exit
bpf, x86: Support unwind_goto and unwind_exit
bpf: Add unwind_goto and unwind_exit insns
tools/bpf: Remove legacy BPF tools
bpf: Annotate data races on bpf_lru_node type
selftests/bpf: Add arena allocation race tests
bpf: Fix arena race between page free and alloc leading to incoherency
bpf: Revalidate faulted arena pages before installing user mappings
...
|
||
|
Linus Torvalds
|
c5fd5e67c7 |
Merge tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
Pull MM updates from Andrew Morton:
- "selftests/mm: improve khugepaged coverage" (Kiryl Shutsemau):
Give khugepaged a proper workout. Extend the tests to all supported
mTHP orders, check exactly which windows collapsed, and make scan
completion deterministic. Race collapse against faults, pinning, fork
and remap; verify that the contents survive.
Add coverage for whole-page-table zaps and memory pressure too.
- "selftests/mm: separate GUP microbenchmarking from functional testing"
(Sarthak Sharma):
Untangle GUP benchmarking from correctness testing. Move the
benchmark into tools/mm, add a standalone harness-based GUP selftest,
and move shared helpers into tools/lib/mm. Give each tool one job
instead of mixing timing measurements with correctness checks.
Cover the mapping, pinning and batch-size combinations explicitly.
- "mm: Switch device DAX to section-based vmemmap optimization" (Muchun
Song):
Apply section-based vmemmap optimization to device DAX. Rework the
backing metadata layout to save memory while preserving the handling
of device memory hotplug and its different alignment and lifetime
requirements.
- "mm: Introduce section-based vmemmap optimization for HugeTLB"
(Muchun Song):
Cut the memory consumed by HugeTLB's struct page array with
section-based vmemmap optimization. Arrange the metadata so large
huge pages need fewer backing pages, while keeping initialization,
teardown and the various hugepage configurations straight.
- "mm/huge_memory: clean up and decouple the anon and file split
helpers" (Kairui Song):
Separate anonymous and file-backed folio splitting, which have rather
different locking and bookkeeping requirements. Move the work into
separate helpers, drop dead checks and tidy swap-cache handling. Fix
error paths, including one that could leave a folio frozen.
Keep the swap cache cases separate from file-cache splitting.
- "mm: Unconditional per-VMA locks and cleanups" (Dave Hansen):
Make per-VMA locking unconditional and clean up the old conditional
code around it. This removes configuration-dependent plumbing and
lets callers depend on one locking model rather than carrying both
cases along.
- "memcg: remove the v1 soft limit" (Shakeel Butt):
Remove the old cgroup v1 memory soft-limit machinery. This has long
since been superseded by the newer memory-control interfaces; hauling
around a separate reclaim scheme for it no longer buys us much.
- "read proc/pid/smaps_rollup under per-vma lock" (Suren Baghdasaryan):
Read /proc/PID/smaps_rollup using per-VMA locks where possible,
rather than holding mmap_lock across the entire walk. Retain the
contention fallback and add tearing tests. This considerably reduces
latency when another thread is busy changing its mappings.
- "mm/damon: introduce data access-as-a-data attribute" (SJ Park):
Teach DAMON to expose memory access observations as data attributes,
making them available to the probe and filtering machinery. Extend
the interfaces, documentation and tests so the new information can be
used rather than merely collected.
- "Remove PG_private by using page/folio->private checks instead" (Zi
Yan):
Get rid of PG_private and use the folio's private pointer to say
whether private data is attached. Convert the filesystem users,
including erofs, f2fs and md, and clean up the old accessors and
flags. There are a few special cases, notably swapcache and hugetlb,
which need explicit care.
- "mm: make userland page table freeing RCU-safe" (Lorenzo Stoakes):
Make freeing userspace page tables safe against RCU readers. Sort out
the page-table lifetime and locking rules so walkers cannot stumble
over freed tables while mappings are being torn down.
- "make unused huge shrinker memcg aware" (Qi Zheng):
Make shmem's unused-huge-folio shrinker aware of memory cgroups, so
reclaim can find those folios under the right memcg. Wire up the
per-memcg tracking and its lifetime rules, and give the shrinklist
its own lockdep class to avoid spurious dependency reports.
- "mm: Drop pxd_ERROR()" (Anshuman Khandual):
Remove the old pxd_ERROR() page-table checks and their
architecture-specific baggage. Simplify the common page-table
interfaces and callers now that these checks no longer serve a
useful purpose.
- "memcg: group struct fields by access pattern" (Shakeel Butt):
Rearrange memcg's hot and cold fields according to how they're
actually accessed. Keep frequently updated counters and their
neighboring state from fighting over cachelines and reduce needless
cacheline traffic.
- "mm/damon/core: fix the size charged for a filter-trimmed region"
(Donggeun Yoo):
Fix DAMOS quota accounting when a filter trims a monitoring region.
Charge only the part left after filtering, not the original region,
and add KUnit coverage. Previously the excluded memory could eat into
the quota and make schemes run more slowly than requested.
- "mm/damon: add kunit and selftests for probes and probe weights" (SJ
Park):
Add more tests for DAMON probes and their weights, including
validation and the results reported to userspace. This is mostly
about catching regressions in the newer probe infrastructure before
they become harder to diagnose.
- "mm: memcontrol: constify the read side of the read side of the memcg
API" (Tal Zussman):
Mark the read-only parts of the memcg interface const, from folio and
lruvec accessors to counters and swap helpers. Most of this is
mechanical, but it makes the contracts clearer and lets the compiler
catch accidental writes.
- "mm/damon/vaddr: support {prep,apply}_probes" (SJ Park):
Support the new prepare/apply probe operations in DAMON's
virtual-address backend. This brings vaddr monitoring into line with
the probe infrastructure and makes the added probe types useful there
too.
- "mm/mglru: clean up folio counters and flag usage" (Kairui Song):
Simplify MGLRU's folio counters and flag handling. Remove redundant
accounting paths and make the state transitions easier to follow
without changing the intended reclaim policy.
- "arch, mm/execmem: resolve confusion about set_direct_map_valid_noflush()"
(Mike Rapoport (Microsoft)):
Clarify and straighten out the set_direct_map_valid_noflush()
interface across architectures and execmem. Make the requested
mapping changes and the callers' expectations agree, rather than
relying on subtly different architecture behavior.
- "mm: zswap: free cold writeback folios promptly" (Alexandre Ghiti):
Free zswap writeback folios as soon as their disk write completes.
They are cold temporary buffers, so putting them on the LRU only
makes reclaim rediscover them later. Extend swap-cache dropbehind and
let zswap keep these folios off the LRU altogether.
- "mm: gup: cleanup gup_fast call chain" (Aristeu Rozanski):
Clean up the GUP-fast call chain and simplify how the fast-path
helpers pass their arguments and results around. Keep the page-pin
behavior the same while making the call sequence easier to maintain.
- "mm: optimize zone-device memmap initialization" (Li Zhe):
Speed up initialization of zone-device struct pages. Avoid redundant
work when setting up large device-memory ranges, where touching every
descriptor can get expensive.
- "mm/zswap: shrink zswap_entry via a pool id" (Jianyue Wu):
Reduce the size of zswap_entry by storing a compact pool identifier
instead of a full pointer. Keep the pool lookup and lifetime handling
correct while saving per-entry metadata.
- "mm/damon: support access monitoring of hugetlb-backed memory"
(Krishna Iyer):
Let DAMON monitor hugetlb-backed memory too. Extend the relevant
access checks and add coverage so those mappings no longer fall
through a hole in the monitoring machinery.
- "mm/execmem: fixes and cleanups for the ROX cache" (Mike Rapoport):
Tidy execmem's read-only-executable cache and fix its handling of the
awkward allocation and release cases. The aim is to make the cache's
ownership and permission transitions more predictable.
- "zsmallc: remove old object read API" (Sergey Senozhatsky):
Drop zsmalloc's old object-read API and move its remaining users to
the supported access mechanism. This removes a duplicate interface
which we no longer need.
- "mm/mempolicy: stop copying state in the interleave paths" (Gregory
Price):
Stop copying mempolicy state in the interleave paths when a reference
to the existing policy will do. This avoids extra work and makes the
ownership rules less obscure.
- "mm/mglru: speed up inc_min_seq() and fix cold/hot inversions" (Barry
Song):
Make MGLRU's generation advancement cheaper and fix cases where cold
and hot folios end up in the wrong order. Adjust the counter and list
handling so reclaim doesn't keep passing over the better eviction
candidates.
- "mm: Fix MF_DELAYED handling on memory failure" (Lisa Wang):
Treat MF_DELAYED as a partially completed memory-failure operation,
not an outright failure. Propagate that result correctly through
shmem and the generic handler, fix the extra-reference accounting,
and add tests for the resulting SIGBUS behavior.
- "tools/mm/page_owner_sort: fix --sort, add module filter, improve
usage" (Ye Liu):
Fix page_owner_sort's --sort handling, add module-based filtering and
make the usage output more helpful. This makes the tool less
frustrating when tracking down who is holding on to pages.
- "mm/damon: move damos filter range arguments validation to core" (SJ
Park):
Move DAMOS filter range validation into the core, where all callers
can share it. Also close the sysfs read-twice races which could pass
inconsistent ranges, and add tests for invalid input.
- "mm/damon: misc cleanups" (Zenghui Yu):
Clean up a collection of small DAMON rough edges in the core,
interfaces and tests. No large behavior change here, just simpler
code and fewer opportunities for confusion.
- "mm/damon: cleanup code, add test cases, and update guidances in
docs" (SJ Park):
Clean up DAMON's core and interfaces, add missing test cases and
bring the documentation into line with the code. These changes should
make the newer facilities easier to understand and maintain.
- "mm/damon: introduce hugepage_size probe filter" (SJ Park):
Add a DAMON probe filter for hugepage size. Let users select a range
of folio sizes through sysfs, validate the range and document the new
interface, so monitoring can concentrate on memory backed by the
desired hugepage sizes.
- "selftests/mm: pagemap_ioctl test fixes and cleanups" (Zenghui Yu):
Fix some pagemap ioctl test assumptions and clean up their setup and
reporting. Make failures point to the operation that went wrong
rather than to test harness bookkeeping.
- "mm: avoid large folio splits when swap is unavailable" (Johannes
Weiner):
Avoid splitting large folios merely to attempt swap when no swap
space is available. There is little point paying for an expensive
split when reclaim cannot actually swap the resulting pages out.
- "mm: workingset: fix the shadow node budget under MGLRU" (Hui Zhu):
Fix the workingset shadow-node budget with MGLRU enabled. Keep the
shadow entries within the intended limits so refault tracking does
not quietly consume more metadata than it should.
- "selftests/mm: TAP output and global-state fixes" (Song Hu):
Fix selftests which produce misleading TAP output or fail to restore
global settings after a run. A test ought to leave the system as it
found it, particularly when a whole suite follows behind.
- "mm: remove page_swap_entry()" (Tal Zussman):
Remove page_swap_entry() and convert its remaining users to the folio
and swap-entry helpers. Lose an old page-level interface and make the
swap paths a little more consistent.
- "mm: refactor zonelist constructors and iterators" (Gregory Price):
Refactor the zonelist construction and walk helpers to reduce
duplicated logic. Keep the allocation ordering rules intact while
making it clearer how zones are selected.
- "mm: kmemleak: batch the struct page scan" (Breno Leitao):
Scan adjacent struct pages in batches instead of taking kmemleak_lock
once for every single page. This preserves scan coverage and the
lock-hold bound, while cutting overhead substantially on debug
kernels.
- "mm/damon: misc improvements in tests and documents" (Zenghui Yu):
Fix some DAMON selftests which leave kdamond running or fail under
Python safe-path mode, and improve the KUnit coverage and
documentation. Nothing especially glamorous, but these are worth
doing.
- "mm/damon: introduce probe_hits_wsum DAMOS core filter" (SJ Park):
Add a DAMOS core filter using weighted probe-hit counts. This gives
schemes another way to select regions based on what the monitoring
probes observed.
- "mm/mglru: fix ineffective memory protection for non-kswapd reclaim"
(Ridong Chen):
Fix MGLRU's memory protection handling for reclaim outside kswapd.
The protection settings weren't having the intended effect in these
paths, leaving protected memory more exposed than requested.
- "mm/damon: improve readability, clarity and test coverage" (SJ Park):
Simplify a handful of DAMON core helpers, clarify their behavior and
expand test coverage. There are no intended userspace-visible changes
in this batch.
- "mm: replace PF_KCOMPACTD/PF_KSWAPD with kthread_func()" (Kefeng
Wang):
Stop identifying reclaim and compaction workers through the PF_KSWAPD
and PF_KCOMPACTD task flags. Use their kthread functions instead and
remove the redundant flag plumbing.
- "mm/hugetlb: fix surplus accounting and availability checks during
demotion" (Longlong Xia):
Fix HugeTLB surplus accounting and availability checks when demoting
huge pages. Keep the counters and reservation decisions consistent as
one huge page turns into several smaller ones.
- "mm/damon: add kunit tests for probe_hits handling and probe params
validation" (Jason Angelov):
Add KUnit tests for DAMON probe-hit handling and invalid probe
parameters. These cover edge cases that were too easy to miss in
ordinary monitoring tests.
- "mm/damon: fix misc bugs in kunit, quota goals and sysfs refresh_ms"
(SJ Park):
Fix a few DAMON bugs in KUnit tests, quota-goal handling and the
sysfs refresh interval. Small fixes, but the incorrect results and
control behavior were real.
- "mm: zswap: optimize zswap invalidate and store" (Kefeng Wang):
Simplify zswap's invalidate and store paths to avoid avoidable
lookups and work. Keep the entry lifetime rules intact while making
those frequent operations cheaper.
- "mm: replace custom ratelimiting logic" (Pedro Falcato):
Replace hand-rolled MM rate limiting with the common kernel helpers.
There is no need to maintain another local version of the same
mechanism.
- "mm/vmalloc: minor cleanups" (Ye Liu):
Tidy some vmalloc initialization and reporting code, and pull
duplicated vmap-area insertion work into a helper. No functional
changes are intended.
- "mm/mglru: clean up isolate_folios for readability and clarity"
(Ridong Chen):
Clean up MGLRU's isolate_folios() so the decisions are easier to read
and verify. Keep reclaim behavior unchanged while removing
unnecessary twists in the control flow.
- "Catch automatic storage in IDA and Maple Tree definitions" (Yury
Norov):
Catch IDA and maple-tree definitions placed in automatic storage
where the lifetime rules are wrong or easy to get wrong. Add checks
so such mistakes are harder to slip through unnoticed.
- "mm, swap: some random fixes and cleanups" (Kemeng Shi):
Fix and simplify a few unrelated swap paths, removing redundant work
and tidying up error handling. A small collection of things which
were better fixed than left lying around.
- "mm/page_io: folio conversion cleanups" (Tal Zussman):
Finish some page-to-folio conversions in swap I/O, removing obsolete
wrappers and using folio operations consistently. The I/O behavior is
intended to remain the same.
- "mm/swap: skip empty clusters in the swapoff scan" (Youngjun Park):
Skip empty swap clusters during swapoff instead of walking slots
which have nothing to release. This avoids pointless work on sparsely
used swap devices.
- "mm: trace: decode architecture-specific VMA flags" (Meijing Zhao):
Teach the VMA trace output to decode architecture-specific flags,
rather than leaving them as unexplained bits. That should make
tracing a good deal more useful when debugging unusual mappings.
- "mm, swap: don't spin on a bad swap entry" (Breno Leitao):
Stop repeatedly retrying a swap entry which is already known to be
bad. Make the error path terminate cleanly instead of wasting CPU
chasing an entry that cannot be used.
- "mm/page_isolation: fix UBSAN shift-out-of-bounds in
isolate_single_pageblock" (Qi Xi):
Fix an out-of-range shift in isolate_single_pageblock() and tighten
its associated boundary handling. UBSAN found the issue; the result
should be more robust pageblock isolation.
- "mm: khugepaged: fix tracepoint UAF" (Vernon Yang):
Fix a use-after-free in khugepaged tracepoint handling. Make sure the
information recorded by tracing stays valid for as long as the trace
event needs it.
- "mm/damon: fix DAMOS bugs in core, paddr and vaddr" (SJ Park):
Fix several DAMOS problems across the core and physical- and
virtual-address backends. The changes close error cases and bring the
implementations back into agreement.
- "mm/hugetlb_cgroup: move the per-node usage along with the folio"
(Hongfu Li):
Move HugeTLB per-node usage accounting when folios migrate between
nodes or are reparented to another cgroup. Otherwise numa_stat can
retain stale charges and even underflow when the folio is freed.
- "docs/core-api: memory-allocation: add k[mz]alloc_obj() and clarify
kmalloc" (Mike Rapoport):
Document the typed k[mz]alloc_obj() allocation helpers and clear up
some kmalloc guidance. Make it easier for callers to choose the right
interface.
- "selftests/mm: Validate selections and scope memfd_secret setup"
(Tianyi Chen):
Check selftest selections before running and confine memfd_secret
setup to the cases that actually need it. Avoid confusing failures
caused by the test harness itself.
- "mm/damon: move zero damos quota target_value handling to the core
layer" (SJ Park):
Handle a zero DAMOS quota goal target in the core rather than making
each user of the interface invent its own rule. Keep quota decisions
consistent across callers.
- "mm/damon: Introduce a huge page collapsing mechanism using auto
tuning" (Asier Gutierrez):
Add an auto-tuned DAMON mechanism for collapsing huge pages. Use
monitoring feedback to decide when collapse is worthwhile rather than
forcing one fixed policy onto every workload.
- "kmemleak: fix stale documentation and raise the verbose default"
(Breno Leitao):
Bring the kmemleak documentation up to date and require a third
confirming scan before verbose kernels print a suspected leak. The
extra scan helps suppress transient false positives in noisy debug
builds.
- "mm/memory: improve insert_pages() error handling" (Avi Weiss):
Tighten insert_pages() error handling so partial failures are
reported and cleaned up consistently. The caller should not have to
guess how far the operation got.
- "mm/damon: allow NULL or unstarted damon_ctx parameter for
damon_call()" (SJ Park):
Make damon_call() handle NULL and not-yet-started contexts safely.
Simplify the rules for callers and avoid an indefinite wait on an
invalid state.
- "mm/damon: introduce pgidle_set probe filter type" (SJ Park):
Add a pgidle_set DAMON probe filter, giving monitoring users another
way to select pages by their idle-tracking state. Wire it through the
filter interfaces and tests.
- "mm/damon/core: remove unused helper functions" (Cheng-Han Wu):
Remove DAMON core helpers which no longer have callers. Keep the API
surface smaller rather than retaining dead code just in case.
- "mm: stop calling pmd_folio() on special PMDs" (Gregory Price):
Avoid calling pmd_folio() on special PMD mappings that have no struct
page. The old assumption could fault in procfs walkers; use the
normal-folio checks and handle those mappings properly.
- "mm: remove three unused helpers from mm.h" (Tal Zussman):
Remove three unused helpers from mm.h. Fewer obsolete interfaces for
the next person to stumble over.
- "mm/damon: improvements in efficiency, error handling, documents"
(Liew Rui Yan):
Skip needless DAMOS quota work once the quota is full, propagate a
missing sysfs error and tidy a couple of comments. The ignored error
could make a monitoring-start request appear successful when it
wasn't.
- "docs/mm/damon/design: add explanation of nr_snapshots" (Liew Rui
Yan):
Explain DAMON's nr_snapshots accounting in the design documentation
and straighten out the associated terminology. Make it clearer what
users should expect from that counter.
- "selftests/cgroup: fixes for test_zswap on single core VM" (Wilson
Felipe Pereira):
Fix two test_zswap failures on single-CPU VMs. Wait for the exiting
child's cgroup to become unpopulated before reconfiguring it, and
avoid another test assumption which breaks under constrained
scheduling.
- "docs/ksm: fix advisor documentation and comment" (Jinjiang Tu):
Correct the KSM advisor documentation and a misleading code comment.
Keep the description aligned with how the advisor actually works.
- "Fix bugs in HugeTLB allocation when mem_cgroup_charge_hugetlb()
fails" (Ackerley Tng):
Fix HugeTLB allocation cleanup when the memcg charge fails. Make sure
failure exits undo what was allocated rather than leaving stale
accounting or resources behind.
- "mm, memcg: fix memory.peak reset clobbering other fds' watermark"
(Ridong Chen):
Fix memory.peak reset handling so resetting one file descriptor
doesn't overwrite the watermark kept for another. Keep those per-open
measurements independent as intended.
- "mm: reject zone device folios in more folio walkers" (Gregory
Price):
Reject zone-device folios in page walkers that cannot safely handle
them. Extend the existing checks rather than assuming every folio has
ordinary system-memory backing.
- "percpu: remove code with no effect" (Sang-Heon Jeon):
Remove a few pieces of percpu code which no longer do anything. There
is no reason to keep maintaining dead paths.
* tag 'mm-stable-2026-10-19-21-33' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm: (539 commits)
mm: shmem: assign a lock class to the unused huge shrinklist
mm: vmscan: put rotation-missed folios at the LRU tail
mm: kmemleak: scan the struct page array in MAX_SCAN_SIZE batches
mm: kmemleak: move the struct page scan into a helper
mm/damon: document that a zero sample_interval is accepted
mm/damon: fix typos in comments
mm/damon/sysfs: propagate damon_call() error in turn_damon_on
mm/damon/core: skip quota score setup when the quota is full
mm/damon/tests/core-kunit: test the size charged for a filter-trimmed region
mm/damon/core: charge only the part of a region the filter left
mm: disallow raw PFN mappings of huge/shared zeropage
selftests/mm: zap whole PTE tables in the khugepaged race harness
selftests/mm: add memory-pressure threads to the khugepaged race harness
selftests/mm: race the collapse of windows with holes
selftests/mm: add khugepaged race harness
selftests/mm: check that one khugepaged pass collapses one window
selftests/mm: run every supported collapse order by default
selftests/mm: cover a shared-source collapse write race
selftests/mm: parameterize the mixed-source collapse case by source order
selftests/mm: add order-parameterized khugepaged collapse cases
...
|
||
|
Linus Torvalds
|
864c07de0c |
Merge tag 'hardening-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux
Pull hardening updates from Kees Cook: - Provide kmalloc_obj*-style API for devm_kmalloc() - Improve seq_buf API for saner C String handling with several bug fixes along the way (Kees Cook, Bill Wendling) - Overflow helper readability improvements (Bill Wendling) - Unexport native_write_cr4() (Dennis Chen) - um: fix CONFIG_GCOV for built-in code (Johannes Berg) - Improve stackinit test coverage (Justin Stitt) - KMSAN: fix memset() when using fortify-source, again (Arnd Bergmann) - lkdtm: use kmalloc() instead of __get_free_page (Mahad Ibrahim) - signalfd: don't dequeue the forced fatal signals (Oleg Nesterov) - Fix randstruct to recognize new container_of() behavior - coccinelle: kmalloc_obj: Convert multi-byte integral allocations too - Fix a couple remaining allocation return type mismatches * tag 'hardening-v7.4-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/kees/linux: (29 commits) overflow: Match arithmetic in both parts of __builtin_choose_expr lkdtm: Make UNSET_SMEP built-in-only and unexport native_write_cr4() extcon: realtek: Remove const from type_c_cfg allocation type coresight: cti: Add const to con_groups allocation type docs: core-api: Document the seq_buf API nvdimm: ndtest: Return the string length from flags_show() powerpc/papr_scm: Return the string length from the sysfs show functions seq_buf: Add seq_buf_init_append() bpf: Remove dead newline stripping from format_disasm_line() seq_buf: Add seq_buf_terminate() seq_buf: Add seq_buf_strlen() seq_buf: Clear what a writer did not claim when a seq_buf overflows seq_buf: Copy what fits when seq_buf_puts() and seq_buf_putmem() overflow scsi: lpfc: Print rx monitor records straight into the seq_buf seq_buf: Do not pop from an overflowed seq_buf seq_buf: Do not print an empty line from an overflowed seq_buf_do_printk() .get_maintainer.ignore: Add Kees Cook devres: Provide kmalloc_obj*-style API for devm_kmalloc mm: Move default_gfp() into gfp_types.h kunit/stackinit: Clean up XFAIL under instrumentation ... |
||
|
Alexei Starovoitov
|
9b27cd0554 |
bpf: Reject unwind_goto and unwind_exit in offloaded progs
A prog with unwind_goto or unwind_exit has to be JITed by a JIT that
supports them. The driver JITs an offloaded prog, and nobody looks at
jit_required there.
nfp rejects unwind_goto in nfp_verify_insn(). netdevsim takes everything.
Nothing runs such a prog, but it's not jited for bpf_insn_prepare_dump(),
which takes unwind_goto for BPF_CALL_ARGS and shows it as a call.
Fixes:
|
||
|
Eduard Zingerman
|
32e962577d |
bpf: Support 32-bit linear scalar evolution
Representation
--------------
Represent ALU32 recurrences as:
w0 += step -> (zext32 (linear r0 step))
Extend simplify() rules to maintain the above form for latch
expressions. E.g. for a post-condition loop such as:
w0 = 0;
l: w0 += 1;
if w0 != 3 goto l;
exit;
SCEV at header: (zext32 (linear r0 1))
Initial SCEV at latch: (zext32 (+ (zext32 (linear r0 1)) 1))
Simplified SCEV at latch: (zext32 (+ (zext32 (linear r0 1)) 1))
-> (zext32 (+ (linear r0 1) 1))
-> (zext32 (linear (+ r0 1) 1))
Iterations count computation
----------------------------
- Extract the main iteration-count logic from compute_max_iters() into
compute_backedge_taken_count(). Reuse its 64-bit arithmetic after
sign/zero extending the latch operands.
- Reject the computed count if the predicted exit value falls outside
signed/unsigned 32-bit range, e.g.:
w0 = 0x7ffffffe;
l: w0 += 1;
if w0 s<= 0x7fffffff goto l;
The 64-bit computation predicts an exit at S32_MAX + 1, but the
32-bit value wraps to S32_MIN and the comparison remains true.
- Handle mixed 64-bit/32-bit comparisons/recurrences only when initial
bounds and step fit in 32-bit ranges, e.g.:
w0 = 0;
r1 = 0x100000003 ll;
l: w0 += 1;
if r0 != r1 goto l;
Truncating the bound to 32 bits would predict three header visits.
However, `w0 += 1` keeps r0 within [0, U32_MAX], so it can never
be equal to the 64-bit bound.
Widened bounds computation
--------------------------
- Update linear_bounds() to truncate the initially computed 64-bit
range to 32 bits, then zero-extend it to obtain the 64-bit range
of iteration values. Step computation needs to handle:
- unsigned 32-bit wraparound;
- entry and zero-extended iteration values disagreeing on base/step.
- Union both ranges with the corresponding register ranges at entry,
since the entry value need not be zero-extended.
- Extend `struct bounds` to keep both {32,64}-bit ranges and pass both
to bpf_set_reg_range(). A 64-bit range alone loses useful 32-bit
information, e.g.:
entry = 0, step = -1, two header visits
r64: [0, U32_MAX]
r32: [-1, 0]
Signed-off-by: Eduard Zingerman <eddyz87@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020-scev-32bit-v1-1-838f34f9f0cd@gmail.com
|
||
|
Alexei Starovoitov
|
a2d2747713 |
bpf: Add unwind_goto and unwind_exit insns
Rust Drop, C++ destructors and __attribute__((cleanup)) in C have to run
when a panic or an exception passes through a frame.
v1 [1] did it with a flag in r3 and 'if r3 != 0 goto pad' after the call.
Three bugs that the bots found there had the same cause: r3 is an ordinary
register, and JITs guessed the meaning from the shape of the code.
Kumar suggested new insns instead [2]:
unwind_goto pad BPF_JMP | BPF_JCOND, src_reg = BPF_UNWIND_GOTO
unwind_exit BPF_JMP | BPF_EXIT, src_reg = BPF_UNWIND_EXIT
'call foo; unwind_goto pad' goes to the pad when foo left with unwind_exit
and falls through when it left with exit. No register says which one it
was. In everything else unwind_exit is exit.
llvm lowers unwinding to them [3]:
ret exit
invoke foo unwind pad call foo; unwind_goto pad
resume r0 = 0; unwind_exit
call _Unwind_RaiseException r0 = 0; unwind_exit
invoke _Unwind_RaiseException goto pad
In a function that returns 16 bytes it's 'r0 = 0; r2 = 0; unwind_exit'.
Landing pads are ordinary blocks, so the verifier checks them like any
other code. A pad that forgets to drop a lock or a reference is a leak on
that path. There are no tables, no stack walk, no kfuncs, no BTF and
nothing to do for libbpf and other loaders.
In the verifier:
- unwind_goto has to follow a call or callx of a subprog and cannot be
a jump target. For cfg, liveness and the rest it's one more conditional
jump, like may_goto.
- How the callee left is one bit of the verifier state. It lives from
the exit of the callee to the next insn. states_equal() looks at it,
since the registers can be the same either way.
- unwind_exit is rejected when unwind_goto doesn't follow the call.
- A global subprog that has unwind_exit can leave either way. Both are
explored after the call, like it's done for might_throw.
- A sync callback cannot unwind. The helper that calls it doesn't expect
that: bpf_loop() would go on.
- unwind_exit of the main prog is exit, unless it's an extension. So it is
in async and exception callbacks. The kernel calls them, and there is no
frame to unwind to.
- An extension leaves either way, like the subprog that it replaced.
freplace is a jump, so the extension returns to the caller of
the subprog. An extension that has unwind_exit can replace only a subprog
that has it too. Otherwise there is no unwind_goto after the calls.
- Of tracing progs only fentry can attach to a subprog or an extension
that has unwind_exit. It would return to the trampoline that called it.
- unwind_exit changes its return address, so the cpu mispredicts
the return and runs what follows unwind_goto speculatively. Without
CAP_PERFMON the verifier looks at that path too.
- unwind_goto goes away together with a pad that turned out to be dead
code. It stays when what follows it is dead code. The cpu predicts that
the callee returns there, so 'goto pc-1' takes the place of the dead
code. Otherwise its removal would move other code there.
JITs don't emit a branch:
unwind_goto:
extable[n++] = { ret_addr - 1 -> pad }
unwind_exit:
my_ret_addr = bpf_unwind_target(my_ret_addr)
epilogue
So exit and the normal path after the call cost nothing. Every frame fixes
only its own return address. There is no stack walk. The lookup is what
the page fault handler does for bpf progs.
The interpreter cannot do that, so the prog has to be JITed, and JITs opt
in with bpf_jit_supports_unwind(). x86-64 and arm64 are in the next
patches.
BPF_JCOND is the same opcode as BPF_CALL_ARGS of the interpreter, and
BPF_UNWIND_GOTO is the same number as BPF_PSEUDO_CALL. may_goto is gone by
the time a prog is dumped, unwind_goto is not. bpf_insn_prepare_dump()
tells them apart: a prog with unwind_goto is always JITed.
Compared to v1 there is no BTF decl_tag and no 'r3 = 0' before every exit.
Tail call and ld_abs are fine in a subprog that can unwind, since they
leave it like exit does.
unwind_exit doesn't set r0. An implicit definition of r0 would need
special cases in backtracking, in liveness and in the check of the return
value of the main prog. A plain 'r0 = 0' in front of it needs none.
Rust
rust-bpf [4] builds with that llvm as it is. The programs are built with
-C panic=unwind. The panic handler prints the message and calls
_Unwind_RaiseException().
exc_cleanup.rs panics three frames deep, with preemption disabled in one
frame and rcu_read_lock held in another. All Drop impls run and the prog
returns 0.
scx_simple.rs is 2195 insns with 48 unwind_goto, 20 of them after callx,
and 16 unwind_exit. With r3 it was 2216 insns. scx_cosmos.rs is 3694 insns
with 68 and 18. It was 3716.
C++
Exceptions work too. This compiles, loads and runs:
struct rcu_guard {
rcu_guard() { bpf_rcu_read_lock(); }
~rcu_guard() { bpf_rcu_read_unlock(); }
};
static __noinline int inner(int len)
{
rcu_guard guard;
if (len > 100)
throw 1;
return len;
}
SEC("tc") int prog(struct __sk_buff *skb)
{
try {
return inner(skb->len);
} catch (...) {
return 42;
}
}
It returns 64 for a packet of 64 bytes and 42 for a packet of 200 bytes.
The destructor runs in both cases, otherwise the verifier would reject
the prog. 'throw' is a jump to the pad with the destructor. 'catch' is
unwind_goto:
inner:
w6 = w1
call bpf_rcu_read_lock
if w6 s< 101 goto ret
r1 = exc_obj ll
*(u32 *)(r1 + 0) = 1
goto pad
ret:
call bpf_rcu_read_unlock
w0 = w6
exit
pad:
call bpf_rcu_read_unlock
r0 = 0
unwind_exit
prog:
w6 = *(u32 *)(r1 + 0)
w1 = w6
call inner
unwind_goto catch
out:
w0 = w6
exit
catch:
w6 = 42
goto out
It needs -fexceptions -funwind-tables and four one line __cxa_*()
functions, where __cxa_throw() calls _Unwind_RaiseException().
Every pad is entered whatever was thrown, so llvm takes catch (...) only.
It's a selftest in patch 5.
[1] https://lore.kernel.org/bpf/20261018010246.1643877-1-alexei.starovoitov@gmail.com/
[2] https://lore.kernel.org/bpf/DM7KR7J8VY2T.1CZIRUF6QRCWG@gmail.com/
[3] https://github.com/llvm/llvm-project/pull/233649
[4] https://github.com/4ast/rust-bpf
Suggested-by: Kumar Kartikeya Dwivedi <memxor@gmail.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Link: https://patch.msgid.link/20261020181320.1284798-2-alexei.starovoitov@gmail.com
|
||
|
Linus Torvalds
|
00b04f4495 |
Merge tag 'vfs-7.4-rc1.idmap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull vfs idmap constification from Christian Brauner:
"This makes struct mnt_idmap const throughout the tree.
A mount's idmapping is immutable. The only thing modified after
alloc_mnt_idmap() is the reference count and that is hidden behind
mnt_idmap_get() and mnt_idmap_put(). Everything else only reads from
it. That is the same model struct cred uses and the idmapping is
sensitive enough to deserve it.
The conversion goes bottom up through the vfs{g,u}id and fs{g,u}id
helpers, the privilege checks, acl, xattr and quota and then every
inode operation that takes an idmap, so ->permission(), ->getattr(),
->setattr(), ->create(), ->mkdir(), ->mknod(), ->symlink(),
->rename(), ->tmpfile(), ->get_acl(), ->set_acl() and ->fileattr_set()
with their implementations in every filesystem, plus the LSM hooks.
struct vfsmount, struct mount_kattr and struct kstatmount store a
const pointer and nop_mnt_idmap and invalid_mnt_idmap live in rodata,
so anything that tries to modify them faults instead of silently
corrupting the initial idmapping"
* tag 'vfs-7.4-rc1.idmap' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (27 commits)
fs: make nop_mnt_idmap and invalid_mnt_idmap const
fs: port mnt_idmap() and file_mnt_idmap() to const mnt_idmap
fs: port vfs_*() helpers to const mnt_idmap
fs: port ->setattr() to pass const mnt_idmap
fs: port ->getattr() to pass const mnt_idmap
fs: port ->create() to pass const mnt_idmap
fs: port ->symlink() to pass const mnt_idmap
fs: port ->mkdir() to pass const mnt_idmap
fs: port ->rename() to pass const mnt_idmap
fs: port ->mknod() to pass const mnt_idmap
fs: port ->tmpfile() to pass const mnt_idmap
fs: port ->get_acl() to pass const mnt_idmap
fs: port ->set_acl() to pass const mnt_idmap
fs: port ->fileattr_set() to pass const mnt_idmap
fs: port xattr to const mnt_idmap
fs: port ->permission() to pass const mnt_idmap
fs: port acl to const mnt_idmap
fs: port inode_init_owner() to const mnt_idmap
fs: port inode_owner_or_capable() to const mnt_idmap
fs: port privilege checking helpers to const mnt_idmap
...
|
||
|
Linus Torvalds
|
a1d4585be1 |
Merge tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull file table updates from Christian Brauner: "This annotates struct fdtable's fd array with __counted_by_ptr so KASAN and the fortified functions can bounds check it against max_fds and adds a KUnit suite for fdtable. A follow-up uses __struct_size() in the test since sparse doesn't know __builtin_dynamic_object_size(). FD_PREPARE() is simplified. It was built as an ACQUIRE-style guard on infrastructure that was never needed. The ACQUIRE machinery and CLASS_INIT() go away, the variable is declared with __cleanup() directly, struct fd_prepare loses its err field since the fd either holds the descriptor or the error, and callers get a const pointer to the guard behind a UNIQUE_ID() name so they can't shadow it. Code generation is identical" * tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: file: make struct fd_prepare const and kill its err field cleanup: remove CLASS_INIT() file: declare the FD_PREPARE() variable with __cleanup() directly file: simplify FD_PREPARE() vfs: use __struct_size() in the fdtable KUnit test vfs: Add KUnit tests for fdtable vfs: Annotate struct fdtable's fd field with __counted_by_ptr |
||
|
Nicholas Velten
|
f25f6319e2 |
bpf: Annotate data races on bpf_lru_node type
KCSAN reports a data race on bpf_lru_node::type between the LRU list
rotation and the free path of the common LRU:
BUG: KCSAN: data-race in __bpf_lru_list_rotate / bpf_lru_push_free
write to 0xffff88811b758022 of 1 bytes by task 4756 on cpu 1:
__bpf_lru_node_move
__bpf_lru_list_rotate_active
__bpf_lru_list_rotate
bpf_lru_list_pop_free_to_local
bpf_common_lru_pop_free
bpf_lru_pop_free
htab_lru_map_update_elem
read to 0xffff88811b758022 of 1 bytes by task 4752 on cpu 0:
bpf_lru_list_push_free
bpf_common_lru_push_free
bpf_lru_push_free
htab_lru_push_free
htab_lru_map_update_elem
value changed: 0x00 -> 0x01
bpf_lru_list_push_free() reads node->type for a WARN_ON_ONCE() sanity
check before taking l->lock. __bpf_lru_node_move(), called right after
under l->lock, does the same check, so drop the unlocked one.
bpf_common_lru_push_free() re-checks node->type under loc_l->lock after
the lockless READ_ONCE() at function entry. By then the node has been
removed from the htab, so only flush and rotate can change its type,
through __bpf_lru_node_move_in() and __bpf_lru_node_move() under
l->lock. Mark that re-check with READ_ONCE() and those two writers with
WRITE_ONCE(), following commit
|
||
|
Emil Tsalapatis
|
7244a2e58d |
bpf: Fix arena race between page free and alloc leading to incoherency
Existing arena kfunc code has an underlying race condition
that can lead to writes being lost from the BPF program's
point of view:
a) A memory range gets freed by operation (1), and its range
is added back to the arena range tree.
b) A concurrent allocation (2) reallocates the range, and does
writes to it. Writes from that CPU may follow the stale TLB
entries into the pages that are about to be freed.
c) (1) invalidates the TLB. The old pages, and any writes done
to them, are now inaccessible. zap_pages() similarly removes the
mappings for userspace threads.
This can be triggered by particularly demanding BPF arena data
structures that constantly allocate and deallocate memory, like
hash table allocations.
Solve this ABA problem by keeping the range allocated until TLB
invalidation and unmapping are complete. Under the arena spinlock,
claim any free pages in the range and clear its kernel PTEs. Then drop
the spinlock, lock and unlock each cleared page to wait for faults that
are still installing a user PTE, flush the kernel TLB and zap the user
page tables. Finally, retake the spinlock to release the range. Frees
serialize on zap_mutex, so an overlapping free cannot release the range
early. Claiming the range only allocates a range tree node when the
whole range is free, and then there is nothing to free.
User faults on a range that is allocated without a page now raise
SIGBUS instead of populating it. That covers a range reserved with
bpf_arena_reserve_pages() and a range whose kernel PTEs a free has
already cleared. Touching a page while a BPF program frees it is a user
space bug, and in a BPF_F_SEGV_ON_FAULT arena it already raises SIGSEGV.
Sleepable frees run synchronously and are deferred only when the arena
spinlock cannot be taken. The deferred free worker retries such a free
in place instead of requeueing itself, so the flush_work() in
arena_map_free() waits for all of its work.
Fixes:
|
||
|
Emil Tsalapatis
|
05bee0638b |
bpf: Revalidate faulted arena pages before installing user mappings
A user fault on an arena pins the page that the kernel arena maps,
drops the arena spinlock and returns the page to the core fault code,
which then locks the folio and installs the user PTE. A concurrent
free can clear the kernel mapping in that window. The page reference
prevents a use-after-free, but the fault can still install a user PTE
for a page the arena has already released. User space and BPF programs
then see different memory at the same arena offset.
Close the window from the fault side. Lock the page, recheck that the
kernel mapping still points to it, and return it with VM_FAULT_LOCKED so
that it stays locked until the core fault code has installed the user
PTE. If a free removed the page, drop it and return VM_FAULT_NOPAGE so
that the fault starts over. Return VM_FAULT_HWPOISON for a poisoned
page.
This change alone does not close the race, because the free path does
not yet wait for the page lock. The following change makes frees wait
for in-flight faults before zapping user mappings.
Fixes:
|
||
|
Kumar Kartikeya Dwivedi
|
19a17b5b2b |
bpf: Refuse CO-RE relocations for a signed program
The signature of a program covers its insns as submitted, along with the
frozen metadata maps of a signed loader, and bpf_prog_verify_signature()
checks it before the verifier does anything else with the program. CO-RE
relocations are the one input besides the insns that changes them: the
loader hands them in with the program, and bpf_check_core_relo() applies
them after the signature check, rewriting the immediate or the offset of
every insn a record names with a value the loader chooses, a field
offset, a type size or id or an enum value of the kernel BTF, or a type
id of the BTF the loader supplies itself. Whoever holds a signed program
can so change its constants once the signature has passed, the offsets a
signed loader reads its metadata map with for instance, and run a
program the signer never saw. The exclusive metadata map of a signed
loader does not make up for it: the hash that binds the map to a program
is chosen by whoever creates the map, so it can name the relocated
program as well.
Refuse a signed load that carries relocations, before the signature is
examined, like the other checks on the shape of a signed load. A signed
loader has none to lose: the programs it embeds are relocated by the
loader itself through bpf_core_apply(), and nothing relocates the loader.
The BTF of the program itself can stay: without relocations it steers
only the verification, through func_info, line_info and the annotations
of global function arguments, and rewrites nothing.
Fixes:
|
||
|
Sasha Levin
|
ced6a0993e |
bpf: Don't reference arena kfuncs when arena isn't built
kernel/bpf/Makefile builds arena.o only for CONFIG_MMU && CONFIG_64BIT, but special_kfunc_list references bpf_arena_alloc_pages() and bpf_arena_free_pages() unconditionally, so on 32-bit and !MMU kernels resolve_btfids can't resolve them: WARN: resolve_btfids: unresolved symbol bpf_arena_free_pages WARN: resolve_btfids: unresolved symbol bpf_arena_alloc_pages Since commit |
||
|
Sasha Levin
|
63be110a96 |
bpf: Drop unused entries from special_kfunc_list again
Commit |
||
|
Kumar Kartikeya Dwivedi
|
a8471eb074 |
bpf: Set up the arena base only in functions that use it
A JIT sets up the arena base in the prologue of every function of a program that has an arena, a callee-saved register loaded with a 64-bit immediate and spilled around the function on x86. That is paid for by every program with an arena, whether it touches the arena or not, and now that libbpf binds the arena of an object to every program in it, by every program of an object that happens to contain an arena. Decide per function, on its final insns, right before it is JITed. bpf_prog_has_arena_insns() looks for an insn that works with the base: an access through an arena pointer, or through a number in a program loaded with BPF_F_ARENA_SCALAR, has been rewritten to a BPF_PROBE_MEM32 form by then, a cast to the user address space keeps BPF_ADDR_SPACE_CAST, and a kfunc call with an __arena argument, which the JIT rebases, has a function model that says so. A load of an __arena struct member is rewritten with an immediate and needs none; dereferencing the result does. Looking at the final insns rather than recording the uses during verification keeps the answer in step with what the JIT sees. A constant NULL passed to a nullable __arena kfunc argument takes the early exit for NULL arguments in the verifier, and the JIT rebases the call all the same; an insn that is dead code has disappeared by then. bpf_prog_jit_compile() caches the answer in bpf_prog_aux for a single function program, jit_subprogs() for each subprogram once it has its kfunc table and arena. It implies that the program has an arena, as the scan says no without one. The JITs with arena support pick up the arena base and user address only when bpf_prog_uses_arena_insns() says so, and so generate the prologue and epilogue of a program bound to an arena it never touches as if it had none. The other users of the arena address of a program stay as they are on purpose. The struct_ops trampoline converts __arena arguments on entry and arena_member_fixup() converts loaded __arena members after the load, both with the base as an immediate. Both have to happen whether or not the program dereferences the result, it may compare, store or pass it on, so neither depends on the base register nor on the flag. The arena fault report reads the user address of the arena the faulting insn belongs to, and such an insn sets the flag anyway. check_map_prog_compatibility() checks the user address when the arena is bound, before any use is known. The rest, in arena.c and sched_ext, works with the map and not with a program. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-14-memxor@gmail.com |
||
|
Kumar Kartikeya Dwivedi
|
7efe1d4e11 |
bpf: Turn loads of __arena pointer members into arena pointers
A kernel struct shared with BPF programs may want to point at arena memory, e.g. a sched_ext argument struct that the kernel builds in the scheduler's arena and that refers to another arena object. A pointer member reaches the program typed as a kernel pointer, so sched_ext publishes a plain u64 holding a pre-translated arena address that the program casts before use. Translate in the verifier instead. When a load through a kernel struct pointer lands on a member tagged __arena, btf_struct_walk() reports it and the destination register becomes PTR_TO_ARENA. The member holds the kernel address of the arena memory; the fixup pass appends the conversion to the arena pointer form that the struct_ops trampoline applies to __arena arguments, after a plain load or a load-acquire of the member: r1 = *(u64 *)(r2 + off) if r1 == 0 goto +1 // __nullable members and probed loads w1 -= lo32(kern_vm_start) A probed load, through a plain or an untrusted pointer, keeps the NULL test as well: it reads as 0 when it faults, and the subtraction must not turn that into an arena pointer. An acquire load only comes through a pointer that cannot fault. Stores to kernel structs are rejected as before, and with them the atomics that write such a member. A member inside a multi-member union is rejected too: an offset into a union resolves to its first member, so a program reading another member there would receive an arena pointer it did not ask for. Loads through program-allocated objects and program BTF are unaffected, there the tag is what the compiler emits for the arena address space. The rewrite is per insn while verification is per path, so the aux data records what an insn loaded on any path: an insn that loaded an arena pointer member on one path and a different member on another is rejected by the fixup pass, while paths disagreeing only on __nullable keep the NULL test, which a never-NULL member can afford. As for __arena arguments, the program needs an associated arena, which also supplies the base to subtract. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-8-memxor@gmail.com |
||
|
Kumar Kartikeya Dwivedi
|
bc7c282468 |
bpf: Recognize __nullable on pointer members of kernel structs
A pointer member of a kernel struct that the verifier derives as trusted or RCU-protected is assumed to be non-NULL, so a load through it is a plain dereference. The BTF_TYPE_SAFE_*_OR_NULL lists exist for members known to be NULL at times, but they live in the verifier and have to be extended by hand. Teach btf_struct_walk() to report the "nullable" BTF type tag on a pointer member as PTR_MAYBE_NULL. The tag combines with any other tag on the member; it only says that the pointer may be NULL. The verifier keeps the flag where NULL would otherwise be assumed away: on trusted and RCU results, which a program dereferences as they are, and on per-CPU results, which the per-CPU helpers shift without a check. Such a pointer has to be tested before it is dereferenced or passed on. Plain and untrusted results are probed anyway and lose the flag, as the BTF_TYPE_SAFE_*_OR_NULL lists never applied to them either, so a tagged member of an existing struct asks no more of the programs that read it than a list entry does, and the tag can take the place of the lists. The next patch uses the tag on __arena members to decide whether the injected address translation has to preserve NULL. Signed-off-by: Kumar Kartikeya Dwivedi <memxor@gmail.com> Signed-off-by: Alexei Starovoitov <ast@kernel.org> Link: https://patch.msgid.link/20261019132203.3034698-7-memxor@gmail.com |
||
|
Kumar Kartikeya Dwivedi
|
55f17779d6 |
bpf: Check exclusive maps once the program digest is final
An exclusive map is bound to the digest of its program, which the verifier compared against prog->digest in check_map_prog_compatibility() as the map was added to the program. For a map an insn refers to that happens in check_and_resolve_insns(), after the digest is computed at the top of that function. For a map bound through a continuous fd_array it happens in process_fd_array(), which runs earlier, so for an unsigned program the comparison saw a zeroed digest and the load failed with -EACCES, "program's hash doesn't match map's excl_prog_hash", for the very program the map is exclusive to. Computing the digest earlier would not do: it has to cover the insns as the kernel verifies and runs them, after the CO-RE relocations it applies, or a loader could submit the insns of the exclusive program together with relocations that change them and pass the check with a different program. Leave the digest where it is and move the comparison instead: check every used map against the digest at the end of check_and_resolve_insns(), whether an insn referred to it or the array bound it. The claims a program makes on its maps move behind the check as well: bpf_map_claim(), which records the program as the user of the map for resolve_func_ptrs(), and the takeover of an insn array by bpf_insn_array_init(). Both ran as the map was added, so a program that is not the owner of an exclusive map could have flipped the owner's single-user state to shared, which a failed load does not undo, or held the owner's insn array while it was being verified. Now they run once every map has passed the check, and a rejected program leaves the maps as it found them. Nothing consults a claim before resolve_func_ptrs() and the fixup passes, which run later. The early digest of a signed program goes with the early comparison. Commit |
||
|
Alexei Starovoitov
|
4f842631f1 |
bpf: Refuse the prologue for an extension if callers keep packet pointers
An extension that writes to the packet of a tc, lwt_xmit or sk_skb program
starts with bpf_unclone_prologue(). If the skb is cloned, that gives it
a new head.
It does not count as changing packet data. changes_pkt_data goes by the
helpers and kfuncs that a program calls, and whether it gets a prologue
is known only after it has been compared with its target. So the extension
can replace a global function that leaves the packet alone, and the callers
of that function keep their packet pointers across the call. These point
into the old head, which is left to the other clone. Once that is freed,
by the reader of a packet socket for example:
BUG: KASAN: slab-use-after-free in bpf_prog_53bc21b9810cadbc_t_read+0x2e8/0x36e
Read of size 1 at addr ffff888107bfe73c by task tt11/102
Freed by task 112:
kfree+0x161/0x500
skb_release_data+0x2dc/0x340
consume_skb+0xa6/0x1c0
packet_recvmsg+0x4b3/0xa20
Nothing is wrong if the target program writes to the packet too: its own
prologue has uncloned the skb, and that of the extension does nothing.
Nor if the callers hold no packet pointers, like a dispatcher, or if the
main function is replaced.
Note for every global function that leaves the packet alone if it is
called while packet pointers, or slices of the dynptr of a packet, are
held. What the callers of a global function hold is not known while it
is verified, so the functions that it calls are taken for such as well.
Refuse an extension with a prologue for such a function, unless the target
program has a prologue, both when the extension is loaded and when it
is attached to another target.
Functions can share a type id in func_info. The first of them is the one
that is replaced, so look it up in one place.
Fixes:
|
||
|
Yiyang Chen
|
c3a0bfddae |
bpf: Enforce trusted BTF ID argument provenance
A global subprogram parameter tagged __arg_trusted is verified with
PTR_TRUSTED, but the common call-site type check accepts bare PTR_TO_BTF_ID
and MEM_RCU registers. The callee can consequently pass a pointer with
weaker provenance to a kfunc that does not accept RCU-protected pointers.
RCU protection only guarantees that the object remains allocated through a
grace period. It does not guarantee that an object's refcount is nonzero,
so an RCU-protected pointer cannot satisfy the trusted-pointer contract.
For example, an RCU-protected bpf_cpumask loaded from a map may have a
zero refcount. Passing it through __arg_trusted lets the callee pass it
to bpf_cpumask_acquire(), whose plain refcount_inc() can retain an object
that will be freed after the grace period.
Make the existing provenance check contract-driven. Mark generated BTF ID
kfunc argument types with PTR_TRUSTED, and add MEM_RCU only for KF_RCU
kfuncs. Normalize PTR_MAYBE_NULL when the argument contract permits it.
Then use the argument flags to require trusted or referenced provenance
and allow MEM_RCU only when the contract says so.
Split is_trusted_reg() so the normalized type is used for type and modifier
checks while reference lookup still uses the original register. Conversion
of scalar-struct BTF ID arguments to fixed-size memory continues to
construct a new ARG_PTR_TO_MEM type without PTR_TRUSTED.
Fixes:
|
||
|
Masoud Aghasi
|
4a9b25fe8d |
bpf: Fix incorrect handling of user flags by percpu hash map updates
For BPF_MAP_TYPE_PERCPU_HASH and BPF_MAP_TYPE_LRU_PERCPU_HASH maps,
htab_map_check_update_flags() and check_flags() are not considering
the possibility of a combination of (BPF_NOEXIST, BPF_EXIST) flags
with (BPF_F_CPU, BPF_F_ALL_CPUS) flags. This causes the
(BPF_NOEXIST, BPF_EXIST) flags to lose their effect in some cases.
For example, when using (BPF_F_CPU | BPF_EXIST) or
(BPF_F_CPU | BPF_NOEXIST) flag combinations with bpf_map_update_elem()
on a percpu hash map, the BPF_EXIST flag does not prevent new
insertions as expected and BPF_NOEXIST flag does not prevent
modification of existing entries as expected.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, htab_map_check_update_flags() rejected
(BPF_F_ALL_CPUS | BPF_EXIST) and (BPF_F_ALL_CPUS | BPF_NOEXIST) with
-EINVAL. After the patch those combinations are accepted.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes:
|
||
|
Masoud Aghasi
|
a898ff0f3e |
bpf: Fix incorrect handling of user flags in bpf_percpu_array_update
For BPF_MAP_TYPE_PERCPU_ARRAY map, bpf_percpu_array_update()
is not considering the possibility of a combination of
(BPF_NOEXIST, BPF_EXIST) flags with (BPF_F_CPU, BPF_F_ALL_CPUS) flags.
This causes the (BPF_NOEXIST, BPF_EXIST) flags to lose their effect
in some cases.
For example, using the (BPF_F_ALL_CPUS | BPF_EXIST) flag combination
with bpf_map_update_elem() results in an incorrect EINVAL error
response, even though the flag combination is valid.
This patch fixes the bug by adding proper flag validations and checks.
Before this patch, bpf_percpu_array_update() rejected
BPF_F_ALL_CPUS | BPF_EXIST and BPF_F_ALL_CPUS | BPF_NOEXIST with
-EINVAL. Also the BPF_F_CPU | BPF_NOEXIST were accepted.
After the patch BPF_F_ALL_CPUS | BPF_EXIST is accepted and
BPF_F_ALL_CPUS | BPF_NOEXIST and BPF_F_CPU | BPF_NOEXIST
result in -EEXIST.
This patch also starts returning -EINVAL for (BPF_EXIST | BPF_NOEXIST),
with or without BPF_F_CPU, which was previously accepted.
Fixes:
|
||
|
KaFai Wan
|
0ceb2f21d3 |
bpf: Determine freplace staticness from BTF linkage
bpf_check_attach_target() rejects a freplace whose target is a static
function, using the subprog "unreliable" flag as a proxy for "static".
That flag is raised whenever a subprogram's BTF prototype disagrees with
the registers at its call site.
`147400e9c27c ("bpf: Compute parameters liveness in
bpf_compute_live_registers()")` made that work for static callees: a
static function with an unused argument no longer matches its BTF
prototype at the call site and is marked unreliable.
Since LLVM commit `29b36a16f111 ("[BPF] Emit ABI-accurate BTF
prototypes for DW_CC_nocall")` in 23.1 [1] that premise is gone:
BTFDebug may now drop arguments removed by the optimizer, so for
__noinline static int subprog(struct __sk_buff *ctx)
{
return bpf_ktime_get_ns();
}
compiled with clang 23 the BTF describes subprog() as
[75] FUNC_PROTO '(anon)' ret_type_id=2 vlen=0
[76] FUNC 'subprog' type_id=75 linkage=static
There is then no argument left to compare, the subprog stays reliable,
and the check in bpf_check_attach_target() is skipped. The freplace is
then rejected by the later linkage check instead, reporting "subprog()
is not a global function", and the selftest "exceptions" fails.
Determine staticness from the BTF linkage instead, and keep the
unreliable check for global functions whose BTF does not match their
call site.
There is no functional change: freplacing a static function was rejected
before and is still rejected with the same -EINVAL. Selftest "exceptions"
is updated to match.
[1] https://github.com/llvm/llvm-project/pull/198426
Fixes:
|
||
|
Woojin Ji
|
1c1dda9ba9 |
bpf: Zero-extend the index in inlined bpf_get_func_arg()
bpf_get_func_arg() takes a u32 index, but the inlined expansion compared
only the low 32 bits of R2 with the argument count and then shifted the
full 64-bit register to compute the context address. R2 is ARG_ANYTHING,
so nothing guarantees that its upper bits are zero: an index of
1ULL << 32 passes the bounds check as index zero while the load is
performed at ctx + (1ULL << 35).
Shift the index as a 32-bit value. When the 32-bit bounds check passes,
the low 32 bits of R2 are below R0, and R0 is at most 255: the trampoline
arm masks the argument count with 0xFF and the raw_tp arm uses the BTF
argument count minus one, which the verifier caps at MAX_BPF_FUNC_ARGS.
The 3-bit shift therefore cannot overflow and it clears the upper bits of
the register. Widening the comparison instead would reject indices whose
low 32 bits are valid. The check itself is unchanged, so this cannot
change which programs are accepted, only the address that is computed.
Fixes:
|
||
|
Josef Bacik
|
a6f8a67c13 |
bpf: Refuse packet slices for packet-changing helpers
The helpers which may change packet data do not take a packet pointer as a
memory argument: they have no pkt_access. They make the skb writable, or
make room in it, before they read what the argument points to, and that
can reallocate the head which it points into.
A slice from bpf_dynptr_slice() of the dynptr of the skb points into the
same head, but it is a PTR_TO_MEM, and the verifier lets it pass.
bpf_lwt_push_encap(), bpf_lwt_seg6_store_bytes() and bpf_lwt_seg6_action()
then read the old head, which has been freed. bpf_skb_store_bytes() follows
the head by now, but a slice is no better than a packet pointer there.
Refuse a slice of a packet where a packet pointer is refused for that
reason. All other helpers take slices as before. bpf_dynptr_write() and
bpf_dynptr_copy() can be used to copy inside of a packet.
Fixes:
|
||
|
Alexei Starovoitov
|
bb72c28668 |
bpf: Unclone the skb for kfuncs that write through a dynptr
bpf_dynptr_copy(), bpf_dynptr_memset() and the kfuncs that read user or kernel memory into a dynptr ask bpf_dynptr_slice_rdwr() for a slice and write to it. For the linear part of an skb that is a pointer into its head. bpf_dynptr_slice_rdwr() counts on the verifier to have uncloned the skb, see bpf_unclone_prologue(). The verifier does so if the program itself calls bpf_dynptr_slice_rdwr() or writes to the packet. A program that only uses these kfuncs gets no prologue, and changes the data of all the clones of the skb. A packet socket, whose clone is taken before tc ingress runs, sees what bpf_dynptr_memset() has written there. Treat a call that may write to an skb like a direct write. Fixes: |
||
|
Alexei Starovoitov
|
aa486cff49 |
bpf: Refuse the prologue for an extension that cannot take it
A tc, lwt_xmit or sk_skb program that writes to packet data starts with
bpf_unclone_prologue(). It takes the skb from R1, and if the skb is cloned
it calls bpf_skb_pull_data() and restores R1 only.
That suits the main function of a program. An extension gets the prologue
too, but it starts with the arguments of the function that it replaces.
If the first one is not the context, something else is taken for an skb:
BUG: KASAN: stack-out-of-bounds in skb_ensure_writable+0x84/0x1a0
Read of size 8 at addr ffffc90000270ad0 by task uncl/105
skb_ensure_writable+0x84/0x1a0
bpf_skb_pull_data+0x32/0xf0
bpf_prog_853eb1991d35e093_ext_slice+0x4f/0x321
bpf_prog_9eb18752b5d0959a_t_glob+0x2ac/0x2de
BUG: unable to handle page fault for address: 0000000082298821
And whatever R2-R5 held is lost, while the verifier goes on trusting
pointers in them.
Refuse an extension that needs a prologue unless it gets the context as
its first argument and scalars beside it. A scalar is still clobbered if
the skb is cloned, which the verifier can cope with. Extensions of XDP,
sk_msg and sockopt programs get no prologue and load as before.
Fixes:
|
||
|
Josef Bacik
|
b7f97db889 |
bpf: Don't take a dynptr argument for a local dynptr
A global function is verified on its own. A dynptr which it gets as an
argument is assumed to be of type BPF_DYNPTR_TYPE_LOCAL, but the callers
may pass one of any type, that of an skb for example.
The type decides whether a write through the dynptr invalidates packet
pointers, and whether a slice of it is invalidated together with them.
Inside of the function neither happens. It can take a packet pointer from
its context argument, or a slice from the dynptr, write through the dynptr
and go on using the pointer, although the write may have reallocated the
head of the skb.
Assume the type of an skb dynptr instead. Nothing becomes forbidden for the
dynptr itself, and a clone inherits the caution. The dynptr which the
callback of bpf_user_ringbuf_drain() gets is known to be local, and stays
so.
Fixes:
|
||
|
Josef Bacik
|
589998616e |
bpf: Let bpf_dynptr_write() mark a global function
The caller of a global function loses its packet pointers if the function
may change packet data. That is found out by a scan for calls of the
helpers which bpf_helper_changes_pkt_data() lists.
bpf_dynptr_write() is not on the list, because it depends on the type of
the dynptr: check_helper_call() looks at it. The scan cannot do that, and
so a global function which makes a dynptr from the skb, or gets one as an
argument, and writes through it counts as harmless. The write can
reallocate the head of the skb, and the caller goes on with pointers into
the old one, which has been freed.
Let every call of bpf_dynptr_write() mark the function, as a tail call
does. The same mark decides whether a program may replace a function.
Fixes:
|
||
|
Josef Bacik
|
19e6ca6b4a |
bpf: Drop packet pointers behind kfuncs writing an skb
bpf_dynptr_write() to the dynptr of an skb ends in
__bpf_skb_store_bytes(), which makes the range writable first. That pulls
data into the head or unclones it, and either can reallocate the head. So
check_helper_call() invalidates all packet pointers and all slices of the
skb behind the helper.
Several kfuncs take the same road: bpf_dynptr_copy() and
bpf_dynptr_memset(), if they get no writable slice of the destination
because it is not linear, and bpf_probe_read_{user,kernel}[_str]_dynptr()
and bpf_copy_from_user[_task][_str]_dynptr() likewise. The verifier does
not know. A program can keep a packet pointer or a slice across such a
call and read and write the old head, which has been freed.
Treat a kfunc of that kind as bpf_dynptr_write() is treated, by the type
of the dynptr it writes to, which is the first argument of all of them.
Where a global function is only scanned for what it may do, the type is
not known yet. There any call of these kfuncs counts, as a tail call does.
Fixes:
|
||
|
Josef Bacik
|
f993512729 |
bpf, cpumap: Hold the device of frames in the queue
A frame that is redirected to another CPU waits in the queue of the cpumap entry with a plain pointer to the device it came from, and an skb from generic XDP with skb->dev. Nothing keeps that device. Closing and unregistering it waits for its NAPI and for RCU, but not for the kthread of the entry. If the kthread is late by more than that, since its CPU is busy with something of a higher priority, it builds skbs for a device that has been freed and hands them to the stack: BUG: KASAN: use-after-free in dev_gro_receive+0x1f51/0x2b30 Read of size 8 at addr ffff88800a7340b0 by task cpumap/1/map:1/54 Take a reference to the device for each entry of the queue. The kthread cannot drop it at the end of the batch, since its GRO node may keep the skb for longer. Let it note the devices and drop the references behind a full GRO flush, which it does whenever the queue is empty, and from now on also when it has noted about 2048 of them. A device whose frames are stuck in a queue is now unregistered only when the kthread has got to them. RTNL is not held meanwhile, but if the device goes with its network namespace, the cleanup of all namespaces waits. Fixes: |
||
|
Josef Bacik
|
4eaf4d71b9 |
bpf: Keep the symbols of a program until it has ended
When the last reference to a program goes, __bpf_prog_put_noref() takes its symbols out of the kallsyms tree and frees its line info at once, and waits for the grace periods only with the rest. But the program may still be running: the link or the array it was called from is gone, the call is not over. search_bpf_extables() finds the exception table of a program by its symbol. A load that faults, and that the table is there to fix up, is fatal from then on: Oops: general protection fault, probably for non-canonical address 0xdead0000000006f8: 0000 [#1] SMP NOPTI RIP: 0010:0xffffffffc06008f6 It takes a program that is detached and closed on one CPU while it runs on another. Unwinders and bpf_prog_find_from_stack() do not know the program any more either, and the stack dump of a BPF stream reads the freed line info. Free the line info behind the grace periods the program needs, as it is done with the BTF that its text comes from. Remove the symbols in bpf_prog_free_deferred(), which runs behind them as well. The symbols are looked up under rcu_read_lock(), so the images and the program itself are freed one more grace period later. All that the program refers to is released when it was before. Fixes: |
||
|
Alexei Starovoitov
|
b6458abbaf |
bpf: Don't take an extension for a function that runs sleepable
An extension cannot be sleepable, and it is verified as a program that
runs under the RCU read lock. But it runs in the context of the function
that it replaces. For a sleepable program, or a function that is called
from a bpf_wq or bpf_task_work callback, that is rcu_read_lock_trace().
The verifier lets such an extension use maps and pointers that rely on
RCU. Its maps are not counted as maps with sleepable users either, so
that what it got from them is freed behind a plain RCU grace period:
WARNING: net/core/sock_map.c:903 at __sock_hash_lookup_elem+0xe8/0xf0
BUG: KASAN: slab-use-after-free in bpf_prog_b3bad285227d855b_ext_storage+0x245/0x297
Write of size 8 at addr ffff888108717a50 by task fr/955
The verifier knows the contexts that a global function is called in.
Refuse to replace one that is called without the RCU read lock, and the
main function of a sleepable program. A function that sleepable code only
calls under bpf_rcu_read_lock() can still be replaced.
The summarization test has replaced functions of sleepable programs by
ones that do not sleep. Expect that to fail.
Fixes:
|
||
|
Josef Bacik
|
f693030a18 |
bpf: Keep values of a dead owner for sleepable programs
bpf_local_storage_destroy() frees the elements of an owner behind a plain
RCU grace period, on the ground that no program can refer to an owner that
goes away. But the pointer that bpf_task_storage_get() and its like
return stays valid for the verifier after the program has released the
owner, or has left the bpf_rcu_read_lock() section it had the owner from.
A sleepable program is in no RCU read side critical section, so it can
write to the value after the owner has died and the element was freed:
BUG: KASAN: slab-use-after-free in _copy_from_user+0x42/0xa0
Write of size 8 at addr ffff8880074404a8 by task lsuaf/55
Free the element behind an RCU Tasks Trace grace period if sleepable code
may use the map. A program that a map is bound to was loaded, and counted
in map->sleepable_refcnt, before it got a value from the live owner, and
stays counted until a grace period behind its last run.
A program can also get at a storage map by a lookup in an outer map. Then
only the outer map is counted, its programs can come after the inner map
was put into it, and can hold on to the inner map after it was taken out.
So note in a map that it has been an inner map, and treat it like one with
sleepable users from then on.
The same goes for what the special fields of the value hold. The page of
a uptr has to stay pinned, and a kptr which the program stores after the
owner has died has to be released. Leave the fields, and the smap that it
takes to free them, to bpf_selem_free_trace_rcu(), as bpf_selem_unlink()
does. bpf_local_storage_map_free() waits for that callback, and leaves
smap alone if it comes across such an element.
Elements of such maps now wait for sleepable programs that are running
when their owner dies. Other maps are freed as before.
Fixes:
|
||
|
Alexei Starovoitov
|
fcbfa99aa6 |
bpf: Don't let an extension reach further into the context than its target
How large the context of a raw tracepoint, tracepoint or syscall program
is depends on what the program is attached to, or is run with.
bpf_probe_register(), perf_event_set_bpf_prog() and the test runs compare
that with max_ctx_offset of the program.
An extension of the program, or of one of its global functions, gets the
same context. How far it reaches is noted in its own max_ctx_offset, which
nobody looks at. On a tracepoint with one argument it reads the stack
behind the arguments, in a test run the memory behind the context, or
from NULL if there is none:
BUG: KASAN: stack-out-of-bounds in bpf_prog_954b5d2051843629_ts28_ext_far+0x7e/0x113
Read of size 8 at addr ffffc90000def7b8 by task ts28/104
BUG: KASAN: slab-out-of-bounds in bpf_prog_954b5d2051843629_ts28_ext_far+0x7e/0x113
Read of size 8 at addr ffff88810712c3e8 by task ts28/104
Refuse to attach an extension that reaches further than the program which
it extends. A program that leaves the context to its extensions has to
touch as much of it as they do.
Fixes:
|
||
|
Alexei Starovoitov
|
b83bacf216 |
bpf: Don't take a program with a target program for an iterator
bpf_iter_prog_supported() goes by the name of the attach function. Commit |
||
|
Josef Bacik
|
5bdd61e98d |
bpf: Free what map_free() unlinks for sleepable readers
bpf_local_storage_map_free() takes the elements of the map off the lists
of their owners, which are alive, and frees them, and the storage of an
owner with its last element, behind a plain RCU grace period. Nobody uses
the map any more, but the other maps of these owners are in use. A
sleepable program calls the get and delete helpers under
rcu_read_lock_trace() only. While it walks the list of an owner for its
own map, or is about to lock the storage, the element it stands on, or the
storage, can be freed under it.
Whatever else takes an element off the list of a live owner frees it
behind an RCU Tasks Trace grace period, which implies a plain one. Do so
in bpf_local_storage_map_free() as well. It waits for these callbacks
before it frees the map.
This does not help a sleepable program that hands an owner to the helpers
which nobody holds, and which may be gone by then.
Fixes:
|
||
|
Josef Bacik
|
8d0358f6e8 |
bpf: Keep iterator programs off special map fields
The bpf_map_elem iterator, for all but the per-cpu maps, and the bpf_sk_storage_map iterator hand the live value of an element to the program as a writable buffer. The verifier does not know the map when it checks the program, so nothing keeps the program from writing over a bpf_spin_lock, a bpf_timer, a kptr or another special field, which neither a map update nor a pointer to a map value can touch. With CAP_BPF and CAP_PERFMON, or a BPF token, a program can so make the kernel follow a pointer of its choice: after a write over a bpf_timer the kernel crashes on the written value when the element is removed. Offsets into a buffer are constants. Let the verifier record the range of bytes that a program may write through a read-write buffer, and refuse the program with -EACCES, at attach and at a link update, if a special field of the map lies in that range. A program that only reads, or that writes beside the special fields, works as before. One that writes on both sides of a special field is refused, since the range covers the field. Fixes: |
||
|
Alexei Starovoitov
|
7fe348f18b |
bpf: Don't take an extension for a writable raw tracepoint program
bpf_probe_register() compares max_tp_access of the program that is
attached with the size of the buffer that the tracepoint lets programs
write to. An extension of that program, or of one of its global
functions, gets the same context and the same pointer to the buffer. What
it accesses is noted in its own max_tp_access, which nobody looks at. So
it reads and writes behind the buffer, of a tracepoint that may have none:
BUG: KASAN: stack-out-of-bounds in bpf_prog_5843f2b820eb4929_tr26_ext_far+0xf4/0x10c
Write of size 4 at addr ffffc90000d8f894 by task wext/103
Refuse such extensions.
Fixes:
|
||
|
Josef Bacik
|
cd0490e671 |
bpf: Assume no bounds for a writable tp with tail calls
A writable raw tracepoint hands a buffer to the program. The verifier
notes how far the program reaches into it, and bpf_probe_register() refuses
a program that reaches further than the tracepoint allows. A program that
does not touch the buffer passes, and can tail-call one of its type that
reads and writes behind it.
Take a BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE program with a tail call to
reach as far as can be, as for the stack, the packet and the buffers of
iterators. It cannot be attached any more. Plain raw tracepoint programs,
for which the limit is zero, cannot get at the buffer, and a tail call
stays within one type: they are left alone.
Fixes:
|
||
|
Josef Bacik
|
7cc639dc45 |
bpf: Don't take an extension for an iterator program
An extension inherits the expected_attach_type of the program it extends.
If that is BPF_TRACE_ITER, check_attach_btf_id() hands the extension to
bpf_iter_prog_supported(), which goes by the name of the attach function.
For an extension that is the name of the subprogram it replaces, so the
load fails with -EINVAL, unless that subprogram is called like the function
of a target, bpf_iter_bpf_map_elem for example.
Then the extension gets the context of that target. If the subprogram
has as many arguments, it can use the key and the value as buffers, but
how far it reaches is noted in the extension, where the test of the
iterator link does not look: it reads and writes behind the elements of
the map. And the id of the subprogram in the BTF of the program is cached
as the id of the target, so that programs of that target which are loaded
already cannot be attached.
Refuse all extensions here.
Fixes:
|
||
|
Josef Bacik
|
7fa323d776 |
bpf: Assume no bounds for buffers behind a tail call
The bpf_map_elem, bpf_sk_storage_map and sockmap iterators hand the key and the value of an element to the program as buffers. The verifier notes how far a program reaches into them, and the iterator compares that with the map when the program is attached. A program that does not touch the buffers passes. It can tail-call another program of the same target, which gets the same context, and whose reach nobody compares with the map: BUG: KASAN: slab-out-of-bounds in ___bpf_prog_run+0xa46a/0xae80 The program array can be filled at any time, so there is nothing to look at when the link is made. For the same reason a program with a tail call is taken to use the whole stack and the whole packet. Take it to reach as far as can be into the buffers as well, so that these iterators refuse it with -EACCES. Fixes: |
||
|
Josef Bacik
|
3e6d7522af |
bpf: Check the maps that sleepable callbacks use
Sleepable programs can only use the kinds of map that do without an RCU read side critical section. check_map_prog_compatibility() sees to that when a map is bound to the program, and asks whether the program is sleepable. That misses the bpf_wq or bpf_task_work callback of a program that is not, which runs sleepable all the same, and it misses an inner map, which is not bound. The bpf_wq callback of a tc program that looks a key up in a sockhash gets as far as WARNING: net/core/sock_map.c:903 at __sock_hash_lookup_elem+0x18f/0x1e0 Ask the same question for the map that is handed to a helper in sleepable context, unless the RCU read lock is held: by bpf_rcu_read_lock(), or by the helper that runs the callback, as bpf_for_each_map_elem() does for a hash map. Programs that take the lock go on working. Disabled preemption or interrupts are not enough, since the maps ask for rcu_read_lock_held(). The other parts of the program may use such a map as before. A global function that is called with preemption or interrupts disabled is verified for the non-sleepable context, in which nothing is known of all that. Verify it for a context of its own: not sleepable, and without the RCU read lock. It differs in what maps can be used, and in nothing else. What is refused from now on runs into the warning above, or uses what the map frees behind a plain RCU grace period: an entry of a devmap, of a perf event array or of a cgroup array. A bloom filter does without RCU. Add it to the list, so that callbacks which use one go on loading. As a side effect sleepable programs can use bloom filters as well. A pointer to the value of a devmap which a program keeps beyond bpf_rcu_read_unlock() is not dealt with here. Fixes: |
||
|
Josef Bacik
|
927c9c985f |
bpf: Count sleepable callbacks as sleepable map users
map->sleepable_refcnt tells bpf_map_fd_put_ptr() whether an inner map that
is taken out of its outer map has to wait for an RCU Tasks Trace grace
period as well. It counts the sleepable programs that use the outer map.
The bpf_wq or bpf_task_work callback of a program that is not sleepable
runs under rcu_read_lock_trace() only, and is not counted.
Such a callback can look an inner map up and keep it while user space
deletes it from the outer map. The inner map is freed behind a plain RCU
grace period, under the callback:
BUG: KASAN: slab-use-after-free in array_map_update_elem+0x6bc/0x7d0
Count a program with such a callback. The maps are bound before the
verifier meets the callback, so raise the count of those that are bound
when it does.
Fixes:
|
||
|
Josef Bacik
|
1e28a5e52d |
bpf: Keep a program until its bpf_wq callback has ended
bpf_wq_work() runs the callback under rcu_read_lock_trace() only, since
the callback may sleep. The program it belongs to need not be sleepable,
and such a program is freed behind a plain RCU grace period. The
reference that the bpf_wq holds is dropped when the map element goes, or
when another callback is set, no matter whether the work is running.
If that was the last reference the program is freed under the running
callback, and the worker goes on in the freed JIT image:
Oops: int3: 0000 [#1] SMP NOPTI
RIP: 0010:0xffffffffc0600947
All it takes is to close the map and the program while the callback runs.
Note in the program that it has a callback that runs sleepable, and wait
for an RCU Tasks Trace grace period, which implies a plain one, before
such a program is freed. Let bpf_wq_work() enter its read side critical
section before it reads callback_fn.
bpf_task_work holds a reference to the program until the callback is done.
Fixes:
|
||
|
Josef Bacik
|
af929bc938 |
bpf: Don't bind a program to a netdev that goes away
bpf_prog_dev_bound_init() looks the device up, takes bpf_devs_lock and
registers the device for the program. The reference it holds does not
keep the device registered. If bpf_dev_bound_netdev_unregister() has run
by the time the lock is taken, nobody takes the program off the device any
more: the bpf_offload_netdev and offload->netdev point to a net_device
that is freed as soon as the reference is dropped.
bpf_dev_bound_resolve_kfunc() dereferences it while the program is
verified, and a new device at the same address finds the old entry.
unregister_netdevice_many_notify() leaves NETREG_REGISTERED before it
calls bpf_dev_bound_netdev_unregister(), which takes bpf_devs_lock. Look
at reg_state under that lock, and treat a device that goes like one that
is not there.
Fixes:
|
||
|
Josef Bacik
|
8e83d53596 |
bpf: Fix stale offload->prog pointer after realloc
bpf_prog_realloc() moves a program that the rewrites of the verifier make outgrow its pages. It updates aux->prog, but not the offload->prog of a device-bound program. When the device is unregistered, __bpf_offload_dev_netdev_unregister() hands the stale pointer to __bpf_prog_offload_destroy(), which faults on the freed program: BUG: unable to handle page fault for address: ffffc90000071050 RIP: 0010:__bpf_prog_offload_destroy+0x3f/0x310 It takes the right to load a device-bound XDP program, CAP_BPF and CAP_NET_ADMIN or a BPF token, and a device that can be removed. Commit |
||
|
Josef Bacik
|
45e912fb1d |
bpf: Check the new program of an iterator link update
The bpf_map_elem, bpf_sk_storage_map and sockmap iterators hand the key and the value of an element to the program as buffers. The verifier only records how far a program reaches into them, and ->attach_target() compares that with the key and value size of the map. bpf_iter_link_replace() does not. BPF_LINK_UPDATE installs a program that an attach refuses with -EACCES, and the program then reads, and for a value writes, behind every element that the iterator visits. It takes CAP_BPF and CAP_PERFMON, or a BPF token. Split the test out of the three ->attach_target() and ask it, as ->check_prog(), for the new program of an update as well. Targets without such a test take every program of their kind, as before. Fixes: |
||
|
Josef Bacik
|
13ac7c05bc |
bpf: Bound the nesting that the BTF show code follows
btf_struct_show() and btf_array_show() recurse for every nested struct
and array. MAX_RESOLVE_DEPTH bounds one resolve walk when BTF is loaded,
not the nesting: a member whose type is resolved already is not walked
again, so BTF that defines its types from the inside out can nest them as
deep as it has types.
Whoever may load BTF, with CAP_BPF or a BPF token, can give such a type
to the value of a map. Reading the file of that map in bpffs then
overruns the kernel stack: with a few hundred levels the stack guard page
is hit and the kernel panics.
Stop at 32 levels. That is MAX_RESOLVE_DEPTH, and __btf_show_indent()
has as many columns. Nothing is shown for a member that lies deeper, and
the status becomes -E2BIG, which bpf_snprintf_btf() and
bpf_seq_printf_btf() return. The types of an x86_64 defconfig kernel nest
14 deep.
Fixes:
|
||
|
Alexei Starovoitov
|
bd897919e4 |
bpf: Link the first element after its storage is published
bpf_local_storage_alloc() links the first element of a new storage to the
bucket of its map, and then publishes the storage in the owner with
cmpxchg(). If another cpu has been faster, the element is taken out of
the bucket again and is freed together with the storage.
In between the iterator over a sk_storage map can find the element. It
stays linked to a storage that points to the socket, but the socket knows
nothing of that storage: bpf_sk_storage_free() neither unlinks the element
nor waits for anybody who uses the storage. So nothing tells the iterator
that the socket has gone away.
Three cpus that create the first storage of a new AF_UNIX socket at the
same time, and a fourth that reads the iterator, are enough: in 20 seconds
its program was shown some 4000 elements of storages that had lost. If the
socket is closed in time, the program is handed a freed socket.
Link the element once the storage is published. Both happens under the
lock of the bucket, which holds off whoever finds the element through the
owner and wants to unlink it.
Fixes:
|