mirrors/linux
0
0 Fork 0
mirror of https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git synced 2026-10-26 10:57:38 +08:00
Code Activity

Merge tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Browse Source 
Pull bpf updates from Alexei Starovoitov:
 "Major changes:

   - Introduce scalar evolution (SCEV) analysis in the verifier.

     It infers how registers and stack slots change in a loop and lets
     the verifier check bounded loops with induction variables as ranges
     instead of walking every iteration. The number of processed insns
     drops ten fold in programs with loops (Eduard Zingerman)

   - Add 'callx' instruction: indirect call of a static BPF subprogram.

     The address is taken with ld_imm64 or loaded from read-only data.
     Supported by x86-64, arm64 and riscv64 JITs (Alexei Starovoitov, Pu
     Lehui)

   - Add unwind_goto and unwind_exit instructions, so that Rust panic or
     C++ exception can run destructors of every frame it passes through.
     The normal path is free of unwind code. Supported by x86-64 and
     arm64 JITs (Alexei Starovoitov)

   - Support programs compiled by rust-bpf: treat load and store through
     a number as arena access with BPF_F_ARENA_SCALAR, accept rust names
     in BTF (Alexei Starovoitov)

   - Raise BPF program stack size from 512 bytes to 2 KiB on x86-64 and
     arm64 (Kumar Kartikeya Dwivedi)

   - Support struct, union and __int128 up to 16 bytes passed and
     returned by value in BPF subprograms and kfuncs, and fix x86-64 and
     arm64 trampolines for such arguments (Yonghong Song)

   - Introduce a common way to attach struct_ops to a cgroup via
     bpf_link. bpf_tcp_ops is the first user and mirrors TCP sock_ops
     callbacks (Martin KaFai Lau, Amery Hung)

   - Describe inlined functions in BTF. New kinds tell where the inline
     sites are and how to fetch the arguments there.

     CONFIG_DEBUG_INFO_BTF_INLINE keeps it apart from the main BTF in
     .BTF.inline section or in btf_vmlinux_inline.ko (Alan Maguire)

   - Fix many memory safety, lifetime and locking bugs in BPF core,
     verifier, iterators, local storage, dynptr, devmap, cpumap and
     seg6local (Josef Bacik, Alexei Starovoitov)

   - Diff stat is dominated by selftests and by removal of
     lib/test_bpf.c. Its 16k lines of instruction tests were converted
     to BPF selftests that BPF CI runs on every architecture it covers
     (Daniel Borkmann)

  Other features and fixes:

   - Add KASAN checks to memory accesses of JITed programs on x86-64
     (Alexis Lothoré)

   - Complete unification of helper, kfunc and subprog argument checks
     in the verifier (Amery Hung)

   - Add bpf_copy_from_user_mm[_str]() kfuncs (Anastasios Papagiannis)

   - Various features and fixes (Daniel Borkmann):
       - Add .bpf keyring for program signature verification, modelled
         after dm-verity keyring, and support ML-DSA in signing tools
       - Emit shorter and faster code in x86-64 JIT for patterns that
         LLVM generates
       - Fix various issues in gotox handling

   - Let BPF LSM set xattrs at inode creation with bpf_inode_init_xattr()
     (David Windsor, Daniel Borkmann)

   - Use sleepable page allocation in sleepable arena paths. Fix races
     between arena free, allocation and user page fault (Emil Tsalapatis)

   - Switch BPF crypto from lskcipher to AES-CBC and AES-ECB libraries
     (Eric Biggers)

   - Add bpf_proactive_reclaim() kfunc (Hui Zhu)

   - Render decl_tags in libbpf btf_dump and add tests for bpftool C
     dump (Ihor Solodrai)

   - Speed up tracing_multi attach on x86 (Jiri Olsa)

   - Various features and fixes (Kumar Kartikeya Dwivedi):
       - Add BPF_PROG_STREAM_OPEN command to read BPF streams through a
         file descriptor with poll support
       - Let kernel structs publish arena pointers with __arena type tag
       - Fix __uninit kfunc output buffers and fastcall rewrite with
         indirect stack accesses
       - Refuse CO-RE relocations for signed programs

   - Add enqueue_rcvq and dequeue_rcvq hooks to bpf_tcp_ops to adjust
     sk_rcvlowat and avoid unnecessary wakeups (Kuniyuki Iwashima)

   - Add bpf_ksock_sendto() kfunc and allow ksock connect and send from
     bpf_wq (Mahe Tardy)

   - Fix racy plt target update in arm64 bpf_arch_text_poke()
     (Matthew Wood)

   - Restructure BPF selftests Makefile as a layered build
     (Mykola Lysenko)

   - Speed up hash map lookups for u32 and u64 keys (Mykyta Yatsenko)

   - Add signed div/mod, movsx, bswap and gotol to MIPS JITs, signed
     div/mod to parisc JITs (Nicholas Dudar, Johan Almbladh)

   - Fix stack overflow in build ID parser on 32-bit (Nickolai
     Zeldovich)

   - Add stack arguments, signed arena loads and gotox to riscv64 JIT.
     Pu Lehui is now co-maintainer of riscv BPF JITs (Pu Lehui, Chen
     Pei, Feng Jiang)

   - Add bpf_call_rcu() and bpf_call_rcu_tasks_trace() kfuncs. Support
     fetching and/or/xor atomics in arena on x86-64 (Puranjay Mohan)

   - Fix flaky selftests, selftests with clang 23 and bpf-gcc
     (Qiliang Yuan, KaFai Wan, Vineet Gupta)

   - Fix builds broken by fatal resolve_btfids warnings, BTF generation
     on ARC, and define arch_cmpxchg_local() on arc, hexagon and
     openrisc (Sasha Levin)

   - Reduce memory use of hash maps by up to 50% per element
     (T.J. Mercier)

   - Mark pointer fields of sched_ext ops arguments as trusted (Tejun
     Heo)

   - Remove legacy bpf_asm, bpf_dbg and bpf_jit_disasm tools (Xia Yan)

   - Fix index overflow in queue and stack maps, arena range tree after
     allocation failure, tail call loop after freplace detach, lost
     signal in bpf_send_signal() (Yuan Chen)

   - Many more fixes and cleanups"

* tag 'bpf-next-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (660 commits)
  selftests/bpf: Time tc_edt from its first packet with data
  selftests/bpf: Test unwind_exit of an exception callback
  selftests/bpf: Tests for 32-bit scalar evolution bounds computation
  bpf: Reject unwind_goto and unwind_exit in offloaded progs
  bpf: Support 32-bit linear scalar evolution
  bpf: Make unwind_exit of an exception callback exit in JITs
  selftests/bpf: Test forbidden bpf_ksock_sendto() LSM attach
  selftests/bpf: Test ksock sendto()
  bpf: Add bpf_ksock_sendto() kfunc
  selftests/bpf: Test unwinding from R0: R2 subprog
  selftests/bpf: Add a C++ prog with exceptions
  selftests/bpf: Add tests for unwind_goto and unwind_exit
  bpf, arm64: Support unwind_goto and unwind_exit
  bpf, x86: Support unwind_goto and unwind_exit
  bpf: Add unwind_goto and unwind_exit insns
  tools/bpf: Remove legacy BPF tools
  bpf: Annotate data races on bpf_lru_node type
  selftests/bpf: Add arena allocation race tests
  bpf: Fix arena race between page free and alloc leading to incoherency
  bpf: Revalidate faulted arena pages before installing user mappings
  ...
This commit is contained in:
Linus Torvalds committed 2026-10-23 13:08:49 -10:00
2 parents 90b754d401 b116ac6087
commit 55fbedcec6
570 files changed
+60732 -26284
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitignore
+1
View File
@@ -73,6 +73,7 @@ modules.order
/vmlinux.symvers
/vmlinux.thinlto-index
/vmlinux.unstripped
/vmlinux.BTF.inline
/vmlinux-gdb.py
/vmlinuz
/System.map
Documentation/admin-guide/kernel-parameters.txt
+16
View File
@@ -657,6 +657,22 @@ Kernel parameters
See Documentation/admin-guide/bootconfig.rst
bpf.keyring_unsealed=
[BPF] When set to 1, leave the bpf keyring unsealed
after initialization so that userspace can provision
keys. Once the keyring is restricted it becomes active
and can be used for BPF program signature verification.
Setting this also means that the bpf keyring is the
only non-system keyring a loader may select for the
rest of the boot: caller-supplied user/session
keyrings are refused with -EPERM, whether or not
provisioning actually completed. The system keyrings
stay selectable. Leaving it unset keeps the prior
behaviour, where a caller-supplied keyring is allowed.
See Documentation/bpf/signing.rst
bttv.card= [HW,V4L] bttv (bt848 + bt878 based grabber cards)
bttv.radio= Most important insmod options are available as
kernel args too.
Documentation/bpf/bpf_design_QA.rst
+8 -3
View File
@@ -221,9 +221,14 @@ newer kernels. BPF programs need to change accordingly when this happens.
Q: How much stack space a BPF program uses?
-------------------------------------------
A: Currently all program types are limited to 512 bytes of stack
space, but the verifier computes the actual amount of stack used
and both interpreter and most JITed code consume necessary amount.
A: A program may use up to 2 KiB of stack, combined over its call
chain, when the JIT of the architecture reports support for large
stacks (currently x86-64 and arm64); a single function may use all of
it, and every frame of a program running on a private stack gets the
whole amount. Elsewhere, and for programs verified for the interpreter,
the limit is 512 bytes; the interpreter itself never runs a frame
larger than that. The verifier computes the actual amount of stack
used and both interpreter and most JITed code consume necessary amount.
Q: Can BPF be offloaded to HW?
------------------------------
Documentation/bpf/bpf_devel_QA.rst
+9 -7
View File
@@ -374,9 +374,10 @@ developers in order to get the feature implemented in a timely manner.
Please refer to the git log (``arch/*/net/``) to locate the necessary
people for helping out.
Also always make sure to add BPF test cases (e.g. test_bpf.c and
test_verifier.c) for new instructions, so that they can receive
broad test coverage and help run-time testing the various BPF JITs.
Also always make sure to add BPF test cases (e.g. the jit_insns test
in test_progs and test_verifier.c) for new instructions, so that they
can receive broad test coverage and help run-time testing the various
BPF JITs.
In case of new BPF instructions, once the changes have been accepted
into the Linux kernel, please implement support into LLVM's BPF back
@@ -514,10 +515,11 @@ A: If you run a kernel ``xyz``, then always run the BPF kernel selftests
from that kernel ``xyz`` as well. Do not expect that the BPF selftest
from the latest mainline tree will pass all the time.
In particular, test_bpf.c and test_verifier.c have a large number of
test cases and are constantly updated with new BPF test sequences, or
existing ones are adapted to verifier changes e.g. due to verifier
becoming smarter and being able to better track certain things.
In particular, test_verifier.c and the instruction tests in test_progs
have a large number of test cases and are constantly updated with new
BPF test sequences, or existing ones are adapted to verifier changes
e.g. due to verifier becoming smarter and being able to better track
certain things.
LLVM
====
Documentation/bpf/btf.rst
+82 -3
View File
@@ -88,6 +88,9 @@ sequentially and type id is assigned to each recognized type starting from id
#define BTF_KIND_DECL_TAG 17 /* Decl Tag */
#define BTF_KIND_TYPE_TAG 18 /* Type Tag */
#define BTF_KIND_ENUM64 19 /* Enumeration up to 64-bit values */
#define BTF_KIND_LOC_PARAM 20 /* Location description (register, const etc) */
#define BTF_KIND_LOC_PROTO 21 /* Set of location parameters for site */
#define BTF_KIND_LOCSEC 22 /* Section with site descriptions */
Note that the type section encodes debug info, not just pure types.
``BTF_KIND_FUNC`` is not a type, and it represents a defined subprogram.
@@ -104,11 +107,13 @@ Each type contains the following common data::
* decl_tag and type_tag
*/
__u32 info;
/* "size" is used by INT, ENUM, STRUCT, UNION and ENUM64.
/* "size" is used by INT, ENUM, STRUCT, UNION, ENUM64 and
* LOC_PARAM.
* "size" tells the size of the type it is describing.
*
* "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT,
* FUNC, FUNC_PROTO, DECL_TAG and TYPE_TAG.
* FUNC, FUNC_PROTO, DECL_TAG and TYPE_TAG. It is unused by
* LOC_PROTO and LOCSEC.
* "type" is a type_id referring to another type.
*/
union {
@@ -416,7 +421,7 @@ If the function has variable arguments, the last parameter is encoded with
* ``info.vlen``: 0
* ``type``: the type of the variable
``btf_type`` is followed by a single ``struct btf_variable`` with the
``btf_type`` is followed by a single ``struct btf_var`` with the
following data::
struct btf_var {
@@ -563,6 +568,80 @@ The ``btf_enum64`` encoding:
If the original enum value is signed and the size is less than 8,
that value will be sign extended into 8 bytes.
2.2.20 BTF_KIND_LOC_PARAM
~~~~~~~~~~~~~~~~~~~~~~~~~~
``struct btf_type`` encoding requirement:
* ``name_off``: 0
* ``info.kind_flag``: 0
* ``info.kind``: BTF_KIND_LOC_PARAM
* ``info.vlen``: number of 32-bit location value words
* ``size``: size in bytes of the represented parameter: 1 through 16
``btf_type`` is followed by a ``struct btf_loc_param`` and ``info.vlen``
number of 32-bit value words.::
struct btf_loc_param {
__u32 flags;
__u32 values[];
};
The ``flags`` field describes how to interpret ``values``:
* ``BTF_LOC_PARAM_CONST`` describes a constant; the value is stored in
low-word, high-word order when it requires 64 bits.
* ``BTF_LOC_PARAM_ADDR | BTF_LOC_PARAM_CONST`` describes an address offset
relative to the runtime base address of the kernel or module image.
* ``BTF_LOC_PARAM_REG`` with one word describes a register number; with two
words it describes a multi-register parameter.
* ``BTF_LOC_PARAM_REG | BTF_LOC_PARAM_OFFSET`` describes an address held in
a register plus an offset. Adding ``BTF_LOC_PARAM_DEREF`` dereferences
that address. ``BTF_LOC_PARAM_REG | BTF_LOC_PARAM_DEREF`` with one word
dereferences the value held in the register.
* ``BTF_LOC_PARAM_SIGNED`` makes a constant or offset signed. A constant's
signed width is ``size``. For a register-relative offset, its signed
width is the number of offset value words times 32 bits.
2.2.21 BTF_KIND_LOC_PROTO
~~~~~~~~~~~~~~~~~~~~~~~~~~
``struct btf_type`` encoding requirement:
* ``name_off``: 0
* ``info.kind_flag``: 0
* ``info.kind``: BTF_KIND_LOC_PROTO
* ``info.vlen``: number of function parameter locations
* ``type``: 0
``btf_type`` is followed by ``info.vlen`` number of ``__u32`` BTF type IDs.
Each entry corresponds to a function parameter at an inline site. An entry is
either 0, meaning that no location information is available, or the type ID
of a ``BTF_KIND_LOC_PARAM``.
2.2.22 BTF_KIND_LOCSEC
~~~~~~~~~~~~~~~~~~~~~~
``struct btf_type`` encoding requirement:
* ``name_off``: offset to a valid ELF section name
* ``info.kind_flag``: 0
* ``info.kind``: BTF_KIND_LOCSEC
* ``info.vlen``: number of inline sites in the section
* ``type``: 0
``btf_type`` is followed by ``info.vlen`` number of ``struct btf_loc``.::
struct btf_loc {
__u32 func;
__u32 loc_proto;
__u32 offset;
};
The ``func`` field is the non-zero type ID of the ``BTF_KIND_FUNC`` being
described. ``loc_proto`` is the non-zero type ID of the associated
``BTF_KIND_LOC_PROTO``. ``offset`` is the location address offset relative to
the runtime base address of the ELF section associated with the LOCSEC.
For example, a LOCSEC named ``inline.text`` contains records for ``.text``
whose offsets are relative to the runtime base of that section.
2.3 Constant Values
-------------------
Documentation/bpf/clang-notes.rst
+5 -2
View File
@@ -23,8 +23,11 @@ For CPU versions prior to 3, Clang v7.0 and later can enable ``BPF_ALU`` support
Jump instructions
=================
If ``-O0`` is used, Clang will generate the ``BPF_CALL | BPF_X | BPF_JMP`` (0x8d)
instruction, which is not supported by the Linux kernel verifier.
Clang generates the ``BPF_CALL | BPF_X | BPF_JMP`` (0x8d) instruction for calls
through a function pointer. The Linux kernel verifier accepts it only when it
can prove that the register holds the address of a static BPF function, see
Documentation/bpf/linux-notes.rst. If ``-O0`` is used, Clang will generate this
instruction for helper calls as well, which is not supported.
Atomic operations
=================
Documentation/bpf/kfuncs.rst
+118 -8
View File
@@ -164,19 +164,32 @@ suffix should be used.
2.3.3 __uninit Annotation
-------------------------
This annotation is used to indicate that the argument will be treated as
uninitialized.
Use ``__uninit`` on a pointer parameter for an output buffer whose incoming
contents the kfunc does not read.
An example is given below::
For generic memory buffers, the kfunc may leave bytes untouched, including
on error returns. The writable range is determined by the pointed-to type
or the associated ``__sz`` or ``__szk`` size argument.
__bpf_kfunc int bpf_dynptr_from_skb(..., struct bpf_dynptr_kern *ptr__uninit)
The annotation does not change the accepted pointer types. A stack-backed
struct passed as a generic memory buffer must still be scalar-only.
Generic stack buffers may be uninitialized before the call. The call does
not make previously uninitialized bytes readable unless the program is
allowed to read uninitialized stack memory (normally requiring
``CAP_PERFMON``). Other callers must initialize those bytes themselves
before reading them. Stack bounds and variable-offset restrictions still
apply.
For dynptr parameters, ``__uninit`` indicates that the kfunc constructs a
dynptr in the supplied storage. For example::
__bpf_kfunc int bpf_dynptr_from_skb(..., struct bpf_dynptr *ptr__uninit)
{
...
}
Here, the dynptr will be treated as an uninitialized dynptr. Without this
annotation, the verifier will reject the program if the dynptr passed in is
not initialized.
Without this annotation, a dynptr argument must already be initialized.
2.3.4 __nullable Annotation
---------------------------
@@ -315,6 +328,29 @@ However, there is no obligation to prove to the verifier that such a pointer is
non-NULL before use, in-line with existing semantics of arena pointers used in
a program (or obtained from any other source).
2.3.9 __ctx_out Annotation
--------------------------
This annotation is used to indicate that the argument is an output
parameter of the attached hook, passed through from the program's
context. The verifier requires the register to be a trusted read-only
pointer to fixed-size memory, which can only be produced by loading an
argument described by the program's ctx_arg_info from the context. The
program itself cannot write through the pointer; the kfunc may.
An example is given below::
__bpf_kfunc int bpf_inode_init_xattr(struct xattr *xattrs,
int *xattr_count__ctx_out,
...)
{
...
}
In this case, a program attached to the ``inode_init_security`` LSM hook
can pass the hook's own ``xattr_count`` argument through to the kfunc,
which claims xattr slots by writing through it.
.. _BPF_kfunc_nodef:
2.4 Using an existing kernel function
@@ -546,7 +582,7 @@ For example, for the following type definition:
The verifier would allow a ``struct bpf_cpumask *`` to be passed to a kfunc
taking a ``cpumask_t *`` (which is a typedef of ``struct cpumask *``). For
instance, both ``struct cpumask *`` and ``struct bpf_cpmuask *`` can be passed
instance, both ``struct cpumask *`` and ``struct bpf_cpumask *`` can be passed
to bpf_cpumask_test_cpu().
In some cases, this type-aliasing behavior is not desired. ``struct
@@ -585,6 +621,80 @@ is also covered by this recovery. A kfunc handed an arena pointer may
therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking
against the arena. Larger accesses must verify the range explicitly.
2.9 kfunc Return Values
-----------------------
A kfunc may return a scalar, a pointer, or a small struct or union by
value. A scalar or pointer of up to 8 bytes is returned in R0, as usual.
A struct or union returned by value must be composed only of scalars and arena
pointers, where a scalar is an integer or an enum and an arena pointer is one
carrying the ``btf_type_tag("arena")`` attribute. Those may be nested in
structs and unions and in arrays of any number of dimensions, in any
combination, as long as what the nesting bottoms out in is a scalar or an arena
pointer. Its bytes are handed back to the program as the raw contents of R0
(and R2), so a member of any other pointer type would be laundered into a
scalar and escape the verifier's pointer provenance and reference tracking. A
struct or union with such a member is therefore rejected at load time, and so
is one with a floating-point member, which the ABI may not return in R0:R2 at
all.
An arena pointer member is handed back as a scalar too, but nothing is lost by
that. The program must ``cast_kern()`` the value before it can be used, and the
verifier allows that cast on any scalar, so a laundered arena address gives the
program no reach it did not already have. The result is confined to the
program's arena in either case.
A kfunc may also return a value larger than 8 bytes and up to 16 bytes -- a
struct or union of scalars and arena pointers, or an ``__int128``. Such a
value is returned in the register pair R0:R2, matching the convention LLVM
uses for the BPF target: the first 8 bytes in R0 and the second 8 bytes in R2.
A struct or union of 8 bytes or less is returned in R0 alone.
::
struct bpf_pair { __u64 a, b; }; /* 16 bytes */
__bpf_kfunc struct bpf_pair bpf_kfunc_get_pair(void)
{
struct bpf_pair p = { .a = 1, .b = 2 };
return p; /* p.a in R0, p.b in R2 */
}
Returning a value in the R0:R2 pair requires the JIT to place the second
half of the return value into R2, which not every architecture supports
right now. A kfunc with a return value larger than 8 bytes is therefore
rejected at load time on a JIT that does not advertise this capability (see
``bpf_jit_supports_kfunc_ret_reg_pair()``), and such a program is never run
by the interpreter. A return value larger than 16 bytes is not supported.
The same R0:R2 convention applies to a BPF subprogram, global or static, that
returns an ``__int128`` or a struct or union larger than 8 bytes. It is only
used when the program is JITed, since the interpreter propagates only R0 out of
a subprogram: without a JIT the return value stays in R0 alone, and a caller
reading R2 is rejected for reading an uninitialized register. A global
subprogram is verified in isolation, so its by-value struct or union return is
restricted to scalars and arena pointers just like a kfunc's; a static
subprogram is verified inline and has no such restriction. The main program is
not covered: its return value is the program's exit code, read out of R0
alone, so a declared upper half is never looked at.
A global subprogram must leave a scalar in *every* register of the pair, so
both halves of the returned value have to be assigned. Leaving the upper half
uninitialized is not merely untidy: the compiler is then free to leave R2
holding whatever it happened to hold, which for a subprogram taking a pointer
argument is typically that pointer. Handing the caller an unknown scalar built
from a pointer is a leak, so the verifier rejects it with::
At subprogram exit the register R2 is not a scalar value (...)
Initialize the whole return value, for example ``struct pair p = {};``, to
avoid this. A static subprogram is exempt from the scalar-only rule: it is
verified inline, so an unassigned R2 is simply passed back to the caller as
uninitialized and only a caller that reads it fails. A stack pointer left in
R2 is still rejected there, just as one in R0 is.
.. _BPF_kfunc_lifecycle_expectations:
3. kfunc lifecycle expectations
Documentation/bpf/linux-notes.rst
+27 -4
View File
@@ -15,10 +15,33 @@ Byte swap instructions
Jump instructions
=================
``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), where the helper function
integer would be read from a specified register, is not currently supported
by the verifier. Any programs with this instruction will fail to load
until such support is added.
``BPF_CALL | BPF_X | BPF_JMP`` (0x8d), ``callx dst``, performs an indirect
call of a BPF function whose address is held in the ``dst`` register. The
``src``, ``offset`` and ``imm`` fields are reserved and must be zero.
The address of a BPF function gets into a register in one of two ways:
* it is loaded by a 64-bit immediate instruction with ``src`` =
``BPF_PSEUDO_FUNC``;
* it is read, with a 64-bit load, from a frozen read-only array map, that no
other program uses, that holds its read-only data: tables of functions,
structures of operations, vtables, where pointers to functions may be mixed
with other data. In the map a pointer to a function is the offset in bytes
of its first instruction in the program, and that is how the verifier
recognizes it. It is replaced with the address of the function when
the program is loaded. The program reads it from there, which requires
``CAP_PERFMON``.
In both cases only static functions can be referenced. Therefore all functions
that can be called indirectly are known to the verifier before it starts to
analyze the program, and ``callx`` is verified as a direct call of every
function that ``dst`` may point to at that instruction. The same rules apply:
the calls can not be recursive, and the depth of the call chain and its
combined stack size are limited.
Calling helper or kernel functions through a register, indirect calls of global
functions, and tail calls in functions that are called via ``callx`` are not
supported. ``callx`` requires the BPF JIT.
Maps
====
Documentation/bpf/signing.rst
+246 -37
View File
@@ -254,21 +254,24 @@ returned. Only after the program has fully loaded, at the next hook
(``security_bpf_prog()``), does ``BPF_SIG_VERIFIED`` carry its full meaning:
validly signed *and* fully verified.
A more realistic admission policy than "is it signed at all": accept programs
signed by a system keyring, accept a user-keyring signature only if the
key/keyring it was verified against is on an explicit allowlist, and emit a
tamper-evident record of every decision so that even denied attempts are
auditable. (Illustrative - error checking elided.)
A more realistic admission policy than "is it signed at all": base trust in
the bpf keyring or an explicit allowlist of a caller-supplied staging key/
keyring, and emit a record of every decision including denied attempts.
(Illustrative - error checking elided.)
.. code-block:: c
/* Serials of user keys/keyrings we additionally trust. */
/*
* Serials of caller-supplied keyrings we are willing to stage. Empty
* on a system that has committed to the bpf keyring, where the kernel
* refuses them anyway.
*/
struct {
__uint(type, BPF_MAP_TYPE_HASH);
__type(key, __s32); /* keyring_serial */
__type(value, __u8);
__uint(max_entries, 64);
} trusted_user_keys SEC(".maps");
} staging_keys SEC(".maps");
/* Audit stream consumed by a userspace logger. */
struct {
@@ -291,11 +294,18 @@ auditable. (Illustrative - error checking elided.)
if (kernel)
return 0; /* trust in-kernel loads */
if (verdict != BPF_SIG_VERIFIED)
if (verdict != BPF_SIG_VERIFIED) {
ret = -EPERM; /* must be validly signed */
else if (ktype == BPF_SIG_KEYRING_USER &&
!bpf_map_lookup_elem(&trusted_user_keys, &serial))
ret = -EPERM; /* key/keyring not allowlisted */
} else switch (ktype) {
case BPF_SIG_KEYRING_BPF:
break;
case BPF_SIG_KEYRING_USER:
if (!bpf_map_lookup_elem(&staging_keys, &serial))
ret = -EPERM;
break;
default:
ret = -EPERM; /* keyring not in policy */
}
d = bpf_ringbuf_reserve(&audit, sizeof(*d), 0);
if (d) {
@@ -309,6 +319,11 @@ auditable. (Illustrative - error checking elided.)
return ret;
}
Such a policy is what makes a caller-supplied keyring usable at all on a system
that does not boot with ``bpf.keyring_unsealed=1``: the allowlist bounds which
staged keys count, and the LSM itself has to protect them from being tampered
with.
Observing a verified load: ``security_bpf_prog()``
--------------------------------------------------
@@ -381,8 +396,9 @@ that verdict covered all of its exclusive maps, rejecting any that did not - so
a deny-by-default admission policy needs no second enforcement point. Use
``security_bpf_prog()`` to record or finally gate the verified programs once
they carry an id. The ``verdict``, ``keyring_type`` and ``keyring_serial`` fields
let a policy distinguish, for example, "verified and signed by a builtin key"
from "verified by a user key". A policy LSM such as IPE could consume the same
let a policy distinguish "verified against the operator's bpf keyring" from
"verified against a keyring the loader supplied itself", which is the
distinction that matters most. A policy LSM such as IPE could consume the same
hooks to enforce system policy without writing any BPF, though none implements
this today.
@@ -390,33 +406,158 @@ Keyrings
========
``keyring_id`` selects the trusted keyring the PKCS#7 signature is verified
against. The well-known ids ``0`` (builtin), ``VERIFY_USE_SECONDARY_KEYRING``
and ``VERIFY_USE_PLATFORM_KEYRING`` select the corresponding system keyrings;
any other value is treated as the serial of a user/session key or keyring.
The keyring is looked up first, before the signature bytes are examined, so a
signature naming a non-existent keyring is rejected up front, and a failed
verification aborts the load - so a program that loads successfully with a
signature always has consistent keyring fields recorded.
against. Four values are well-known; anything else is resolved as a caller-
supplied key or keyring, named either by its serial or by one of the other
``KEY_SPEC_*`` ids:
.. list-table::
:header-rows: 1
* - ``keyring_id``
- Keyring
* - ``0``
- builtin trusted keyring
* - ``VERIFY_USE_SECONDARY_KEYRING`` (``1``)
- secondary trusted keyring
* - ``VERIFY_USE_PLATFORM_KEYRING`` (``2``)
- platform keyring
* - ``KEY_SPEC_BPF_KEYRING`` (``-9``)
- the bpf keyring
* - anything else
- a caller-supplied user/session key or keyring
The keyring is resolved first, before the signature bytes are examined, so a
signature naming a keyring that cannot be used is rejected up front, and a
failed verification aborts the load - a program that loads successfully with
a signature therefore always has consistent keyring fields recorded.
The bpf keyring
---------------
A system keyring needs a kernel rebuild or a vouched-for enrollment to rotate a
key, and grants BPF-signing trust to keys trusted for everything else in the
kernel too. A caller-supplied keyring, at the other extreme, is filled by the
very process that loads the program and so carries no trust of its own.
The bpf keyring fills that gap and is the trust anchor which a signed BPF
deployment should be built on top of: a keyring named ``.bpf``, selected with
``KEY_SPEC_BPF_KEYRING``, that an operator provisions at boot with a key
scoped to BPF program loading and nothing else in the kernel's trust hierarchy.
It is owned by the operator rather than by the loader, and rotatable across a
reboot without touching the kernel image. It is modelled after the dm-verity
keyring (see ``dm_verity.keyring_unsealed=``) and provisioned the same way: an
initrd runs the ``keyctl`` steps below before handing off to the rootfs.
Provisioning
~~~~~~~~~~~~
The keyring is created during ``late_initcall`` and is **sealed empty** by
default: it carries a reject-all restriction, so no key can ever be added and
``KEY_SPEC_BPF_KEYRING`` fails with ``-ENOKEY`` for the whole boot.
``bpf.keyring_unsealed=1`` leaves it unrestricted at init so the initrd can
provision it. The keyring is not linked into any process keyring, but the
``KEY_SPEC_BPF_KEYRING`` special key id addresses it directly, so its serial
does not have to be looked up first. Steps would be as follows::
keyctl padd asymmetric "" -9 < signing_key.der
keyctl restrict_keyring -9
Both steps are required: the keyring is consulted only once it is **non-empty
and restricted**. An unrestricted keyring is ignored even when it holds keys,
so a half-provisioned keyring is inert rather than a weaker trust anchor, and a
load naming it fails with ``-ENOKEY`` and a verifier log. Restricting cannot
be undone.
More than one key is enrolled by repeating the ``keyctl padd`` step; the
restriction is applied once, after the last of them::
for key in /etc/bpf/keys/*.der; do
keyctl padd asymmetric "" -9 < $key
done
keyctl restrict_keyring -9
keyctl show -9
The restriction bounds what can be added, never what can be taken away. A key
that is already enrolled can still be unlinked, and the keyring cleared or
revoked, by anything running as root. That does not weaken the anchor, since
a keyring left empty is no longer consulted and a load naming it fails with
``-ENOKEY``, but it does take signed loading out until the next boot. Dropping
the user permissions the keyring no longer needs would close that; as a third
step in the initrd::
keyctl setperm -9 0x08030000
What remains is ``KEY_POS_SEARCH`` for the in-kernel search during verification,
plus ``KEY_USR_VIEW`` and ``KEY_USR_READ`` so the keyring stays visible in
``/proc/keys``. ``KEY_SPEC_BPF_KEYRING`` names the keyring but conveys no rights
of its own - the keyring is not possessed by anyone, so the ``KEY_POS_*`` bits
are only ever reached by the in-kernel search, and userspace is left with what
the ``KEY_USR_*`` bits grant. Once they are dropped, addressing it by the serial
``/proc/keys`` reports gets no further than the special id does.
Provisioning has to complete before control passes to the rootfs. The keyring
takes any number of keys for as long as it is unrestricted, so it is the
restriction that bounds the enrolled set, not the first enrollment. Before
the initrd hands off control, it must therefore restrict the keyring after
the last enrollment.
Enforcement
~~~~~~~~~~~
``bpf.keyring_unsealed=1`` states that the bpf keyring is *the* trust anchor for
this boot, so it does more than unseal. From the first program load onwards a
caller-supplied user/session keyring is refused with ``-EPERM`` and a verifier
log message, whether or not provisioning ever completed. The system keyrings
stay selectable.
Enforcement is readable at ``/sys/module/bpf/parameters/keyring_unsealed``, and
read-only there: the flag is ``__ro_after_init`` behind a 0444 parameter. It is
also derived from the boot flag rather than from the keyring's runtime state,
so there is no window early in boot during which a caller-supplied keyring is
still accepted.
Caller-supplied keyrings are for staging
----------------------------------------
A ``keyring_id`` naming a user or session key or keyring is a *staging*
mechanism, not a trust anchor: it is filled by the same userspace that loads the
program, so verifying against it establishes only that the loader signed what it
loaded. Its purpose is to let software installed onto a running system - whose
signing key is not enrolled anywhere yet - run signed until that key reaches the
bpf keyring on the next boot.
A system that has committed to the bpf keyring refuses this path outright (see
`Enforcement`_). A system that has not can still allow it, but a policy must
never treat ``BPF_SIG_KEYRING_USER`` as equivalent to the bpf or system
keyrings; it should allowlist the specific serials it is willing to stage and
pair that with a BPF LSM policy protecting those keys from tampering, as in
`Enforcement via LSMs`_.
Recorded fields
---------------
Two fields are recorded in ``prog->aux->sig`` for an LSM to inspect:
``keyring_type`` (``enum bpf_sig_keyring``)
Classified purely from ``keyring_id`` whenever the program is signed:
``BPF_SIG_KEYRING_BUILTIN``, ``_SECONDARY``, ``_PLATFORM`` for the system
keyrings, or ``_USER`` for a user/session keyring. It is
``BPF_SIG_KEYRING_NONE`` for an unsigned program.
keyrings, ``_BPF`` for the bpf keyring, or ``_USER`` for a caller-supplied
user/session keyring. It is ``BPF_SIG_KEYRING_NONE`` for an unsigned
program.
``keyring_serial`` (``s32``)
Set **only** on a successful verification, to the serial of the
**user/session key or keyring** that ``keyring_id`` resolved to - the
**caller-supplied key or keyring** that ``keyring_id`` resolved to - the
object the signature was verified against, not the individual asymmetric
key inside it that matched the signer. Passing
``KEY_SPEC_SESSION_KEYRING``, for example, records the session keyring's
serial. The system keyrings are trusted as a whole and expose no serial
here, so the serial is ``0`` for builtin, secondary and platform
signatures, and ``0`` for unsigned programs. In other words, a non-zero
``keyring_serial`` is exactly "verified against the user key/keyring with
this serial".
serial. The system keyrings and the bpf keyring are trusted as a whole and
expose no serial here, so the serial is ``0`` for them, and ``0`` for
unsigned programs. A non-zero ``keyring_serial`` is therefore exactly
"verified against the caller-supplied key/keyring with this serial", which
is exactly the case a policy has to scrutinise.
.. list-table::
:header-rows: 1
@@ -436,16 +577,51 @@ Two fields are recorded in ``prog->aux->sig`` for an LSM to inspect:
* - ``VERIFY_USE_PLATFORM_KEYRING``
- ``BPF_SIG_KEYRING_PLATFORM``
- ``0``
* - other (a user/session key serial)
* - ``KEY_SPEC_BPF_KEYRING``
- ``BPF_SIG_KEYRING_BPF``
- ``0``
* - other (a caller-supplied key serial)
- ``BPF_SIG_KEYRING_USER``
- serial of the resolved key/keyring
Producing a signed object
==========================
Producing and loading a signed object
=====================================
Generating a signing key
------------------------
Signing is algorithm agnostic: the algorithm comes from the X.509 certificate
and the PKCS#7 ``SignerInfo``. Anything the X.509 and PKCS#7 parsers understand
works with no BPF-side change. Both examples below read the certificate request
config from ``x509.genkey``, for which ``certs/x509.genkey`` serves as a
template (see Documentation/admin-guide/module-signing.rst). RSA::
openssl req -new -nodes -utf8 -sha256 -days 36500 -batch -x509 \
-config x509.genkey -outform PEM \
-out signing_key.pem -keyout signing_key.pem
openssl x509 -in signing_key.pem -outform der -out signing_key.der
ML-DSA-87 (FIPS-204), which needs openssl 3.5 or later, and both
``CONFIG_CRYPTO_MLDSA`` and ``CONFIG_CRYPTO_SHA512`` in the kernel - the latter
for the signedAttrs digest described below, which ``CONFIG_CRYPTO_MLDSA`` does
not select. Note the absence of a digest option: ML-DSA hashes the message
itself, so openssl ignores an explicit digest for it::
openssl req -new -nodes -utf8 -days 36500 -batch -x509 \
-newkey ML-DSA-87 -config x509.genkey -outform PEM \
-out signing_key.pem -keyout signing_key.pem
openssl x509 -in signing_key.pem -outform der -out signing_key.der
``bpftool`` handles the following internally: openssl before 4.0 cannot combine
ML-DSA with ``CMS_NOATTR``, so it falls back to signedAttrs, where only SHA-512
is permitted. This mirrors what module signing does as well.
Signing
-------
``bpftool`` generates and signs a light skeleton in one step::
bpftool gen skeleton -L -S -k <private_key.pem> -i <certificate.x509> \
bpftool gen skeleton -L -S -k signing_key.pem -i signing_key.der \
obj.bpf.o > obj.lskel.h
``-L`` selects the light-skeleton (``gen_loader``) backend and ``-S`` enables
@@ -454,12 +630,36 @@ signing; ``-k`` and ``-i`` supply the signing key and its X.509 certificate.
reconstructs - and also computes ``excl_prog_hash`` as the digest of the loader
instructions so the metadata map can be bound to the loader. The signature and
hash are embedded in the generated header; the certificate is used only for
signing and is not included. Loading the skeleton performs the
create/populate/freeze/load sequence described above.
signing and is not included.
At runtime the trusted public key must be present in the chosen keyring (for
example added to the session keyring, or built into the kernel's builtin trusted
keyring) for verification to succeed.
Loading
-------
The generated skeleton exposes ``keyring_id``, which selects the keyring the
kernel verifies against. Set it between open and load; loading then performs
the create/populate/freeze/load sequence described above::
struct obj *skel = obj__open();
skel->keyring_id = KEY_SPEC_BPF_KEYRING;
err = obj__load(skel);
For the staging case the same object is loaded against a keyring the caller
populated itself, which only works on a system that has not set
``bpf.keyring_unsealed=1``::
/*
* Staging only: this keyring is under the loader's own control and
* carries no trust of its own. See "Caller-supplied keyrings are for
* staging".
*/
key_id = add_key("asymmetric", "", der, der_sz, KEY_SPEC_SESSION_KEYRING);
skel->keyring_id = KEY_SPEC_SESSION_KEYRING;
err = obj__load(skel);
Either way the trusted public key must already be in the chosen keyring for
verification to succeed. For the bpf keyring that enrollment happens once at
boot, see `Provisioning`_.
UAPI reference
==============
@@ -487,6 +687,13 @@ UAPI reference
The map content is not hashed separately at all - it is covered, as bytes,
by the program signature.
Kernel command line:
``bpf.keyring_unsealed=``
Set to ``1`` to leave the bpf keyring unsealed for provisioning, and to make
it the only non-system keyring a loader may select for the rest of the boot
(see `The bpf keyring`_).
Notes and limitations
======================
@@ -495,3 +702,5 @@ Notes and limitations
exceed it.
- The metadata container is a single-element array map, accessed through
``map_direct_value_addr``.
- The bpf keyring needs ``CONFIG_KEYS``; without it there is no bpf keyring
and ``KEY_SPEC_BPF_KEYRING`` never resolves.
Documentation/networking/filter.rst
+14 -227
View File
@@ -183,12 +183,9 @@ access to BPF code as well.
BPF engine and instruction set
------------------------------
Under tools/bpf/ there's a small helper tool called bpf_asm which can
be used to write low-level filters for example scenarios mentioned in the
previous section. Asm-like syntax mentioned here has been implemented in
bpf_asm and will be used for further explanations (instead of dealing with
less readable opcodes directly, principles are the same). The syntax is
closely modelled after Steven McCanne's and Van Jacobson's BPF paper.
The examples below use an assembly notation instead of less readable
opcodes. The notation is closely modelled after Steven McCanne's and
Van Jacobson's BPF paper.
The BPF architecture consists of the following basic elements:
@@ -201,8 +198,8 @@ The BPF architecture consists of the following basic elements:
store", addressable from 0 to 15
======= ====================================================
A program, that is translated by bpf_asm into "opcodes" is an array that
consists of the following elements (as already mentioned)::
A BPF program is an array of instructions with the following elements
(as already mentioned)::
op:16, jt:8, jf:8, k:32
@@ -213,9 +210,8 @@ contains a miscellaneous argument that can be interpreted in different
ways depending on the given instruction in op.
The instruction set consists of load, store, branch, alu, miscellaneous
and return instructions that are also represented in bpf_asm syntax. This
table lists all bpf_asm instructions available resp. what their underlying
opcodes as defined in linux/filter.h stand for:
and return instructions. This table lists the assembly notation and what
the corresponding opcodes defined in linux/filter.h do:
=========== =================== =====================
Instruction Addressing mode Description
@@ -374,141 +370,14 @@ Examples for low-level BPF extension:
ret #-1
drop: ret #0
The above example code can be placed into a file (here called "foo"), and
then be passed to the bpf_asm tool for generating opcodes, output that xt_bpf
and cls_bpf understands and can directly be loaded with. Example with above
ARP code::
The ARP filter shown above corresponds to the following sock_filter
entries::
$ ./bpf_asm foo
4,40 0 0 12,21 0 1 2054,6 0 0 4294967295,6 0 0 0,
In copy and paste C-like output::
$ ./bpf_asm -c foo
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 1, 0x00000806 },
{ 0x06, 0, 0, 0xffffffff },
{ 0x06, 0, 0, 0000000000 },
In particular, as usage with xt_bpf or cls_bpf can result in more complex BPF
filters that might not be obvious at first, it's good to test filters before
attaching to a live system. For that purpose, there's a small tool called
bpf_dbg under tools/bpf/ in the kernel source directory. This debugger allows
for testing BPF filters against given pcap files, single stepping through the
BPF code on the pcap's packets and to do BPF machine register dumps.
Starting bpf_dbg is trivial and just requires issuing::
# ./bpf_dbg
In case input and output do not equal stdin/stdout, bpf_dbg takes an
alternative stdin source as a first argument, and an alternative stdout
sink as a second one, e.g. `./bpf_dbg test_in.txt test_out.txt`.
Other than that, a particular libreadline configuration can be set via
file "~/.bpf_dbg_init" and the command history is stored in the file
"~/.bpf_dbg_history".
Interaction in bpf_dbg happens through a shell that also has auto-completion
support (follow-up example commands starting with '>' denote bpf_dbg shell).
The usual workflow would be to ...
* load bpf 6,40 0 0 12,21 0 3 2048,48 0 0 23,21 0 1 1,6 0 0 65535,6 0 0 0
Loads a BPF filter from standard output of bpf_asm, or transformed via
e.g. ``tcpdump -iem1 -ddd port 22 | tr '\n' ','``. Note that for JIT
debugging (next section), this command creates a temporary socket and
loads the BPF code into the kernel. Thus, this will also be useful for
JIT developers.
* load pcap foo.pcap
Loads standard tcpdump pcap file.
* run [<n>]
bpf passes:1 fails:9
Runs through all packets from a pcap to account how many passes and fails
the filter will generate. A limit of packets to traverse can be given.
* disassemble::
l0: ldh [12]
l1: jeq #0x800, l2, l5
l2: ldb [23]
l3: jeq #0x1, l4, l5
l4: ret #0xffff
l5: ret #0
Prints out BPF code disassembly.
* dump::
/* { op, jt, jf, k }, */
{ 0x28, 0, 0, 0x0000000c },
{ 0x15, 0, 3, 0x00000800 },
{ 0x30, 0, 0, 0x00000017 },
{ 0x15, 0, 1, 0x00000001 },
{ 0x06, 0, 0, 0x0000ffff },
{ 0x06, 0, 0, 0000000000 },
Prints out C-style BPF code dump.
* breakpoint 0::
breakpoint at: l0: ldh [12]
* breakpoint 1::
breakpoint at: l1: jeq #0x800, l2, l5
...
Sets breakpoints at particular BPF instructions. Issuing a `run` command
will walk through the pcap file continuing from the current packet and
break when a breakpoint is being hit (another `run` will continue from
the currently active breakpoint executing next instructions):
* run::
-- register dump --
pc: [0] <-- program counter
code: [40] jt[0] jf[0] k[12] <-- plain BPF code of current instruction
curr: l0: ldh [12] <-- disassembly of current instruction
A: [00000000][0] <-- content of A (hex, decimal)
X: [00000000][0] <-- content of X (hex, decimal)
M[0,15]: [00000000][0] <-- folded content of M (hex, decimal)
-- packet dump -- <-- Current packet from pcap (hex)
len: 42
0: 00 19 cb 55 55 a4 00 14 a4 43 78 69 08 06 00 01
16: 08 00 06 04 00 01 00 14 a4 43 78 69 0a 3b 01 26
32: 00 00 00 00 00 00 0a 3b 01 01
(breakpoint)
>
* breakpoint::
breakpoints: 0 1
Prints currently set breakpoints.
* step [-<n>, +<n>]
Performs single stepping through the BPF program from the current pc
offset. Thus, on each step invocation, above register dump is issued.
This can go forwards and backwards in time, a plain `step` will break
on the next BPF instruction, thus +1. (No `run` needs to be issued here.)
* select <n>
Selects a given packet from the pcap file to continue from. Thus, on
the next `run` or `step`, the BPF program is being evaluated against
the user pre-selected packet. Numbering starts just as in Wireshark
with index 1.
* quit
Exits bpf_dbg.
JIT compiler
------------
@@ -540,83 +409,6 @@ setting bpf_jit_enable to 2, since dumping the final JIT image into the kernel l
is discouraged and introspection through bpftool (under tools/bpf/bpftool/) is the
generally recommended approach instead.
In the kernel source tree under tools/bpf/, there's bpf_jit_disasm for
generating disassembly out of the kernel log's hexdump::
# ./bpf_jit_disasm
70 bytes emitted from JIT compiler (pass:3, flen:6)
ffffffffa0069c8f + <x>:
0: push %rbp
1: mov %rsp,%rbp
4: sub $0x60,%rsp
8: mov %rbx,-0x8(%rbp)
c: mov 0x68(%rdi),%r9d
10: sub 0x6c(%rdi),%r9d
14: mov 0xd8(%rdi),%r8
1b: mov $0xc,%esi
20: callq 0xffffffffe0ff9442
25: cmp $0x800,%eax
2a: jne 0x0000000000000042
2c: mov $0x17,%esi
31: callq 0xffffffffe0ff945e
36: cmp $0x1,%eax
39: jne 0x0000000000000042
3b: mov $0xffff,%eax
40: jmp 0x0000000000000044
42: xor %eax,%eax
44: leaveq
45: retq
Issuing option `-o` will "annotate" opcodes to resulting assembler
instructions, which can be very useful for JIT developers:
# ./bpf_jit_disasm -o
70 bytes emitted from JIT compiler (pass:3, flen:6)
ffffffffa0069c8f + <x>:
0: push %rbp
55
1: mov %rsp,%rbp
48 89 e5
4: sub $0x60,%rsp
48 83 ec 60
8: mov %rbx,-0x8(%rbp)
48 89 5d f8
c: mov 0x68(%rdi),%r9d
44 8b 4f 68
10: sub 0x6c(%rdi),%r9d
44 2b 4f 6c
14: mov 0xd8(%rdi),%r8
4c 8b 87 d8 00 00 00
1b: mov $0xc,%esi
be 0c 00 00 00
20: callq 0xffffffffe0ff9442
e8 1d 94 ff e0
25: cmp $0x800,%eax
3d 00 08 00 00
2a: jne 0x0000000000000042
75 16
2c: mov $0x17,%esi
be 17 00 00 00
31: callq 0xffffffffe0ff945e
e8 28 94 ff e0
36: cmp $0x1,%eax
83 f8 01
39: jne 0x0000000000000042
75 07
3b: mov $0xffff,%eax
b8 ff ff 00 00
40: jmp 0x0000000000000044
eb 02
42: xor %eax,%eax
31 c0
44: leaveq
c9
45: retq
c3
For BPF JIT developers, bpf_jit_disasm, bpf_asm and bpf_dbg provides a useful
toolchain for developing and testing the kernel's JIT compiler.
BPF kernel internals
--------------------
Internally, for the kernel interpreter, a different instruction set
@@ -656,16 +448,11 @@ from eBPF instruction set.
Testing
-------
Next to the BPF toolchain, the kernel also ships a test module that contains
various test cases for classic and eBPF that can be executed against
the BPF interpreter and JIT compiler. It can be found in lib/test_bpf.c and
enabled via Kconfig::
CONFIG_TEST_BPF=m
After the module has been built and installed, the test suite can be executed
via insmod or modprobe against 'test_bpf' module. Results of the test cases
including timings in nsec can be found in the kernel log (dmesg).
The BPF selftests under tools/testing/selftests/bpf ship instruction
tests in test_progs, see jit_insns, which runs the eBPF
instructions over a range of operand magnitudes and signs, register
combinations, memory offsets and jump offsets against the BPF interpreter
or the JIT compiler, depending on the net.core.bpf_jit_enable setting.
Misc
----
Documentation/networking/net_cachelines/tcp_sock.rst
+1
View File
@@ -152,6 +152,7 @@ u32 urg_seq
unsigned_int keepalive_time
unsigned_int keepalive_intvl
int linger2
u32 bpf_tcp_ops_flags read_mostly read_mostly bpf_tcp_ops_hdr_opt_len,bpf_skops_write_hdr_opt(tx);bpf_tcp_ops_parse_hdr,tcp_bpf_rtt,tcp_cleanup_rbuf,tcp_queue_rcv,tcp_ofo_queue,tcp_fastopen_add_skb(rx);
u8 bpf_sock_ops_cb_flags
u8:1 bpf_chg_cc_inprogress
u16 timeout_rehash
MAINTAINERS
+2 -4
View File
@@ -4895,6 +4895,7 @@ F: arch/powerpc/net/
BPF JIT for RISC-V (32-bit)
M: Luke Nelson <luke.r.nels@gmail.com>
M: Xi Wang <xi.wang@gmail.com>
M: Pu Lehui <pulehui@huawei.com>
L: bpf@vger.kernel.org
S: Maintained
F: arch/riscv/net/
@@ -4902,7 +4903,7 @@ X: arch/riscv/net/bpf_jit_comp64.c
BPF JIT for RISC-V (64-bit)
M: Björn Töpel <bjorn@kernel.org>
R: Pu Lehui <pulehui@huawei.com>
M: Pu Lehui <pulehui@huawei.com>
R: Puranjay Mohan <puranjay@kernel.org>
L: bpf@vger.kernel.org
S: Maintained
@@ -4959,8 +4960,6 @@ BPF [CRYPTO]
M: Vadim Fedorenko <vadim.fedorenko@linux.dev>
L: bpf@vger.kernel.org
S: Maintained
F: crypto/bpf_crypto_skcipher.c
F: include/linux/bpf_crypto.h
F: kernel/bpf/crypto.c
BPF [DOCUMENTATION] (Related to Standardization)
@@ -5005,7 +5004,6 @@ F: include/uapi/linux/filter.h
F: kernel/bpf/
F: kernel/trace/bpf_trace.c
F: lib/buildid.c
F: lib/test_bpf.c
F: net/bpf/
F: net/core/filter.c
F: net/sched/act_bpf.c
Makefile
+1
View File
@@ -1800,6 +1800,7 @@ endif # CONFIG_MODULES
CLEAN_FILES += vmlinux.symvers modules-only.symvers \
modules.builtin modules.builtin.modinfo modules.nsdeps \
modules.builtin.ranges vmlinux.o.map vmlinux.unstripped \
vmlinux.BTF.inline \
vmlinux.thinlto-index builtin.order \
compile_commands.json rust/test \
rust-project.json .vmlinux.objs .vmlinux.export.c \
arch/arc/Makefile
+8 -1
View File
@@ -86,7 +86,14 @@ core-$(CONFIG_ARC_PLAT_TB10X) += arch/arc/plat-tb10x/
core-$(CONFIG_ARC_PLAT_AXS10X) += arch/arc/plat-axs10x/
core-$(CONFIG_ARC_SOC_HSDK) += arch/arc/plat-hsdk/
libs-y += arch/arc/lib/ $(LIBGCC)
libs-y += arch/arc/lib/
ifdef CONFIG_DEBUG_INFO_BTF
export LIBGCC := $(LIBGCC)
libs-y += arch/arc/lib/libgcc.a
else
libs-y += $(LIBGCC)
endif
boot := arch/arc/boot
arch/arc/include/asm/cmpxchg.h
+4
View File
@@ -62,6 +62,8 @@
_prev_; \
})
#define arch_cmpxchg_local arch_cmpxchg_relaxed
#else
#define arch_cmpxchg(ptr, old, new) \
@@ -83,6 +85,8 @@
_prev_; \
})
#define arch_cmpxchg_local arch_cmpxchg
#endif
/*
arch/arc/lib/Makefile
+8
View File
@@ -13,3 +13,11 @@ lib-$(CONFIG_ISA_ARCV2) +=memcpy-archs-unaligned.o
else
lib-$(CONFIG_ISA_ARCV2) +=memcpy-archs.o
endif
# libgcc can contain complex floating-point debug types unsupported by BTF.
# Strip only the debug information from a local copy of the selected archive.
always-$(CONFIG_DEBUG_INFO_BTF) += libgcc.a
OBJCOPYFLAGS_libgcc.a := --strip-debug
$(obj)/libgcc.a: $(LIBGCC) FORCE
$(call if_changed,objcopy)
arch/arm64/net/bpf_jit_comp.c
+257 -55
View File
@@ -11,6 +11,7 @@
#include <linux/bitfield.h>
#include <linux/bpf.h>
#include <linux/cfi.h>
#include <linux/extable.h>
#include <linux/filter.h>
#include <linux/memory.h>
#include <linux/printk.h>
@@ -22,7 +23,6 @@
#include <asm/debug-monitors.h>
#include <asm/insn.h>
#include <asm/text-patching.h>
#include <asm/set_memory.h>
#include "bpf_jit.h"
@@ -1045,8 +1045,9 @@ static void __maybe_unused build_bhb_mitigation(struct jit_ctx *ctx)
}
}
static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
static void build_epilogue(struct jit_ctx *ctx, bool was_classic, bool unwind)
{
const u8 tmp = bpf2a64[TMP_REG_1];
const u8 r0 = bpf2a64[BPF_REG_0];
const u8 ptr = bpf2a64[TCCNT_PTR];
@@ -1074,6 +1075,10 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic)
if (IS_ENABLED(CONFIG_ARM64_PTR_AUTH_KERNEL))
emit(A64_AUTIASP, ctx);
/* tmp is the offset of the pad, see emit_unwind_exit() */
if (unwind)
emit(A64_ADD(1, A64_LR, A64_LR, tmp), ctx);
emit(A64_RET(A64_LR), ctx);
}
@@ -1222,6 +1227,82 @@ static int add_exception_handler(const struct bpf_insn *insn,
return 0;
}
/*
* There is no branch for unwind_goto, only an extable entry
* { .insn = ret_addr - 1, .fixup = pad - ret_addr }
* where ret_addr is the return address of the call in front of it. unwind_exit
* adds .fixup to its LR.
* ret_addr - 1, because what follows the call can have an entry of its own.
* No insn starts there, so a fault cannot hit the entry.
* 'bl foo; mov x8, x0' is in front of unwind_goto. The pad doesn't miss the mov:
* r0 is x8 and the epilogue of the callee copies it to x0 and leaves x8 as it is.
*/
static int add_unwind_exentry(struct jit_ctx *ctx, int pad)
{
int ret_addr = ctx->idx - 1;
struct exception_table_entry *ex;
off_t ins_offset;
if (!ctx->image)
/* First pass */
return 0;
if (!ctx->prog->aux->extable ||
WARN_ON_ONCE(ctx->exentry_idx >= ctx->prog->aux->num_exentries))
return -EINVAL;
ex = &ctx->prog->aux->extable[ctx->exentry_idx];
ins_offset = (long)&ctx->ro_image[ret_addr] - 1 - (long)&ex->insn;
if (WARN_ON_ONCE(ins_offset >= 0 || ins_offset < INT_MIN))
return -ERANGE;
/* switch ex to rw buffer for writing */
ex = (void *)ctx->image + ((void *)ex - (void *)ctx->ro_image);
ex->insn = ins_offset;
ex->fixup = (ctx->offset[pad] - ret_addr) * AARCH64_INSN_SIZE;
ex->type = EX_TYPE_BPF;
ctx->exentry_idx++;
return 0;
}
static long bpf_unwind_fixup(u64 lr)
{
const struct exception_table_entry *e;
e = search_bpf_extables(ptrauth_strip_kernel_insn_pac(lr) - 1);
/*
* The verifier made sure that unwind_goto follows a call in a prog. When
* the kernel called an async callback, it's exit.
*/
return e ? e->fixup : 0;
}
/*
* unwind_exit: return to the pad. LR in the frame record is signed. Instead of
* signing the address of the pad, let the epilogue authenticate LR as it is and
* add the offset of the pad to it. r0 and r2 are returned.
*/
static void emit_unwind_exit(struct jit_ctx *ctx)
{
const u8 tmp = bpf2a64[TMP_REG_1];
const u8 r0 = bpf2a64[BPF_REG_0];
const u8 r2 = bpf2a64[BPF_REG_2];
emit(A64_PUSH(r0, r2, A64_SP), ctx);
emit(A64_LDR64I(A64_R(0), A64_FP, 8), ctx);
emit_call((u64)bpf_unwind_fixup, ctx);
emit(A64_MOV(1, tmp, A64_R(0)), ctx);
emit(A64_POP(r0, r2, A64_SP), ctx);
build_epilogue(ctx, false, true);
}
static const struct bpf_jit_arg_abi arm64_arg_abi = {
.nr_arg_regs = 8,
.even_reg_align = true,
.even_stack_align = true,
};
static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) };
#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg)
@@ -1264,19 +1345,20 @@ static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct
* kern_vm_start. A nullable arg preserves NULL by skipping the add, tested
* on the truncated value as arena NULL is offset 0.
*/
static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *insn)
static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct btf_func_model *fm)
{
const u8 arena_vm_base = bpf2a64[ARENA_VM_START];
const struct btf_func_model *fm;
int i;
int i, slot;
fm = bpf_jit_find_kfunc_model(ctx->prog, insn);
if (!fm)
return -EINVAL;
for (i = 0; i < min_t(int, fm->nr_args, MAX_BPF_FUNC_REG_ARGS); i++) {
const u8 reg = bpf2a64[BPF_REG_1 + i];
for (i = 0, slot = 0; i < fm->nr_args; i++) {
u32 arg_regs = (fm->arg_size[i] + 7) / 8;
u8 flags = fm->arg_flags[i];
u8 reg;
if (slot + arg_regs > MAX_BPF_FUNC_REG_ARGS)
break;
reg = bpf2a64[BPF_REG_1 + slot];
slot += arg_regs;
if (!(flags & BTF_FMODEL_ARENA_ARG))
continue;
@@ -1295,6 +1377,52 @@ static int emit_kfunc_arena_args(struct jit_ctx *ctx, const struct bpf_insn *ins
return 0;
}
static bool a64_arg_on_stack(u8 slot)
{
return slot >= arm64_arg_abi.nr_arg_regs;
}
static s32 a64_arg_stack_off(u8 slot)
{
return (slot - arm64_arg_abi.nr_arg_regs) * sizeof(u64);
}
/*
* Move the arguments AAPCS64 places somewhere other than the argument slot the
* BPF calling convention gave them. Slot N is X(N) up to the eighth, and the
* outgoing stack argument area from SP beyond it, both for the slot an
* argument comes from and for the one it goes to.
*
* AAPCS64 only ever moves an argument to a higher slot, so no move here ever
* takes BPF_JIT_ARG_TMP: bpf_jit_plan_arg_moves() hands out the scratch only
* for a convention that moves one down, which needs a register to carry the
* value past its own destination.
*/
static void emit_kfunc_arg_moves(struct jit_ctx *ctx, const struct btf_func_model *fm)
{
struct bpf_jit_arg_move moves[BPF_JIT_MAX_ARG_MOVES];
const u8 tmp = bpf2a64[TMP_REG_1];
u32 i, n;
n = bpf_jit_plan_arg_moves(&arm64_arg_abi, fm, moves);
for (i = 0; i < n; i++) {
u8 dst = moves[i].dst, src = moves[i].src, reg;
if (a64_arg_on_stack(src)) {
reg = tmp;
emit(A64_LDR64I(reg, A64_SP, a64_arg_stack_off(src)), ctx);
} else {
reg = src;
}
if (a64_arg_on_stack(dst))
emit(A64_STR64I(reg, A64_SP, a64_arg_stack_off(dst)), ctx);
else if (reg != dst)
emit(A64_MOV(1, dst, reg), ctx);
}
}
/* JITs an eBPF instruction.
* Returns:
* 0 - successfully JITed an 8-byte eBPF instruction.
@@ -1713,9 +1841,15 @@ emit_cond_jmp:
if (ret < 0)
return ret;
if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
ret = emit_kfunc_arena_args(ctx, insn);
const struct btf_func_model *fm;
fm = bpf_jit_find_kfunc_model(ctx->prog, insn);
if (!fm)
return -EINVAL;
ret = emit_kfunc_arena_args(ctx, fm);
if (ret < 0)
return ret;
emit_kfunc_arg_moves(ctx, fm);
}
emit_call(func_addr, ctx);
/*
@@ -1727,13 +1861,34 @@ emit_cond_jmp:
emit(A64_MOV(1, r0, A64_R(0)), ctx);
break;
}
/* indirect call of a bpf subprog, dst holds its address */
case BPF_JMP | BPF_CALL | BPF_X:
/*
* It's the same as a direct call of a subprog that is out of
* range of BL: the subprog starts with BTI JC, the arguments
* are in place, and the registers that hold the tail call
* counter and the private stack are callee saved.
*/
emit(A64_BLR(dst), ctx);
emit(A64_MOV(1, bpf2a64[BPF_REG_0], A64_R(0)), ctx);
break;
/* tail call */
case BPF_JMP | BPF_TAIL_CALL:
if (emit_bpf_tail_call(ctx))
return -EFAULT;
break;
/* unwind_goto. The verifier expanded may_goto. */
case BPF_JMP | BPF_JCOND:
ret = add_unwind_exentry(ctx, i + 1 + off);
if (ret)
return ret;
break;
/* function return */
case BPF_JMP | BPF_EXIT:
if (bpf_is_unwind_exit(insn) && bpf_prog_can_unwind(ctx->prog)) {
emit_unwind_exit(ctx);
break;
}
/* Optimization: when last instruction is EXIT,
simply fallthrough to epilogue. */
if (i == ctx->prog->len - 1)
@@ -2210,8 +2365,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
if (ctx.offset == NULL)
goto out_off;
ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
if (bpf_prog_uses_arena_insns(prog)) {
ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
}
out_cnt = bpf_out_stack_arg_cnt(env, prog);
if (out_cnt) {
@@ -2236,7 +2393,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
goto out_off;
ctx.epilogue_offset = ctx.idx;
build_epilogue(&ctx, was_classic);
build_epilogue(&ctx, was_classic, false);
build_plt(&ctx);
extable_align = __alignof__(struct exception_table_entry);
@@ -2293,7 +2450,7 @@ skip_init_ctx:
WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset))
goto out_free_hdr;
build_epilogue(&ctx, was_classic);
build_epilogue(&ctx, was_classic, false);
build_plt(&ctx);
/* Extra pass to validate JITed code. */
@@ -2385,6 +2542,21 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
bool bpf_jit_supports_kfunc_ret_reg_pair(void)
{
return true;
}
bool bpf_jit_supports_unwind(void)
{
return true;
}
const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void)
{
return &arm64_arg_abi;
}
bool bpf_jit_supports_stack_args(void)
{
return true;
@@ -2413,6 +2585,16 @@ bool bpf_jit_supports_subprog_tailcalls(void)
return true;
}
bool bpf_jit_supports_callx(void)
{
return true;
}
bool bpf_jit_supports_large_stack(void)
{
return true;
}
static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_image *im,
struct bpf_tramp_node *node, int bargs_off,
int retval_off, int run_ctx_off, bool save_ret)
@@ -2528,33 +2710,41 @@ struct arg_aux {
* arguments to be properly aligned)
*/
int ostack_for_args;
/* where AAPCS64 puts each argument slot: an argument register below
* the eighth, an on-stack argument slot from it up
*/
u8 pos_of_slot[MAX_BPF_FUNC_ARG_SLOTS];
};
static int calc_arg_aux(const struct btf_func_model *m,
struct arg_aux *a)
{
int stack_slots, nregs, slots, i;
int slots, i, slot, total;
total = bpf_jit_place_args(&arm64_arg_abi, m, a->pos_of_slot);
if (total > MAX_BPF_FUNC_ARGS)
return -ENOTSUPP;
/* verifier ensures m->nr_args <= MAX_BPF_FUNC_ARGS */
for (i = 0, nregs = 0; i < m->nr_args; i++) {
for (i = 0, slot = 0; i < m->nr_args; i++) {
slots = (m->arg_size[i] + 7) / 8;
if (nregs + slots <= 8) /* passed through register ? */
nregs += slots;
else
if (a64_arg_on_stack(a->pos_of_slot[slot])) /* passed through register ? */
break;
slot += slots;
}
a->args_in_regs = i;
a->regs_for_args = nregs;
a->regs_for_args = slot;
a->ostack_for_args = 0;
a->bstack_for_args = 0;
/* the rest arguments are passed through stack */
for (; i < m->nr_args; i++) {
stack_slots = (m->arg_size[i] + 7) / 8;
a->bstack_for_args += stack_slots * 8;
a->ostack_for_args = a->ostack_for_args + stack_slots * 8;
}
for (; i < m->nr_args; i++)
a->bstack_for_args += ((m->arg_size[i] + 7) / 8) * 8;
/* the outgoing area reaches the last slot, over any alignment hole */
if (a->bstack_for_args)
a->ostack_for_args = a64_arg_stack_off(a->pos_of_slot[total - 1]) + 8;
return 0;
}
@@ -2595,7 +2785,7 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
{
u8 tmp = bpf2a64[TMP_REG_1];
u8 base_lo = bpf2a64[TMP_REG_2];
int i, reg, doff, soff, slots;
int i, reg, slot, soff, slots;
/* only the low 32 bits of the base take part in the subtraction */
if (arena_base)
@@ -2604,12 +2794,13 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
/* store arguments to the stack for the bpf program, or restore
* arguments from stack for the original function
*/
for (i = 0, reg = 0; i < a->args_in_regs; i++) {
for (i = 0, slot = 0; i < a->args_in_regs; i++) {
bool arena_arg = arena_base && (m->arg_flags[i] & BTF_FMODEL_ARENA_ARG);
bool nullable = m->arg_flags[i] & BTF_FMODEL_NULLABLE_ARG;
slots = (m->arg_size[i] + 7) / 8;
while (slots-- > 0) {
reg = a->pos_of_slot[slot++];
if (for_call_origin) {
emit(A64_LDR64I(reg, A64_SP, bargs_off), ctx);
} else if (arena_arg) {
@@ -2618,7 +2809,6 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
} else {
emit(A64_STR64I(reg, A64_SP, bargs_off), ctx);
}
reg++;
bargs_off += 8;
}
}
@@ -2630,9 +2820,11 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
* (FP/LR) frames, so the arguments start at FP + 32. A struct_ops
* callback is called indirectly and only the FP/LR frame is saved, so
* they start at FP + 16.
*
* The outgoing area mirrors the incoming one, hole and all; only the
* bpf program takes the arguments packed.
*/
soff = is_struct_ops ? 16 : 32;
doff = (for_call_origin ? oargs_off : bargs_off);
/* save on stack arguments */
for (i = a->args_in_regs; i < m->nr_args; i++) {
@@ -2642,7 +2834,9 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
slots = (m->arg_size[i] + 7) / 8;
/* verifier ensures arg_size <= 16, so slots equals 1 or 2 */
while (slots-- > 0) {
emit(A64_LDR64I(tmp, A64_FP, soff), ctx);
int off = a64_arg_stack_off(a->pos_of_slot[slot++]);
emit(A64_LDR64I(tmp, A64_FP, soff + off), ctx);
/* if there is unused space in the last slot, clear
* the garbage contained in the space.
*/
@@ -2657,19 +2851,21 @@ static void save_args(struct jit_ctx *ctx, int bargs_off, int oargs_off,
*/
if (arena_arg)
emit_arena_arg_conv(ctx, tmp, tmp, nullable, base_lo);
emit(A64_STR64I(tmp, A64_SP, doff), ctx);
soff += 8;
doff += 8;
if (for_call_origin)
emit(A64_STR64I(tmp, A64_SP, oargs_off + off), ctx);
else
emit(A64_STR64I(tmp, A64_SP, bargs_off), ctx);
bargs_off += 8;
}
}
}
static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs)
static void restore_args(struct jit_ctx *ctx, int bargs_off, const struct arg_aux *a)
{
int reg;
int slot;
for (reg = 0; reg < nregs; reg++) {
emit(A64_LDR64I(reg, A64_SP, bargs_off), ctx);
for (slot = 0; slot < a->regs_for_args; slot++) {
emit(A64_LDR64I(a->pos_of_slot[slot], A64_SP, bargs_off), ctx);
bargs_off += 8;
}
}
@@ -2937,7 +3133,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im,
}
if (flags & BPF_TRAMP_F_RESTORE_REGS)
restore_args(ctx, bargs_off, a->regs_for_args);
restore_args(ctx, bargs_off, a);
/* restore callee saved register x19 and x20 */
emit(A64_LDR64I(A64_R(19), A64_SP, regs_off), ctx);
@@ -3226,21 +3422,22 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
*/
plt_target = (u64)&dummy_tramp;
if (plt_target) {
/* non-zero plt_target indicates we're patching a bpf prog,
* which is read only.
*/
if (set_memory_rw(PAGE_MASK & ((uintptr_t)&plt->target), 1))
return -EFAULT;
WRITE_ONCE(plt->target, plt_target);
set_memory_ro(PAGE_MASK & ((uintptr_t)&plt->target), 1);
/* since plt target points to either the new trampoline
* or dummy_tramp, even if another CPU reads the old plt
* target value before fetching the bl instruction to plt,
* it will be brought back by dummy_tramp, so no barrier is
* required here.
*/
}
/*
* non-zero plt_target indicates we're patching a bpf prog,
* which is read only. The prog may share its page in the bpf
* prog pack with other progs, so write the aligned target
* through the text patching fixmap without flipping the
* page permissions to avoid racing with concurrent pokers.
*
* since plt target points to either the new trampoline
* or dummy_tramp, even if another CPU reads the old plt
* target value before fetching the bl instruction to plt,
* it will be brought back by dummy_tramp, so no barrier is
* required here.
*/
if (plt_target &&
aarch64_insn_write_literal_u64(&plt->target, plt_target))
return -EFAULT;
/* if the old target and the new target are both long jumps, no
* patching is required
@@ -3305,6 +3502,11 @@ bool bpf_jit_supports_arena(void)
return true;
}
bool bpf_jit_supports_arena_scalar(void)
{
return true;
}
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
{
if (!in_arena)
arch/hexagon/include/asm/cmpxchg.h
+6
View File
@@ -8,6 +8,8 @@
#ifndef _ASM_CMPXCHG_H
#define _ASM_CMPXCHG_H
#include <asm-generic/cmpxchg-local.h>
/*
* __arch_xchg - atomically exchange a register and a memory location
* @x: value to swap
@@ -72,4 +74,8 @@ __arch_xchg(unsigned long x, volatile void *ptr, int size)
__oldval; \
})
#define arch_cmpxchg_local(ptr, o, n) \
((__typeof__(*(ptr)))__generic_cmpxchg_local((ptr), \
(unsigned long)(o), (unsigned long)(n), sizeof(*(ptr))))
#endif /* _ASM_CMPXCHG_H */
arch/loongarch/net/bpf_jit.c
+4 -2
View File
@@ -2245,8 +2245,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
memset(&ctx, 0, sizeof(ctx));
ctx.prog = prog;
ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
if (bpf_prog_uses_arena_insns(prog)) {
ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
}
ctx.offset = kvcalloc(prog->len + 1, sizeof(u32), GFP_KERNEL);
if (ctx.offset == NULL)
arch/m68k/configs/amiga_defconfig
-1
View File
@@ -579,7 +579,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/apollo_defconfig
-1
View File
@@ -533,7 +533,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/atari_defconfig
-1
View File
@@ -558,7 +558,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/bvme6000_defconfig
-1
View File
@@ -525,7 +525,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/hp300_defconfig
-1
View File
@@ -535,7 +535,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/mac_defconfig
-1
View File
@@ -554,7 +554,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/multi_defconfig
-1
View File
@@ -644,7 +644,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/mvme147_defconfig
-1
View File
@@ -525,7 +525,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/mvme16x_defconfig
-1
View File
@@ -526,7 +526,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/q40_defconfig
-1
View File
@@ -544,7 +544,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/sun3_defconfig
-1
View File
@@ -522,7 +522,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/m68k/configs/sun3x_defconfig
-1
View File
@@ -523,7 +523,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_WORKQUEUE=m
CONFIG_TEST_BPF=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_REGION_ALLOC_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/mips/include/asm/uasm.h
+8
View File
@@ -85,15 +85,20 @@ Ip_u1u2(_ctc1);
Ip_u2u1(_ctcmsa);
Ip_u2u1s3(_daddiu);
Ip_u3u1u2(_daddu);
Ip_u1u2(_ddiv);
Ip_u3u1u2(_ddiv_r6);
Ip_u1u2(_ddivu);
Ip_u3u1u2(_ddivu_r6);
Ip_u1(_di);
Ip_u2u1msbu3(_dins);
Ip_u2u1msbu3(_dinsm);
Ip_u2u1msbu3(_dinsu);
Ip_u1u2(_div);
Ip_u3u1u2(_div_r6);
Ip_u1u2(_divu);
Ip_u3u1u2(_divu_r6);
Ip_u1u2u3(_dmfc0);
Ip_u3u1u2(_dmod);
Ip_u3u1u2(_dmodu);
Ip_u1u2u3(_dmtc0);
Ip_u1u2(_dmultu);
@@ -135,6 +140,7 @@ Ip_u1u2u3(_mfc0);
Ip_u1u2u3(_mfhc0);
Ip_u1(_mfhi);
Ip_u1(_mflo);
Ip_u3u1u2(_mod);
Ip_u3u1u2(_modu);
Ip_u3u1u2(_movn);
Ip_u3u1u2(_movz);
@@ -156,6 +162,8 @@ Ip_u2s3u1(_sb);
Ip_u2s3u1(_sc);
Ip_u2s3u1(_scd);
Ip_u2s3u1(_sd);
Ip_u2u1(_seb);
Ip_u2u1(_seh);
Ip_u3u1u2(_seleqz);
Ip_u3u1u2(_selnez);
Ip_u2s3u1(_sh);
arch/mips/mm/uasm-mips.c
+12
View File
@@ -75,6 +75,9 @@ static const struct insn insn_table[insn_invalid] = {
[insn_ctcmsa] = {M(msa_op, 0, msa_ctc_op, 0, 0, msa_elm_op), RD | RE},
[insn_daddiu] = {M(daddiu_op, 0, 0, 0, 0, 0), RS | RT | SIMM},
[insn_daddu] = {M(spec_op, 0, 0, 0, 0, daddu_op), RS | RT | RD},
[insn_ddiv] = {M(spec_op, 0, 0, 0, 0, ddiv_op), RS | RT},
[insn_ddiv_r6] = {M(spec_op, 0, 0, 0, ddiv_ddiv6_op, ddiv_op),
RS | RT | RD},
[insn_ddivu] = {M(spec_op, 0, 0, 0, 0, ddivu_op), RS | RT},
[insn_ddivu_r6] = {M(spec_op, 0, 0, 0, ddivu_ddivu6_op, ddivu_op),
RS | RT | RD},
@@ -82,10 +85,15 @@ static const struct insn insn_table[insn_invalid] = {
[insn_dins] = {M(spec3_op, 0, 0, 0, 0, dins_op), RS | RT | RD | RE},
[insn_dinsm] = {M(spec3_op, 0, 0, 0, 0, dinsm_op), RS | RT | RD | RE},
[insn_dinsu] = {M(spec3_op, 0, 0, 0, 0, dinsu_op), RS | RT | RD | RE},
[insn_div] = {M(spec_op, 0, 0, 0, 0, div_op), RS | RT},
[insn_div_r6] = {M(spec_op, 0, 0, 0, div_div6_op, div_op),
RS | RT | RD},
[insn_divu] = {M(spec_op, 0, 0, 0, 0, divu_op), RS | RT},
[insn_divu_r6] = {M(spec_op, 0, 0, 0, divu_divu6_op, divu_op),
RS | RT | RD},
[insn_dmfc0] = {M(cop0_op, dmfc_op, 0, 0, 0, 0), RT | RD | SET},
[insn_dmod] = {M(spec_op, 0, 0, 0, ddiv_dmod_op, ddiv_op),
RS | RT | RD},
[insn_dmodu] = {M(spec_op, 0, 0, 0, ddivu_dmodu_op, ddivu_op),
RS | RT | RD},
[insn_dmtc0] = {M(cop0_op, dmtc_op, 0, 0, 0, 0), RT | RD | SET},
@@ -140,6 +148,8 @@ static const struct insn insn_table[insn_invalid] = {
[insn_mfhc0] = {M(cop0_op, mfhc0_op, 0, 0, 0, 0), RT | RD | SET},
[insn_mfhi] = {M(spec_op, 0, 0, 0, 0, mfhi_op), RD},
[insn_mflo] = {M(spec_op, 0, 0, 0, 0, mflo_op), RD},
[insn_mod] = {M(spec_op, 0, 0, 0, div_mod_op, div_op),
RS | RT | RD},
[insn_modu] = {M(spec_op, 0, 0, 0, divu_modu_op, divu_op),
RS | RT | RD},
[insn_movn] = {M(spec_op, 0, 0, 0, 0, movn_op), RS | RT | RD},
@@ -177,6 +187,8 @@ static const struct insn insn_table[insn_invalid] = {
[insn_scd] = {M6(spec3_op, 0, 0, 0, scd6_op), RS | RT | SIMM9},
#endif
[insn_sd] = {M(sd_op, 0, 0, 0, 0, 0), RS | RT | SIMM},
[insn_seb] = {M(spec3_op, 0, 0, 0, seb_op, bshfl_op), RT | RD},
[insn_seh] = {M(spec3_op, 0, 0, 0, seh_op, bshfl_op), RT | RD},
[insn_seleqz] = {M(spec_op, 0, 0, 0, 0, seleqz_op), RS | RT | RD},
[insn_selnez] = {M(spec_op, 0, 0, 0, 0, selnez_op), RS | RT | RD},
[insn_sh] = {M(sh_op, 0, 0, 0, 0, 0), RS | RT | SIMM},
arch/mips/mm/uasm.c
+17 -7
View File
@@ -49,19 +49,21 @@ enum opcode {
insn_addiu, insn_addu, insn_and, insn_andi, insn_bbit0, insn_bbit1,
insn_beq, insn_beql, insn_bgez, insn_bgezl, insn_bgtz, insn_blez,
insn_bltz, insn_bltzl, insn_bne, insn_break, insn_cache, insn_cfc1,
insn_cfcmsa, insn_ctc1, insn_ctcmsa, insn_daddiu, insn_daddu, insn_ddivu,
insn_ddivu_r6, insn_di, insn_dins, insn_dinsm, insn_dinsu, insn_divu,
insn_divu_r6, insn_dmfc0, insn_dmodu, insn_dmtc0, insn_dmultu,
insn_cfcmsa, insn_ctc1, insn_ctcmsa, insn_daddiu, insn_daddu, insn_ddiv,
insn_ddiv_r6, insn_ddivu, insn_ddivu_r6, insn_di, insn_dins, insn_dinsm,
insn_dinsu, insn_div, insn_div_r6, insn_divu, insn_divu_r6, insn_dmfc0,
insn_dmod, insn_dmodu, insn_dmtc0, insn_dmultu,
insn_dmulu, insn_drotr, insn_drotr32, insn_dsbh, insn_dshd, insn_dsll,
insn_dsll32, insn_dsllv, insn_dsra, insn_dsra32, insn_dsrav, insn_dsrl,
insn_dsrl32, insn_dsrlv, insn_dsubu, insn_eret, insn_ext, insn_ins,
insn_j, insn_jal, insn_jalr, insn_jr, insn_lb, insn_lbu, insn_ld,
insn_lddir, insn_ldpte, insn_ldx, insn_lh, insn_lhu, insn_ll, insn_lld,
insn_lui, insn_lw, insn_lwu, insn_lwx, insn_mfc0, insn_mfhc0, insn_mfhi,
insn_mflo, insn_modu, insn_movn, insn_movz, insn_mtc0, insn_mthc0,
insn_mthi, insn_mtlo, insn_mul, insn_multu, insn_mulu, insn_muhu, insn_nor,
insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr, insn_sb, insn_sc,
insn_scd, insn_seleqz, insn_selnez, insn_sd, insn_sh, insn_sll,
insn_mflo, insn_mod, insn_modu, insn_movn, insn_movz, insn_mtc0,
insn_mthc0, insn_mthi, insn_mtlo, insn_mul, insn_multu, insn_mulu,
insn_muhu, insn_nor, insn_or, insn_ori, insn_pref, insn_rfe, insn_rotr,
insn_sb, insn_sc, insn_scd, insn_seb, insn_seh,
insn_seleqz, insn_selnez, insn_sd, insn_sh, insn_sll,
insn_sllv, insn_slt, insn_slti, insn_sltiu, insn_sltu, insn_sra,
insn_srav, insn_srl, insn_srlv, insn_subu, insn_sw, insn_sync,
insn_syscall, insn_tlbp, insn_tlbr, insn_tlbwi, insn_tlbwr, insn_wait,
@@ -287,9 +289,12 @@ I_u1u2(_cfc1)
I_u2u1(_cfcmsa)
I_u1u2(_ctc1)
I_u2u1(_ctcmsa)
I_u1u2(_ddiv)
I_u3u1u2(_ddiv_r6)
I_u1u2(_ddivu)
I_u3u1u2(_ddivu_r6)
I_u1u2u3(_dmfc0)
I_u3u1u2(_dmod)
I_u3u1u2(_dmodu)
I_u1u2u3(_dmtc0)
I_u1u2(_dmultu)
@@ -297,6 +302,8 @@ I_u3u1u2(_dmulu)
I_u2u1s3(_daddiu)
I_u3u1u2(_daddu)
I_u1(_di);
I_u1u2(_div)
I_u3u1u2(_div_r6)
I_u1u2(_divu)
I_u3u1u2(_divu_r6)
I_u2u1(_dsbh);
@@ -332,6 +339,7 @@ I_u2s3u1(_lw)
I_u2s3u1(_lwu)
I_u1u2u3(_mfc0)
I_u1u2u3(_mfhc0)
I_u3u1u2(_mod)
I_u3u1u2(_modu)
I_u3u1u2(_movn)
I_u3u1u2(_movz)
@@ -353,6 +361,8 @@ I_u2s3u1(_sb)
I_u2s3u1(_sc)
I_u2s3u1(_scd)
I_u2s3u1(_sd)
I_u2u1(_seb)
I_u2u1(_seh)
I_u3u1u2(_seleqz)
I_u3u1u2(_selnez)
I_u2s3u1(_sh)
arch/mips/net/bpf_jit_comp.c
+42 -15
View File
@@ -208,7 +208,7 @@ void emit_mov_r(struct jit_context *ctx, u8 dst, u8 src)
}
/* Validate ALU immediate range */
bool valid_alu_i(u8 op, s32 imm)
bool valid_alu_i(u8 op, s32 imm, s16 off)
{
switch (BPF_OP(op)) {
case BPF_NEG:
@@ -237,6 +237,9 @@ bool valid_alu_i(u8 op, s32 imm)
return imm == 0 || (imm > 0 && is_power_of_2(imm));
case BPF_DIV:
case BPF_MOD:
/* Do not use unsigned shift/mask rewrites for signed div/mod. */
if (off == 1)
return false;
/* imm must be an 17-bit power of two */
return (u32)imm <= 0x10000 && is_power_of_2((u32)imm);
}
@@ -338,8 +341,42 @@ void emit_alu_i(struct jit_context *ctx, u8 dst, s32 imm, u8 op)
clobber_reg(ctx, dst);
}
/* ALU division operation (32-bit) */
static void emit_div(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
if (cpu_has_mips32r6) {
if (off == 1)
emit(ctx, div_r6, dst, dst, src);
else
emit(ctx, divu_r6, dst, dst, src);
} else {
if (off == 1)
emit(ctx, div, dst, src);
else
emit(ctx, divu, dst, src);
emit(ctx, mflo, dst);
}
}
/* ALU modulo operation (32-bit) */
static void emit_mod(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
if (cpu_has_mips32r6) {
if (off == 1)
emit(ctx, mod, dst, dst, src);
else
emit(ctx, modu, dst, dst, src);
} else {
if (off == 1)
emit(ctx, div, dst, src);
else
emit(ctx, divu, dst, src);
emit(ctx, mfhi, dst);
}
}
/* ALU register operation (32-bit) */
void emit_alu_r(struct jit_context *ctx, u8 dst, u8 src, u8 op)
void emit_alu_r(struct jit_context *ctx, u8 dst, u8 src, u8 op, s16 off)
{
switch (BPF_OP(op)) {
/* dst = dst & src */
@@ -385,21 +422,11 @@ void emit_alu_r(struct jit_context *ctx, u8 dst, u8 src, u8 op)
break;
/* dst = dst / src */
case BPF_DIV:
if (cpu_has_mips32r6) {
emit(ctx, divu_r6, dst, dst, src);
} else {
emit(ctx, divu, dst, src);
emit(ctx, mflo, dst);
}
emit_div(ctx, dst, src, off);
break;
/* dst = dst % src */
case BPF_MOD:
if (cpu_has_mips32r6) {
emit(ctx, modu, dst, dst, src);
} else {
emit(ctx, divu, dst, src);
emit(ctx, mfhi, dst);
}
emit_mod(ctx, dst, src, off);
break;
}
clobber_reg(ctx, dst);
@@ -823,7 +850,7 @@ void emit_jmp_r(struct jit_context *ctx, u8 dst, u8 src, s32 off, u8 op)
}
/* Jump always */
int emit_ja(struct jit_context *ctx, s16 off)
int emit_ja(struct jit_context *ctx, s32 off)
{
int target = get_target(ctx, ctx->bpf_index + off + 1);
arch/mips/net/bpf_jit_comp.h
+3 -3
View File
@@ -163,7 +163,7 @@ void emit_mov_i(struct jit_context *ctx, u8 dst, s32 imm);
void emit_mov_r(struct jit_context *ctx, u8 dst, u8 src);
/* Validate ALU/ALU64 immediate range */
bool valid_alu_i(u8 op, s32 imm);
bool valid_alu_i(u8 op, s32 imm, s16 off);
/* Rewrite ALU/ALU64 immediate operation */
bool rewrite_alu_i(u8 op, s32 imm, u8 *alu, s32 *val);
@@ -172,7 +172,7 @@ bool rewrite_alu_i(u8 op, s32 imm, u8 *alu, s32 *val);
void emit_alu_i(struct jit_context *ctx, u8 dst, s32 imm, u8 op);
/* ALU register operation (32-bit) */
void emit_alu_r(struct jit_context *ctx, u8 dst, u8 src, u8 op);
void emit_alu_r(struct jit_context *ctx, u8 dst, u8 src, u8 op, s16 off);
/* Atomic read-modify-write (32-bit) */
void emit_atomic_r(struct jit_context *ctx, u8 dst, u8 src, s16 off, u8 code);
@@ -204,7 +204,7 @@ void emit_jmp_i(struct jit_context *ctx, u8 dst, s32 imm, s32 off, u8 op);
void emit_jmp_r(struct jit_context *ctx, u8 dst, u8 src, s32 off, u8 op);
/* Jump always */
int emit_ja(struct jit_context *ctx, s16 off);
int emit_ja(struct jit_context *ctx, s32 off);
/* Jump to epilogue */
int emit_exit(struct jit_context *ctx);
arch/mips/net/bpf_jit_comp32.c
+95 -14
View File
@@ -190,6 +190,69 @@ static void emit_zext_ver(struct jit_context *ctx, const u8 dst[])
}
}
/* Sign-extend an 8- or 16-bit field into a native register. */
static void emit_movsx_r(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
int shift;
if (cpu_has_mips32r2 || cpu_has_mips32r6) {
if (off == 8)
emit(ctx, seb, dst, src);
else
emit(ctx, seh, dst, src);
} else {
shift = 32 - off;
emit(ctx, sll, dst, src, shift);
emit(ctx, sra, dst, dst, shift);
}
}
/* Register move operation (32-bit), optionally with sign extension */
static void emit_mov_r32(struct jit_context *ctx, const u8 dst[],
const u8 src[], s16 off)
{
switch (off) {
case 8:
case 16:
emit_movsx_r(ctx, lo(dst), lo(src), off);
break;
default:
/* off == 0 is MOV; the verifier rejects other offsets. */
emit_mov_r(ctx, lo(dst), lo(src));
break;
}
clobber_reg(ctx, lo(dst));
emit_zext_ver(ctx, dst);
}
/* Register move operation (64-bit), optionally with sign extension */
static void emit_mov_r64(struct jit_context *ctx, const u8 dst[],
const u8 src[], s16 off)
{
switch (off) {
case 8:
case 16:
emit_movsx_r(ctx, lo(dst), lo(src), off);
emit(ctx, sra, hi(dst), lo(dst), 31);
break;
case 32:
emit(ctx, move, lo(dst), lo(src));
emit(ctx, sra, hi(dst), lo(dst), 31);
break;
default:
/*
* off == 0 is ordinary MOV. The verifier rejects other
* offsets; defined exceptions require
* bpf_jit_supports_percpu_insn() or bpf_jit_supports_arena(),
* neither implemented by MIPS.
*/
emit_mov_r(ctx, lo(dst), lo(src));
emit_mov_r(ctx, hi(dst), hi(src));
break;
}
clobber_reg64(ctx, dst);
}
/* Load delay slot, if ISA mandates it */
static void emit_load_delay(struct jit_context *ctx)
{
@@ -512,7 +575,7 @@ static void emit_mul_r64(struct jit_context *ctx,
clobber_reg64(ctx, dst);
}
/* Helper function for 64-bit modulo */
/* Helper function for unsigned 64-bit modulo */
static u64 jit_mod64(u64 a, u64 b)
{
u64 rem;
@@ -521,13 +584,22 @@ static u64 jit_mod64(u64 a, u64 b)
return rem;
}
/* Helper function for signed 64-bit modulo */
static s64 jit_smod64(s64 a, s64 b)
{
s64 quot = div64_s64(a, b);
return a - quot * b;
}
/* ALU div/mod register (64-bit) */
static void emit_divmod_r64(struct jit_context *ctx,
const u8 dst[], const u8 src[], u8 op)
static void emit_divmod_r64(struct jit_context *ctx, const u8 dst[],
const u8 src[], u8 op, s16 off)
{
const u8 *r0 = bpf2mips32[BPF_REG_0]; /* Mapped to v0-v1 */
const u8 *r1 = bpf2mips32[BPF_REG_1]; /* Mapped to a0-a1 */
const u8 *r2 = bpf2mips32[BPF_REG_2]; /* Mapped to a2-a3 */
bool is_signed = off == 1;
int exclude, k;
u32 addr = 0;
@@ -546,11 +618,11 @@ static void emit_divmod_r64(struct jit_context *ctx,
switch (BPF_OP(op)) {
/* dst = dst / src */
case BPF_DIV:
addr = (u32)&div64_u64;
addr = is_signed ? (u32)&div64_s64 : (u32)&div64_u64;
break;
/* dst = dst % src */
case BPF_MOD:
addr = (u32)&jit_mod64;
addr = is_signed ? (u32)&jit_smod64 : (u32)&jit_mod64;
break;
}
emit_mov_i(ctx, MIPS_R_T9, addr);
@@ -1485,8 +1557,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
/* Special mov32 for zext */
emit_mov_i(ctx, hi(dst), 0);
} else {
emit_mov_r(ctx, lo(dst), lo(src));
emit_zext_ver(ctx, dst);
emit_mov_r32(ctx, dst, src, off);
}
break;
/* dst = -dst */
@@ -1516,9 +1587,9 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU | BPF_MUL | BPF_K:
case BPF_ALU | BPF_DIV | BPF_K:
case BPF_ALU | BPF_MOD | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
if (!valid_alu_i(BPF_OP(code), imm, off)) {
emit_mov_i(ctx, MIPS_R_T6, imm);
emit_alu_r(ctx, lo(dst), MIPS_R_T6, BPF_OP(code));
emit_alu_r(ctx, lo(dst), MIPS_R_T6, BPF_OP(code), off);
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_alu_i(ctx, lo(dst), val, alu);
}
@@ -1546,7 +1617,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU | BPF_MUL | BPF_X:
case BPF_ALU | BPF_DIV | BPF_X:
case BPF_ALU | BPF_MOD | BPF_X:
emit_alu_r(ctx, lo(dst), lo(src), BPF_OP(code));
emit_alu_r(ctx, lo(dst), lo(src), BPF_OP(code), off);
emit_zext_ver(ctx, dst);
break;
/* dst = imm (64-bit) */
@@ -1555,8 +1626,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
break;
/* dst = src (64-bit) */
case BPF_ALU64 | BPF_MOV | BPF_X:
emit_mov_r(ctx, lo(dst), lo(src));
emit_mov_r(ctx, hi(dst), hi(src));
emit_mov_r64(ctx, dst, src, off);
break;
/* dst = -dst (64-bit) */
case BPF_ALU64 | BPF_NEG:
@@ -1599,7 +1669,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
* and then do the operation on this register.
*/
emit_mov_se_i64(ctx, tmp, imm);
emit_divmod_r64(ctx, dst, tmp, BPF_OP(code));
emit_divmod_r64(ctx, dst, tmp, BPF_OP(code), off);
break;
/* dst = dst & src (64-bit) */
/* dst = dst | src (64-bit) */
@@ -1629,7 +1699,11 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
/* dst = dst % src (64-bit) */
case BPF_ALU64 | BPF_DIV | BPF_X:
case BPF_ALU64 | BPF_MOD | BPF_X:
emit_divmod_r64(ctx, dst, src, BPF_OP(code));
emit_divmod_r64(ctx, dst, src, BPF_OP(code), off);
break;
/* dst = bswap(dst) */
case BPF_ALU64 | BPF_END:
emit_bswap_r64(ctx, dst, imm);
break;
/* dst = htole(dst) */
/* dst = htobe(dst) */
@@ -1803,6 +1877,13 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += imm */
case BPF_JMP32 | BPF_JA:
if (imm == 0)
break;
if (emit_ja(ctx, imm) < 0)
goto toofar;
break;
/* PC += off if dst == src */
/* PC += off if dst != src */
/* PC += off if dst & src */
arch/mips/net/bpf_jit_comp64.c
+113 -26
View File
@@ -120,6 +120,58 @@ static void emit_zext_ver(struct jit_context *ctx, u8 dst)
emit_zext(ctx, dst);
}
/* Sign-extend an 8- or 16-bit field into a native register. */
static void emit_movsx_r(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
/* seb/seh would need a sign-extended 32-bit source first. */
int shift = 32 - off;
/* dsll32 and dsra32 add 32 to the shift argument. */
emit(ctx, dsll32, dst, src, shift);
emit(ctx, dsra32, dst, dst, shift);
}
/* Register move operation (32-bit), optionally with sign extension */
static void emit_mov_r32(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
switch (off) {
case 8:
case 16:
emit_movsx_r(ctx, dst, src, off);
break;
default:
/* off == 0 is MOV; the verifier rejects other offsets. */
emit_mov_r(ctx, dst, src);
break;
}
clobber_reg(ctx, dst);
emit_zext_ver(ctx, dst);
}
/* Register move operation (64-bit), optionally with sign extension */
static void emit_mov_r64(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
switch (off) {
case 8:
case 16:
emit_movsx_r(ctx, dst, src, off);
break;
case 32:
emit_sext(ctx, dst, src);
break;
default:
/*
* off == 0 is ordinary MOV. The verifier rejects other
* offsets; defined exceptions require
* bpf_jit_supports_percpu_insn() or bpf_jit_supports_arena(),
* neither implemented by MIPS.
*/
emit_mov_r(ctx, dst, src);
break;
}
clobber_reg(ctx, dst);
}
/* dst = imm (64-bit) */
static void emit_mov_i64(struct jit_context *ctx, u8 dst, u64 imm64)
{
@@ -197,8 +249,43 @@ static void emit_alu_i64(struct jit_context *ctx, u8 dst, s32 imm, u8 op)
clobber_reg(ctx, dst);
}
/* ALU division operation (64-bit) */
static void emit_div64(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
if (cpu_has_mips64r6) {
if (off == 1)
emit(ctx, ddiv_r6, dst, dst, src);
else
emit(ctx, ddivu_r6, dst, dst, src);
} else {
if (off == 1)
emit(ctx, ddiv, dst, src);
else
emit(ctx, ddivu, dst, src);
emit(ctx, mflo, dst);
}
}
/* ALU modulo operation (64-bit) */
static void emit_mod64(struct jit_context *ctx, u8 dst, u8 src, s16 off)
{
if (cpu_has_mips64r6) {
if (off == 1)
emit(ctx, dmod, dst, dst, src);
else
emit(ctx, dmodu, dst, dst, src);
} else {
if (off == 1)
emit(ctx, ddiv, dst, src);
else
emit(ctx, ddivu, dst, src);
emit(ctx, mfhi, dst);
}
}
/* ALU register operation (64-bit) */
static void emit_alu_r64(struct jit_context *ctx, u8 dst, u8 src, u8 op)
static void emit_alu_r64(struct jit_context *ctx, u8 dst, u8 src, u8 op,
s16 off)
{
switch (BPF_OP(op)) {
/* dst = dst << src */
@@ -235,25 +322,15 @@ static void emit_alu_r64(struct jit_context *ctx, u8 dst, u8 src, u8 op)
break;
/* dst = dst / src */
case BPF_DIV:
if (cpu_has_mips64r6) {
emit(ctx, ddivu_r6, dst, dst, src);
} else {
emit(ctx, ddivu, dst, src);
emit(ctx, mflo, dst);
}
emit_div64(ctx, dst, src, off);
break;
/* dst = dst % src */
case BPF_MOD:
if (cpu_has_mips64r6) {
emit(ctx, dmodu, dst, dst, src);
} else {
emit(ctx, ddivu, dst, src);
emit(ctx, mfhi, dst);
}
emit_mod64(ctx, dst, src, off);
break;
default:
/* Width-generic operations */
emit_alu_r(ctx, dst, src, op);
emit_alu_r(ctx, dst, src, op, off);
}
clobber_reg(ctx, dst);
}
@@ -655,8 +732,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
/* Special mov32 for zext */
emit_zext(ctx, dst);
} else {
emit_mov_r(ctx, dst, src);
emit_zext_ver(ctx, dst);
emit_mov_r32(ctx, dst, src, off);
}
break;
/* dst = -dst */
@@ -673,9 +749,9 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU | BPF_AND | BPF_K:
case BPF_ALU | BPF_XOR | BPF_K:
case BPF_ALU | BPF_LSH | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
if (!valid_alu_i(BPF_OP(code), imm, off)) {
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code));
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code), off);
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_alu_i(ctx, dst, val, alu);
}
@@ -695,10 +771,10 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU | BPF_MUL | BPF_K:
case BPF_ALU | BPF_DIV | BPF_K:
case BPF_ALU | BPF_MOD | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
if (!valid_alu_i(BPF_OP(code), imm, off)) {
emit_sext(ctx, dst, dst);
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code));
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code), off);
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_sext(ctx, dst, dst);
emit_alu_i(ctx, dst, val, alu);
@@ -713,7 +789,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU | BPF_OR | BPF_X:
case BPF_ALU | BPF_XOR | BPF_X:
case BPF_ALU | BPF_LSH | BPF_X:
emit_alu_r(ctx, dst, src, BPF_OP(code));
emit_alu_r(ctx, dst, src, BPF_OP(code), off);
emit_zext_ver(ctx, dst);
break;
/* dst = dst >> src */
@@ -732,7 +808,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU | BPF_MOD | BPF_X:
emit_sext(ctx, dst, dst);
emit_sext(ctx, MIPS_R_T4, src);
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code));
emit_alu_r(ctx, dst, MIPS_R_T4, BPF_OP(code), off);
emit_zext_ver(ctx, dst);
break;
/* dst = imm (64-bit) */
@@ -741,7 +817,7 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
break;
/* dst = src (64-bit) */
case BPF_ALU64 | BPF_MOV | BPF_X:
emit_mov_r(ctx, dst, src);
emit_mov_r64(ctx, dst, src, off);
break;
/* dst = -dst (64-bit) */
case BPF_ALU64 | BPF_NEG:
@@ -769,9 +845,9 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU64 | BPF_MUL | BPF_K:
case BPF_ALU64 | BPF_DIV | BPF_K:
case BPF_ALU64 | BPF_MOD | BPF_K:
if (!valid_alu_i(BPF_OP(code), imm)) {
if (!valid_alu_i(BPF_OP(code), imm, off)) {
emit_mov_i(ctx, MIPS_R_T4, imm);
emit_alu_r64(ctx, dst, MIPS_R_T4, BPF_OP(code));
emit_alu_r64(ctx, dst, MIPS_R_T4, BPF_OP(code), off);
} else if (rewrite_alu_i(BPF_OP(code), imm, &alu, &val)) {
emit_alu_i64(ctx, dst, val, alu);
}
@@ -798,7 +874,11 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
case BPF_ALU64 | BPF_MUL | BPF_X:
case BPF_ALU64 | BPF_DIV | BPF_X:
case BPF_ALU64 | BPF_MOD | BPF_X:
emit_alu_r64(ctx, dst, src, BPF_OP(code));
emit_alu_r64(ctx, dst, src, BPF_OP(code), off);
break;
/* dst = bswap(dst) */
case BPF_ALU64 | BPF_END:
emit_bswap_r64(ctx, dst, imm);
break;
/* dst = htole(dst) */
/* dst = htobe(dst) */
@@ -961,6 +1041,13 @@ int build_insn(const struct bpf_insn *insn, struct jit_context *ctx)
if (finish_jmp(ctx, jmp, off) < 0)
goto toofar;
break;
/* PC += imm */
case BPF_JMP32 | BPF_JA:
if (imm == 0)
break;
if (emit_ja(ctx, imm) < 0)
goto toofar;
break;
/* PC += off if dst == src */
/* PC += off if dst != src */
/* PC += off if dst & src */
arch/openrisc/include/asm/cmpxchg.h
+2
View File
@@ -140,6 +140,8 @@ static inline unsigned long __cmpxchg(volatile void *ptr, unsigned long old,
sizeof(*(ptr))); \
})
#define arch_cmpxchg_local arch_cmpxchg
/*
* This function doesn't exist, so you'll get a linker error if
* something tries to do an invalidly-sized xchg().
arch/parisc/net/bpf_jit.h
+2
View File
@@ -467,6 +467,8 @@ static inline u32 hppa_t21_insn(u8 opcode, u8 r2, u8 r1, u8 ext8, u8 t)
u64 hppa_div64(u64 div, u64 divisor);
u64 hppa_div64_rem(u64 div, u64 divisor);
u64 hppa_sdiv64(u64 div, u64 divisor);
u64 hppa_sdiv64_rem(u64 div, u64 divisor);
/* Helper functions that emit HPPA instructions when possible. */
arch/parisc/net/bpf_jit_comp32.c
+19 -10
View File
@@ -321,6 +321,8 @@ static void bpf_put_reg32(const s8 *reg, const s8 *src,
extern void $$mulI(void);
extern void $$divU(void);
extern void $$remU(void);
extern void $$divI(void);
extern void $$remI(void);
static void emit_call_millicode(void *func, const s8 arg0,
const s8 arg1, u8 opcode, struct hppa_jit_context *ctx)
@@ -562,12 +564,14 @@ static void emit_alu_i32(const s8 *dst, s32 imm,
}
static void emit_alu_r64(const s8 *dst, const s8 *src,
struct hppa_jit_context *ctx, const u8 op)
struct hppa_jit_context *ctx, const u8 op,
bool is_signed)
{
const s8 *tmp1 = regmap[TMP_REG_1];
const s8 *tmp2 = regmap[TMP_REG_2];
const s8 *rd;
const s8 *rs = bpf_get_reg64(src, tmp2, ctx);
void *func;
if (op == BPF_MOV)
rd = bpf_get_reg64_ref(dst, tmp1, false, ctx);
@@ -604,10 +608,12 @@ static void emit_alu_r64(const s8 *dst, const s8 *src,
emit_call_libgcc_ll(__muldi3, rd, rs, op, ctx);
break;
case BPF_DIV:
emit_call_libgcc_ll(&hppa_div64, rd, rs, op, ctx);
func = is_signed ? &hppa_sdiv64 : &hppa_div64;
emit_call_libgcc_ll(func, rd, rs, op, ctx);
break;
case BPF_MOD:
emit_call_libgcc_ll(&hppa_div64_rem, rd, rs, op, ctx);
func = is_signed ? &hppa_sdiv64_rem : &hppa_div64_rem;
emit_call_libgcc_ll(func, rd, rs, op, ctx);
break;
case BPF_LSH:
emit_call_libgcc_ll(__ashldi3, rd, rs, op, ctx);
@@ -630,7 +636,8 @@ static void emit_alu_r64(const s8 *dst, const s8 *src,
}
static void emit_alu_r32(const s8 *dst, const s8 *src,
struct hppa_jit_context *ctx, const u8 op)
struct hppa_jit_context *ctx, const u8 op,
bool is_signed)
{
const s8 *tmp1 = regmap[TMP_REG_1];
const s8 *tmp2 = regmap[TMP_REG_2];
@@ -666,10 +673,12 @@ static void emit_alu_r32(const s8 *dst, const s8 *src,
emit_call_millicode($$mulI, lo(rd), lo(rs), op, ctx);
break;
case BPF_DIV:
emit_call_millicode($$divU, lo(rd), lo(rs), op, ctx);
emit_call_millicode(is_signed ? $$divI : $$divU,
lo(rd), lo(rs), op, ctx);
break;
case BPF_MOD:
emit_call_millicode($$remU, lo(rd), lo(rs), op, ctx);
emit_call_millicode(is_signed ? $$remI : $$remU,
lo(rd), lo(rs), op, ctx);
break;
case BPF_LSH:
emit(hppa_subi(0x1f, lo(rs), HPPA_REG_T0), ctx);
@@ -1163,12 +1172,12 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct hppa_jit_context *ctx,
emit_imm32(tmp2, imm, ctx);
src = tmp2;
}
emit_alu_r64(dst, src, ctx, BPF_OP(code));
emit_alu_r64(dst, src, ctx, BPF_OP(code), off == 1);
break;
/* dst = -dst */
case BPF_ALU64 | BPF_NEG:
emit_alu_r64(dst, tmp2, ctx, BPF_OP(code));
emit_alu_r64(dst, tmp2, ctx, BPF_OP(code), false);
break;
case BPF_ALU64 | BPF_MOV | BPF_K:
@@ -1211,7 +1220,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct hppa_jit_context *ctx,
emit_imm32(tmp2, imm, ctx);
src = tmp2;
}
emit_alu_r32(dst, src, ctx, BPF_OP(code));
emit_alu_r32(dst, src, ctx, BPF_OP(code), off == 1);
break;
/* dst = dst OP imm */
@@ -1236,7 +1245,7 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct hppa_jit_context *ctx,
* src is ignored---choose tmp2 as a dummy register since it
* is not on the stack.
*/
emit_alu_r32(dst, tmp2, ctx, BPF_OP(code));
emit_alu_r32(dst, tmp2, ctx, BPF_OP(code), false);
break;
/* dst = BSWAP##imm(dst) */
arch/parisc/net/bpf_jit_comp64.c
+26 -9
View File
@@ -502,14 +502,19 @@ static void emit_call(u64 addr, bool fixed, struct hppa_jit_context *ctx)
emit_hppa_copy(HPPA_REG_RET0, regmap[BPF_REG_0], ctx);
}
static void emit_call_libgcc_ll(void *func, const s8 arg0,
const s8 arg1, u8 opcode, struct hppa_jit_context *ctx)
static void emit_call_libgcc_ll(void *func, const s8 arg0, const s8 arg1,
u8 opcode, bool is_signed, struct hppa_jit_context *ctx)
{
u64 func_addr;
if (BPF_CLASS(opcode) == BPF_ALU) {
emit_hppa64_zext32(arg0, HPPA_REG_ARG0, ctx);
emit_hppa64_zext32(arg1, HPPA_REG_ARG1, ctx);
if (is_signed) {
emit_hppa64_sext32(arg0, HPPA_REG_ARG0, ctx);
emit_hppa64_sext32(arg1, HPPA_REG_ARG1, ctx);
} else {
emit_hppa64_zext32(arg0, HPPA_REG_ARG0, ctx);
emit_hppa64_zext32(arg1, HPPA_REG_ARG1, ctx);
}
} else {
emit_hppa_copy(arg0, HPPA_REG_ARG0, ctx);
emit_hppa_copy(arg1, HPPA_REG_ARG1, ctx);
@@ -600,6 +605,8 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct hppa_jit_context *ctx,
u8 rd = -1, rs = -1, code = insn->code;
s16 off = insn->off;
s32 imm = insn->imm;
bool is_signed;
void *func;
init_regs(&rd, &rs, insn, ctx);
@@ -656,29 +663,39 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct hppa_jit_context *ctx,
fallthrough;
case BPF_ALU | BPF_MUL | BPF_X:
case BPF_ALU64 | BPF_MUL | BPF_X:
emit_call_libgcc_ll(__muldi3, rd, rs, code, ctx);
emit_call_libgcc_ll(__muldi3, rd, rs, code, false, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
break;
case BPF_ALU | BPF_DIV | BPF_K:
case BPF_ALU64 | BPF_DIV | BPF_K:
emit_imm(HPPA_REG_T1, is64 ? (s64)(s32)imm : (u32)imm, HPPA_REG_T2, ctx);
is_signed = (off == 1);
emit_imm(HPPA_REG_T1,
is64 || is_signed ? (s64)(s32)imm : (u32)imm,
HPPA_REG_T2, ctx);
rs = HPPA_REG_T1;
fallthrough;
case BPF_ALU | BPF_DIV | BPF_X:
case BPF_ALU64 | BPF_DIV | BPF_X:
emit_call_libgcc_ll(&hppa_div64, rd, rs, code, ctx);
is_signed = (off == 1);
func = is_signed ? &hppa_sdiv64 : &hppa_div64;
emit_call_libgcc_ll(func, rd, rs, code, is_signed, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
break;
case BPF_ALU | BPF_MOD | BPF_K:
case BPF_ALU64 | BPF_MOD | BPF_K:
emit_imm(HPPA_REG_T1, is64 ? (s64)(s32)imm : (u32)imm, HPPA_REG_T2, ctx);
is_signed = (off == 1);
emit_imm(HPPA_REG_T1,
is64 || is_signed ? (s64)(s32)imm : (u32)imm,
HPPA_REG_T2, ctx);
rs = HPPA_REG_T1;
fallthrough;
case BPF_ALU | BPF_MOD | BPF_X:
case BPF_ALU64 | BPF_MOD | BPF_X:
emit_call_libgcc_ll(&hppa_div64_rem, rd, rs, code, ctx);
is_signed = (off == 1);
func = is_signed ? &hppa_sdiv64_rem : &hppa_div64_rem;
emit_call_libgcc_ll(func, rd, rs, code, is_signed, ctx);
if (!is64 && !aux->verifier_zext)
emit_zext_32(rd, ctx);
break;
arch/parisc/net/bpf_jit_core.c
+14
View File
@@ -190,3 +190,17 @@ u64 hppa_div64_rem(u64 div, u64 divisor)
div64_u64_rem(div, divisor, &rem);
return rem;
}
u64 hppa_sdiv64(u64 div, u64 divisor)
{
s64 sdiv = div64_s64((s64)div, (s64)divisor);
return (u64)sdiv;
}
u64 hppa_sdiv64_rem(u64 div, u64 divisor)
{
s64 sdiv = div64_s64((s64)div, (s64)divisor);
return (u64)((s64)div - sdiv * (s64)divisor);
}
arch/powerpc/configs/ppc64_defconfig
-1
View File
@@ -442,7 +442,6 @@ CONFIG_TEST_IDA=m
CONFIG_TEST_BITOPS=m
CONFIG_TEST_VMALLOC=m
CONFIG_TEST_USER_COPY=m
CONFIG_TEST_BPF=m
CONFIG_TEST_BLACKHOLE_DEV=m
CONFIG_FIND_BIT_BENCHMARK=m
CONFIG_TEST_FIRMWARE=m
arch/powerpc/net/bpf_jit_comp.c
+4 -2
View File
@@ -267,8 +267,10 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
/* Make sure that the stack is quadword aligned. */
cgctx.stack_size = round_up(fp->aux->stack_depth, 16);
cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
if (bpf_prog_uses_arena_insns(fp)) {
cgctx.arena_vm_start = bpf_arena_get_kern_vm_start(fp->aux->arena);
cgctx.user_vm_start = bpf_arena_get_user_vm_start(fp->aux->arena);
}
cgctx.is_subprog = bpf_is_subprog(fp);
cgctx.exception_boundary = fp->aux->exception_boundary;
cgctx.exception_cb = fp->aux->exception_cb;
arch/riscv/net/bpf_jit.h
+4
View File
@@ -12,6 +12,9 @@
#include <linux/bpf.h>
#include <linux/filter.h>
#define RV_MAX_REG_ARGS 8
#define RV_EXTRA_STK_ARGS (RV_MAX_REG_ARGS - MAX_BPF_FUNC_REG_ARGS)
/* verify runtime detection extension status */
#define rv_ext_enabled(ext) \
(IS_ENABLED(CONFIG_RISCV_ISA_##ext) && riscv_has_extension_likely(RISCV_ISA_EXT_##ext))
@@ -82,6 +85,7 @@ struct rv_jit_context {
unsigned long flags;
int stack_size;
int tcc_offset;
int stack_arg_sz;
u64 arena_vm_start;
u64 user_vm_start;
};
arch/riscv/net/bpf_jit_comp64.c
+116 -24
View File
@@ -15,7 +15,6 @@
#include <asm/percpu.h>
#include "bpf_jit.h"
#define RV_MAX_REG_ARGS 8
#define RV_FENTRY_NINSNS 2
#define RV_FENTRY_NBYTES (RV_FENTRY_NINSNS * 4)
/* imm that allows emit_imm to emit max count insns */
@@ -23,11 +22,11 @@
/* fentry and TCC init insns will be skipped on tailcall */
#define RV_TAILCALL_OFFSET ((RV_FENTRY_NINSNS + 1) * 4)
#define RV_REG_TCC RV_REG_A6
#define RV_REG_TCC RV_REG_T5
#define RV_REG_ARENA RV_REG_S7 /* For storing arena_vm_start */
static const int regmap[] = {
[BPF_REG_0] = RV_REG_A5,
[BPF_REG_0] = RV_REG_T6,
[BPF_REG_1] = RV_REG_A0,
[BPF_REG_2] = RV_REG_A1,
[BPF_REG_3] = RV_REG_A2,
@@ -47,13 +46,13 @@ static const int pt_regmap[] = {
[RV_REG_A2] = offsetof(struct pt_regs, a2),
[RV_REG_A3] = offsetof(struct pt_regs, a3),
[RV_REG_A4] = offsetof(struct pt_regs, a4),
[RV_REG_A5] = offsetof(struct pt_regs, a5),
[RV_REG_S1] = offsetof(struct pt_regs, s1),
[RV_REG_S2] = offsetof(struct pt_regs, s2),
[RV_REG_S3] = offsetof(struct pt_regs, s3),
[RV_REG_S4] = offsetof(struct pt_regs, s4),
[RV_REG_S5] = offsetof(struct pt_regs, s5),
[RV_REG_T0] = offsetof(struct pt_regs, t0),
[RV_REG_T6] = offsetof(struct pt_regs, t6),
};
enum {
@@ -239,7 +238,7 @@ static void __build_epilogue(bool is_tail_call, struct rv_jit_context *ctx)
emit_addi(RV_REG_SP, RV_REG_SP, stack_adjust, ctx);
/* Set return value. */
if (!is_tail_call)
emit_addiw(RV_REG_A0, RV_REG_A5, 0, ctx);
emit_addiw(RV_REG_A0, regmap[BPF_REG_0], 0, ctx);
emit_jalr(RV_REG_ZERO, is_tail_call ? RV_REG_T3 : RV_REG_RA,
is_tail_call ? RV_TAILCALL_OFFSET : 0, ctx);
}
@@ -498,6 +497,18 @@ static void emit_ldx(u8 rd, s16 off, u8 rs, u8 size, bool sign_ext,
ctx->ex_jmp_off = ctx->ninsns;
}
static void emit_stack_arg_ldx(u8 rd, s16 off, struct rv_jit_context *ctx)
{
int idx = off / 8 - 1;
if (idx < RV_EXTRA_STK_ARGS) {
emit_mv(rd, RV_REG_A5 + idx, ctx);
return;
}
emit_ldx_insn(rd, (idx - RV_EXTRA_STK_ARGS) * 8, RV_REG_FP, BPF_DW, false, ctx);
}
static void emit_st(u8 rd, s16 off, s32 imm, u8 size, struct rv_jit_context *ctx)
{
emit_imm(RV_REG_T1, imm, ctx);
@@ -515,6 +526,19 @@ static void emit_st(u8 rd, s16 off, s32 imm, u8 size, struct rv_jit_context *ctx
ctx->ex_jmp_off = ctx->ninsns;
}
static void emit_stack_arg_st(s16 off, s32 imm, struct rv_jit_context *ctx)
{
int idx = -off / 8 - 1;
if (idx < RV_EXTRA_STK_ARGS) {
emit_imm(RV_REG_A5 + idx, imm, ctx);
return;
}
emit_imm(RV_REG_T1, imm, ctx);
emit_stx_insn(RV_REG_SP, (idx - RV_EXTRA_STK_ARGS) * 8, RV_REG_T1, BPF_DW, ctx);
}
static void emit_stx(u8 rd, s16 off, u8 rs, u8 size, struct rv_jit_context *ctx)
{
if (is_12b_int(off)) {
@@ -531,6 +555,18 @@ static void emit_stx(u8 rd, s16 off, u8 rs, u8 size, struct rv_jit_context *ctx)
ctx->ex_jmp_off = ctx->ninsns;
}
static void emit_stack_arg_stx(s16 off, u8 rs, struct rv_jit_context *ctx)
{
int idx = -off / 8 - 1;
if (idx < RV_EXTRA_STK_ARGS) {
emit_mv(RV_REG_A5 + idx, rs, ctx);
return;
}
emit_stx_insn(RV_REG_SP, (idx - RV_EXTRA_STK_ARGS) * 8, rs, BPF_DW, ctx);
}
static int emit_atomic_ld_st(u8 rd, u8 rs, const struct bpf_insn *insn,
struct rv_jit_context *ctx)
{
@@ -747,6 +783,8 @@ static int add_exception_handler(const struct bpf_insn *insn, int dst_reg,
if (BPF_MODE(insn->code) != BPF_PROBE_MEM &&
BPF_MODE(insn->code) != BPF_PROBE_MEMSX &&
BPF_MODE(insn->code) != BPF_PROBE_MEM32 &&
!(BPF_MODE(insn->code) == BPF_PROBE_MEM32SX &&
BPF_CLASS(insn->code) == BPF_LDX) &&
BPF_MODE(insn->code) != BPF_PROBE_ATOMIC)
return 0;
@@ -875,7 +913,8 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type old_t,
return ret;
}
static void store_args(int nr_arg_slots, int args_off, struct rv_jit_context *ctx)
static void store_args(int nr_arg_slots, int args_off, int stack_args_off,
struct rv_jit_context *ctx)
{
int i;
@@ -883,8 +922,8 @@ static void store_args(int nr_arg_slots, int args_off, struct rv_jit_context *ct
if (i < RV_MAX_REG_ARGS) {
emit_sd(RV_REG_FP, -args_off, RV_REG_A0 + i, ctx);
} else {
/* skip slots for T0 and FP of traced function */
emit_ld(RV_REG_T1, 16 + (i - RV_MAX_REG_ARGS) * 8, RV_REG_FP, ctx);
emit_ld(RV_REG_T1, stack_args_off +
(i - RV_MAX_REG_ARGS) * 8, RV_REG_FP, ctx);
emit_sd(RV_REG_FP, -args_off, RV_REG_T1, ctx);
}
args_off -= 8;
@@ -1100,7 +1139,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
save_ret = flags & (BPF_TRAMP_F_CALL_ORIG | BPF_TRAMP_F_RET_FENTRY_RET);
if (save_ret)
stack_size += 16; /* Save both A5 (BPF R0) and A0 */
stack_size += 16; /* Save both RV_REG_T6 (BPF R0) and RV_REG_A0 */
retval_off = stack_size;
stack_size += nr_arg_slots * 8;
@@ -1179,7 +1218,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
func_meta = nr_arg_slots;
emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx);
store_args(nr_arg_slots, args_off, ctx);
/*
* A direct struct_ops call has its first stack argument at the incoming
* SP, which the trampoline keeps as FP. The fentry path pushes the
* parent frame first, so its incoming stack arguments start at FP + 16.
*/
store_args(nr_arg_slots, args_off, is_struct_ops ? 0 : 16, ctx);
if (bpf_fsession_cnt(tnodes)) {
/* clear all session cookies' value */
@@ -1675,6 +1719,11 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
emit_zextw(rd, rd, ctx);
break;
/* JUMP reg */
case BPF_JMP | BPF_JA | BPF_X:
emit_jalr(RV_REG_ZERO, rd, 0, ctx);
break;
/* JUMP off */
case BPF_JMP | BPF_JA:
case BPF_JMP32 | BPF_JA:
@@ -1844,11 +1893,21 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
return -EINVAL;
for (idx = 0; idx < fm->nr_args; idx++) {
u8 reg = bpf_to_rv_reg(BPF_REG_1 + idx, ctx);
bool sign = fm->arg_flags[idx] & BTF_FMODEL_SIGNED_ARG;
u8 arg_sz = fm->arg_size[idx];
if (sign_extend(reg, reg, fm->arg_size[idx], sign, ctx))
return -EINVAL;
if (arg_sz == 8 || (arg_sz != 4 && !sign))
continue;
if (idx < RV_MAX_REG_ARGS) {
if (sign_extend(RV_REG_A0 + idx, RV_REG_A0 + idx, arg_sz, sign, ctx))
return -EINVAL;
} else {
emit_ld(RV_REG_T1, (idx - RV_MAX_REG_ARGS) * 8, RV_REG_SP, ctx);
if (sign_extend(RV_REG_T1, RV_REG_T1, arg_sz, sign, ctx))
return -EINVAL;
emit_sd(RV_REG_SP, (idx - RV_MAX_REG_ARGS) * 8, RV_REG_T1, ctx);
}
}
}
@@ -1873,6 +1932,13 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_A0, ctx);
break;
}
/* indirect call to a bpf subprog via the address in rd */
case BPF_JMP | BPF_CALL | BPF_X:
mark_call(ctx);
emit_jalr(RV_REG_RA, rd, 0, ctx);
break;
/* tail call */
case BPF_JMP | BPF_TAIL_CALL:
if (emit_bpf_tail_call(i, ctx))
@@ -1930,18 +1996,27 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
case BPF_LDX | BPF_PROBE_MEM32 | BPF_H:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_W:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW:
/* LDX | PROBE_MEM32SX: dst = *(signed size *)(src + RV_REG_ARENA + off) */
case BPF_LDX | BPF_PROBE_MEM32SX | BPF_B:
case BPF_LDX | BPF_PROBE_MEM32SX | BPF_H:
case BPF_LDX | BPF_PROBE_MEM32SX | BPF_W:
{
bool sign_ext;
sign_ext = BPF_MODE(insn->code) == BPF_MEMSX ||
BPF_MODE(insn->code) == BPF_PROBE_MEMSX;
BPF_MODE(insn->code) == BPF_PROBE_MEMSX ||
BPF_MODE(insn->code) == BPF_PROBE_MEM32SX;
if (BPF_MODE(insn->code) == BPF_PROBE_MEM32) {
if (BPF_MODE(insn->code) == BPF_PROBE_MEM32 ||
BPF_MODE(insn->code) == BPF_PROBE_MEM32SX) {
emit_add(RV_REG_T2, rs, RV_REG_ARENA, ctx);
rs = RV_REG_T2;
}
emit_ldx(rd, off, rs, BPF_SIZE(code), sign_ext, ctx);
if (is_stack_arg_ldx(insn))
emit_stack_arg_ldx(rd, off, ctx);
else
emit_ldx(rd, off, rs, BPF_SIZE(code), sign_ext, ctx);
ret = add_exception_handler(insn, rd, ctx);
if (ret)
@@ -1971,7 +2046,10 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
rd = RV_REG_T3;
}
emit_st(rd, off, imm, BPF_SIZE(code), ctx);
if (is_stack_arg_st(insn))
emit_stack_arg_st(off, imm, ctx);
else
emit_st(rd, off, imm, BPF_SIZE(code), ctx);
ret = add_exception_handler(insn, REG_DONT_CLEAR_MARKER, ctx);
if (ret)
@@ -1993,7 +2071,10 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
rd = RV_REG_T2;
}
emit_stx(rd, off, rs, BPF_SIZE(code), ctx);
if (is_stack_arg_stx(insn))
emit_stack_arg_stx(off, rs, ctx);
else
emit_stx(rd, off, rs, BPF_SIZE(code), ctx);
ret = add_exception_handler(insn, REG_DONT_CLEAR_MARKER, ctx);
if (ret)
@@ -2043,9 +2124,9 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,
void bpf_jit_build_prologue(struct rv_jit_context *ctx, bool is_subprog)
{
int i, stack_adjust = 0, store_offset, bpf_stack_adjust;
int i, stack_adjust = 0, store_offset, bpf_stack_adjust = ctx->stack_arg_sz;
bpf_stack_adjust = round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
bpf_stack_adjust += round_up(ctx->prog->aux->stack_depth, STACK_ALIGN);
if (bpf_stack_adjust)
mark_fp(ctx);
@@ -2143,6 +2224,11 @@ bool bpf_jit_supports_kfunc_call(void)
return true;
}
bool bpf_jit_supports_kfunc_ret_reg_pair(void)
{
return true;
}
bool bpf_jit_supports_ptr_xchg(void)
{
return true;
@@ -2170,10 +2256,6 @@ bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
if (insn->imm == BPF_CMPXCHG)
return rv_ext_enabled(ZACAS);
break;
case BPF_LDX | BPF_MEMSX | BPF_B:
case BPF_LDX | BPF_MEMSX | BPF_H:
case BPF_LDX | BPF_MEMSX | BPF_W:
return false;
}
}
@@ -2211,3 +2293,13 @@ bool bpf_jit_supports_timed_may_goto(void)
{
return true;
}
bool bpf_jit_supports_stack_args(void)
{
return true;
}
bool bpf_jit_supports_callx(void)
{
return true;
}
arch/riscv/net/bpf_jit_core.c
+26 -4
View File
@@ -26,10 +26,13 @@ static int build_body(struct rv_jit_context *ctx, bool extra_pass, int *offset)
int ret;
ret = bpf_jit_emit_insn(insn, ctx, extra_pass);
if (ret > 0)
i++; /* skip the next instruction */
if (offset)
offset[i] = ctx->ninsns;
if (ret > 0) {
i++; /* skip the next instruction */
if (offset)
offset[i] = ctx->ninsns;
}
if (ret < 0)
return ret;
}
@@ -48,6 +51,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
int pass = 0, prev_ninsns = 0, i;
struct rv_jit_data *jit_data;
struct rv_jit_context *ctx;
u16 stack_arg_cnt;
if (!prog->jit_requested)
return prog;
@@ -69,8 +73,16 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr
goto skip_init_ctx;
}
ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
if (bpf_prog_uses_arena_insns(prog)) {
ctx->arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena);
ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena);
}
stack_arg_cnt = bpf_out_stack_arg_cnt(env, prog);
/* First 3 stack args in regs, rest on stack */
ctx->stack_arg_sz = stack_arg_cnt > RV_EXTRA_STK_ARGS ?
round_up((stack_arg_cnt - RV_EXTRA_STK_ARGS) * 8, STACK_ALIGN) : 0;
ctx->prog = prog;
ctx->offset = kvzalloc_objs(int, prog->len);
if (!ctx->offset)
@@ -169,6 +181,15 @@ skip_init_ctx:
for (i = 0; i < prog->len; i++)
ctx->offset[i] = ninsns_rvoff(ctx->offset[i]);
bpf_prog_fill_jited_linfo(prog, ctx->offset);
/*
* bpf_prog_update_insn_ptrs() wants the start of each insn, so
* shift ctx->offset[] by one and get insn 0 from the prologue.
*/
for (i = prog->len - 1; i > 0; i--)
ctx->offset[i] = ctx->offset[i - 1];
ctx->offset[0] = ninsns_rvoff(ctx->prologue_len);
bpf_prog_update_insn_ptrs(prog, ctx->offset, jit_data->ro_image);
out_offset:
kvfree(ctx->offset);
kfree(jit_data);
@@ -237,6 +258,7 @@ void bpf_jit_free(struct bpf_prog *prog)
kvfree(jit_data->ctx.offset);
kfree(jit_data);
}
prog->bpf_func = (void *)prog->bpf_func - cfi_get_offset();
hdr = bpf_jit_binary_pack_hdr(prog);
bpf_jit_binary_pack_free(hdr, NULL);
WARN_ON_ONCE(!bpf_prog_kallsyms_verify_off(prog));
arch/riscv/net/bpf_timed_may_goto.S
+4 -4
View File
@@ -10,7 +10,7 @@
* - output: updated count in BPF_REG_AX (t0)
*
* Calls bpf_check_timed_may_goto(ptr) with the standard RISC-V ABI, where
* ptr = BPF_REG_FP (s5) + BPF_REG_AX (t0). BPF R0-R5 (a5, a0-a4) are saved
* ptr = BPF_REG_FP (s5) + BPF_REG_AX (t0). BPF R0-R5 (t6, a0-a4) are saved
* across the call; BPF_REG_FP (s5) is callee-saved and needs no saving.
*/
@@ -20,8 +20,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
REG_S s0, 6*SZREG(sp)
addi s0, sp, 8*SZREG
/* Save BPF registers R0-R5 (a5, a0-a4) */
REG_S a5, 5*SZREG(sp)
/* Save BPF registers R0-R5 (t6, a0-a4) */
REG_S t6, 5*SZREG(sp)
REG_S a0, 4*SZREG(sp)
REG_S a1, 3*SZREG(sp)
REG_S a2, 2*SZREG(sp)
@@ -38,7 +38,7 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
REG_L a2, 2*SZREG(sp)
REG_L a1, 3*SZREG(sp)
REG_L a0, 4*SZREG(sp)
REG_L a5, 5*SZREG(sp)
REG_L t6, 5*SZREG(sp)
REG_L s0, 6*SZREG(sp)
REG_L ra, 7*SZREG(sp)
arch/s390/configs/debug_defconfig
-1
View File
@@ -941,6 +941,5 @@ CONFIG_INTERVAL_TREE_TEST=m
CONFIG_PERCPU_TEST=m
CONFIG_ATOMIC64_SELFTEST=y
CONFIG_TEST_BITOPS=m
CONFIG_TEST_BPF=m
CONFIG_PAGE_TABLE_CHECK=y
CONFIG_PAGE_TABLE_CHECK_ENFORCED=y
arch/s390/configs/defconfig
-1
View File
@@ -865,4 +865,3 @@ CONFIG_LKDTM=m
CONFIG_KPROBES_SANITY_TEST=m
CONFIG_PERCPU_TEST=m
CONFIG_ATOMIC64_SELFTEST=y
CONFIG_TEST_BPF=m
arch/s390/net/bpf_jit_comp.c
+24 -12
View File
@@ -828,6 +828,14 @@ static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp,
return 0;
}
static void emit_shift_count32(struct bpf_jit *jit, u32 src_reg)
{
/* risbg %w1,%src,59,128+63,0 */
_EMIT6(0xec000000 | reg(REG_W1, src_reg) << 16 | 59 << 8 | (0x80 | 63), 0x0055);
REG_SET_SEEN(REG_W1);
REG_SET_SEEN(src_reg);
}
static int emit_ldx(struct bpf_jit *jit, struct bpf_prog *fp, struct bpf_insn *insn)
{
struct bpf_jit_probe probe;
@@ -1400,9 +1408,10 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
/*
* BPF_LSH
*/
case BPF_ALU | BPF_LSH | BPF_X: /* dst = (u32) dst << (u32) src */
/* sll %dst,0(%src) */
EMIT4_DISP(0x89000000, dst_reg, src_reg, 0);
case BPF_ALU | BPF_LSH | BPF_X: /* dst = (u32) dst << ((u32) src & 31) */
emit_shift_count32(jit, src_reg);
/* sll %dst,0(%w1) */
EMIT4_DISP(0x89000000, dst_reg, REG_W1, 0);
EMIT_ZERO(dst_reg);
break;
case BPF_ALU64 | BPF_LSH | BPF_X: /* dst = dst << src */
@@ -1425,9 +1434,10 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
/*
* BPF_RSH
*/
case BPF_ALU | BPF_RSH | BPF_X: /* dst = (u32) dst >> (u32) src */
/* srl %dst,0(%src) */
EMIT4_DISP(0x88000000, dst_reg, src_reg, 0);
case BPF_ALU | BPF_RSH | BPF_X: /* dst = (u32) dst >> ((u32) src & 31) */
emit_shift_count32(jit, src_reg);
/* srl %dst,0(%w1) */
EMIT4_DISP(0x88000000, dst_reg, REG_W1, 0);
EMIT_ZERO(dst_reg);
break;
case BPF_ALU64 | BPF_RSH | BPF_X: /* dst = dst >> src */
@@ -1450,9 +1460,10 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
/*
* BPF_ARSH
*/
case BPF_ALU | BPF_ARSH | BPF_X: /* ((s32) dst) >>= src */
/* sra %dst,%dst,0(%src) */
EMIT4_DISP(0x8a000000, dst_reg, src_reg, 0);
case BPF_ALU | BPF_ARSH | BPF_X: /* ((s32) dst) >>= (u32) src & 31 */
emit_shift_count32(jit, src_reg);
/* sra %dst,0(%w1) */
EMIT4_DISP(0x8a000000, dst_reg, REG_W1, 0);
EMIT_ZERO(dst_reg);
break;
case BPF_ALU64 | BPF_ARSH | BPF_X: /* ((s64) dst) >>= src */
@@ -2260,10 +2271,11 @@ static int bpf_jit_prog(struct bpf_jit *jit, struct bpf_prog *fp,
else
jit->frame_off = 0;
kern_arena = bpf_arena_get_kern_vm_start(fp->aux->arena);
if (kern_arena)
if (bpf_prog_uses_arena_insns(fp)) {
kern_arena = bpf_arena_get_kern_vm_start(fp->aux->arena);
jit->kern_arena = _EMIT_CONST_U64(kern_arena);
jit->user_arena = bpf_arena_get_user_vm_start(fp->aux->arena);
jit->user_arena = bpf_arena_get_user_vm_start(fp->aux->arena);
}
bpf_jit_prologue(jit, fp);
if (bpf_set_addr(jit, 0) < 0)
arch/x86/Kconfig
+1
View File
@@ -234,6 +234,7 @@ config X86
select HAVE_SAMPLE_FTRACE_DIRECT if X86_64
select HAVE_SAMPLE_FTRACE_DIRECT_MULTI if X86_64
select HAVE_EBPF_JIT
select HAVE_EBPF_JIT_KASAN if X86_64
select HAVE_EFFICIENT_UNALIGNED_ACCESS
select HAVE_EISA if X86_32
select HAVE_EXIT_THREAD
arch/x86/net/bpf_jit_comp.c
+1389 -415
View File
File diff suppressed because it is too large. Load diff
arch/x86/net/bpf_timed_may_goto.S
+2 -8
View File
@@ -11,12 +11,6 @@
SYM_FUNC_START(arch_bpf_timed_may_goto)
ANNOTATE_NOENDBR
/*
* r10 passes us stack depth, load the pointer to count and timestamp
* into r10 by adding it to BPF frame pointer.
*/
leaq (%rbp, %r10, 1), %r10
/* Setup frame. */
pushq %rbp
movq %rsp, %rbp
@@ -30,8 +24,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto)
pushq %r8
/*
* r10 has the pointer to count and timestamp, pass it as first
* argument.
* r10 has the pointer to count and timestamp. JIT added the frame
* pointer to the offset. Pass it as first argument.
*/
movq %r10, %rdi
crypto/Makefile
-3
View File
@@ -22,9 +22,6 @@ crypto_skcipher-y += lskcipher.o
crypto_skcipher-y += skcipher.o
obj-$(CONFIG_CRYPTO_SKCIPHER2) += crypto_skcipher.o
ifeq ($(CONFIG_BPF_SYSCALL),y)
obj-$(CONFIG_CRYPTO_SKCIPHER2) += bpf_crypto_skcipher.o
endif
obj-$(CONFIG_CRYPTO_SEQIV) += seqiv.o
obj-$(CONFIG_CRYPTO_ECHAINIV) += echainiv.o
crypto/bpf_crypto_skcipher.c
-83
View File
@@ -1,83 +0,0 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2024 Meta, Inc */
#include <linux/types.h>
#include <linux/module.h>
#include <linux/bpf_crypto.h>
#include <crypto/skcipher.h>
static void *bpf_crypto_lskcipher_alloc_tfm(const char *algo)
{
return crypto_alloc_lskcipher(algo, 0, 0);
}
static void bpf_crypto_lskcipher_free_tfm(void *tfm)
{
crypto_free_lskcipher(tfm);
}
static int bpf_crypto_lskcipher_has_algo(const char *algo)
{
return crypto_has_skcipher(algo, CRYPTO_ALG_TYPE_LSKCIPHER, CRYPTO_ALG_TYPE_MASK);
}
static int bpf_crypto_lskcipher_setkey(void *tfm, const u8 *key, unsigned int keylen)
{
return crypto_lskcipher_setkey(tfm, key, keylen);
}
static u32 bpf_crypto_lskcipher_get_flags(void *tfm)
{
return crypto_lskcipher_get_flags(tfm);
}
static unsigned int bpf_crypto_lskcipher_ivsize(void *tfm)
{
return crypto_lskcipher_ivsize(tfm);
}
static unsigned int bpf_crypto_lskcipher_statesize(void *tfm)
{
return crypto_lskcipher_statesize(tfm);
}
static int bpf_crypto_lskcipher_encrypt(void *tfm, const u8 *src, u8 *dst,
unsigned int len, u8 *siv)
{
return crypto_lskcipher_encrypt(tfm, src, dst, len, siv);
}
static int bpf_crypto_lskcipher_decrypt(void *tfm, const u8 *src, u8 *dst,
unsigned int len, u8 *siv)
{
return crypto_lskcipher_decrypt(tfm, src, dst, len, siv);
}
static const struct bpf_crypto_type bpf_crypto_lskcipher_type = {
.alloc_tfm = bpf_crypto_lskcipher_alloc_tfm,
.free_tfm = bpf_crypto_lskcipher_free_tfm,
.has_algo = bpf_crypto_lskcipher_has_algo,
.setkey = bpf_crypto_lskcipher_setkey,
.encrypt = bpf_crypto_lskcipher_encrypt,
.decrypt = bpf_crypto_lskcipher_decrypt,
.ivsize = bpf_crypto_lskcipher_ivsize,
.statesize = bpf_crypto_lskcipher_statesize,
.get_flags = bpf_crypto_lskcipher_get_flags,
.owner = THIS_MODULE,
.name = "skcipher",
};
static int __init bpf_crypto_skcipher_init(void)
{
return bpf_crypto_register_type(&bpf_crypto_lskcipher_type);
}
static void __exit bpf_crypto_skcipher_exit(void)
{
int err = bpf_crypto_unregister_type(&bpf_crypto_lskcipher_type);
WARN_ON_ONCE(err);
}
module_init(bpf_crypto_skcipher_init);
module_exit(bpf_crypto_skcipher_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Symmetric key cipher support for BPF");
drivers/net/ethernet/netronome/nfp/bpf/verifier.c
+1 -1
View File
@@ -105,7 +105,7 @@ static bool nfp_bpf_map_update_value_ok(struct bpf_verifier_env *env)
unsigned int soff;
soff = -(off + i) - 1;
stack_entry = &state->stack[soff / BPF_REG_SIZE];
stack_entry = bpf_stack_slot(state, soff / BPF_REG_SIZE);
if (stack_entry->slot_type[soff % BPF_REG_SIZE] == STACK_ZERO)
continue;
fs/bpf_fs_kfuncs.c
+99
View File
@@ -11,6 +11,7 @@
#include <linux/fsnotify.h>
#include <linux/file.h>
#include <linux/kernfs.h>
#include <linux/lsm_hooks.h>
#include <linux/mm.h>
#include <linux/net.h>
#include <linux/xattr.h>
@@ -328,6 +329,89 @@ __bpf_kfunc int bpf_remove_dentry_xattr(struct dentry *dentry, const char *name_
return ret;
}
static int bpf_inode_init_xattrs_claimed(const struct xattr *xattrs, int xattr_count)
{
const size_t suffix_len = sizeof(XATTR_BPF_LSM_SUFFIX) - 1;
int i, claimed = 0;
for (i = 0; i < xattr_count; i++) {
const char *name = xattrs[i].name;
if (name && !strncmp(name, XATTR_BPF_LSM_SUFFIX, suffix_len))
claimed++;
}
return claimed;
}
/**
* bpf_inode_init_xattr - attach a xattr to an inode that is being created
* @xattrs: xattr array the inode_init_security hook was handed
* @xattr_count__ctx_out: xattr count the inode_init_security hook was handed
* @name__str: name of the xattr
* @value_p: xattr value
*
* Claim one of the slots the BPF LSM reserved in the xattr array, so that
* the xattr is written out as part of the transaction creating the inode.
*
* For security reasons, only *name__str* with prefix "security.bpf." is
* allowed. The slot stores the name without that "security." prefix, which
* the filesystem's initxattrs() callback puts back.
*
* At most BPF_LSM_INODE_INIT_XATTRS xattrs can be attached to one inode,
* matching the number of slots the BPF LSM reserves.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_inode_init_xattr(struct xattr *xattrs,
int *xattr_count__ctx_out,
const char *name__str,
const struct bpf_dynptr *value_p)
{
const struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;
int *xattr_count = xattr_count__ctx_out;
const char *suffix;
struct xattr *slot;
const void *value;
size_t name_len;
u32 value_len;
char *buf;
if (!match_security_bpf_prefix(name__str))
return -EPERM;
if (bpf_inode_init_xattrs_claimed(xattrs, *xattr_count) >=
BPF_LSM_INODE_INIT_XATTRS)
return -ENOSPC;
suffix = name__str + XATTR_SECURITY_PREFIX_LEN;
name_len = strlen(suffix);
if (name_len <= sizeof(XATTR_BPF_LSM_SUFFIX) - 1 ||
name_len > XATTR_NAME_MAX - XATTR_SECURITY_PREFIX_LEN)
return -EINVAL;
value_len = __bpf_dynptr_size(value_ptr);
value = __bpf_dynptr_data(value_ptr, value_len);
if (!value)
return -EINVAL;
if (value_len > XATTR_SIZE_MAX)
return -E2BIG;
buf = kmalloc(value_len + name_len + 1, GFP_NOWAIT | __GFP_NOWARN);
if (!buf)
return -ENOMEM;
memcpy(buf, value, value_len);
memcpy(buf + value_len, suffix, name_len + 1);
slot = lsm_get_xattr_slot(xattrs, xattr_count);
if (!slot) {
kfree(buf);
return -ENOSPC;
}
slot->value = buf;
slot->value_len = value_len;
slot->name = buf + value_len;
return 0;
}
#ifdef CONFIG_CGROUPS
/**
* bpf_cgroup_read_xattr - read xattr of a cgroup's node in cgroupfs
@@ -425,6 +509,7 @@ BTF_ID_FLAGS(func, bpf_get_file_xattr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_set_dentry_xattr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_remove_dentry_xattr, KF_SLEEPABLE)
BTF_ID_FLAGS(func, bpf_real_data_inode, KF_SLEEPABLE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_inode_init_xattr)
#ifdef CONFIG_NET
BTF_ID_FLAGS(func, bpf_sock_read_xattr, KF_RCU)
#endif
@@ -436,10 +521,24 @@ BTF_ID(func, bpf_set_dentry_xattr)
BTF_ID(func, bpf_remove_dentry_xattr)
BTF_SET_END(bpf_fs_kfunc_lsm_only_ids)
BTF_ID_LIST_SINGLE(bpf_inode_init_xattr_ids, func, bpf_inode_init_xattr)
BTF_SET_START(bpf_inode_init_xattr_hooks)
BTF_ID(func, bpf_lsm_inode_init_security)
BTF_SET_END(bpf_inode_init_xattr_hooks)
static int bpf_fs_kfuncs_filter(const struct bpf_prog *prog, u32 kfunc_id)
{
if (!btf_id_set8_contains(&bpf_fs_kfunc_set_ids, kfunc_id))
return 0;
if (kfunc_id == bpf_inode_init_xattr_ids[0]) {
if (prog->type != BPF_PROG_TYPE_LSM ||
prog->expected_attach_type != BPF_LSM_MAC ||
!btf_id_set_contains(&bpf_inode_init_xattr_hooks,
prog->aux->attach_btf_id))
return -EACCES;
return 0;
}
if (prog->type == BPF_PROG_TYPE_LSM)
return 0;
if (prog->type != BPF_PROG_TYPE_STRUCT_OPS)
fs/exec.c
+5 -2
View File
@@ -1471,9 +1471,12 @@ void bprm_drop_loader(struct linux_binprm *bprm)
static void free_bprm(struct linux_binprm *bprm)
{
if (bprm->mm) {
struct mm_struct *mm = bprm->mm;
if (mm) {
acct_arg_size(bprm, 0);
mmput(bprm->mm);
bprm->mm = NULL;
mmput(mm);
}
if (bprm->user_ns)
put_user_ns(bprm->user_ns);
include/asm-generic/vmlinux.lds.h
+31
View File
@@ -675,12 +675,43 @@
/*
* .BTF
*/
#ifdef CONFIG_DEBUG_INFO_BTF_INLINE
#define BTF_INLINE \
. = ALIGN(PAGE_SIZE); \
.BTF.inline : AT(ADDR(.BTF.inline) - LOAD_OFFSET) { \
BOUNDED_SECTION_BY(.BTF.inline, _BTF_inline) \
}
#else
#define BTF_INLINE
#endif
#ifdef CONFIG_DEBUG_INFO_BTF_INLINE_MODULE
/* __MODULE_NAME_LEN + SHA-256 digest + u32 BTF size */
#ifdef CONFIG_64BIT
#define BTF_LINK_SIZE (64 - 8 + 32 + 4)
#else
#define BTF_LINK_SIZE (64 - 4 + 32 + 4)
#endif
#define BTF_INLINE_LINK \
. = ALIGN(PAGE_SIZE); \
.BTF.inline.link : AT(ADDR(.BTF.inline.link) - LOAD_OFFSET) { \
__start_BTF_inline_link = .; \
BYTE(0) \
. += BTF_LINK_SIZE - 1; \
__stop_BTF_inline_link = .; \
}
#else
#define BTF_INLINE_LINK
#endif
#ifdef CONFIG_DEBUG_INFO_BTF
#define BTF \
. = ALIGN(PAGE_SIZE); \
.BTF : AT(ADDR(.BTF) - LOAD_OFFSET) { \
BOUNDED_SECTION_BY(.BTF, _BTF) \
} \
BTF_INLINE \
BTF_INLINE_LINK \
. = ALIGN(PAGE_SIZE); \
.BTF_ids : AT(ADDR(.BTF_ids) - LOAD_OFFSET) { \
*(.BTF_ids) \
include/linux/bpf-cgroup-defs.h
+1
View File
@@ -47,6 +47,7 @@ enum cgroup_bpf_attach_type {
CGROUP_INET6_GETSOCKNAME,
CGROUP_UNIX_GETSOCKNAME,
CGROUP_INET_SOCK_RELEASE,
CGROUP_TCP_SOCK_OPS,
CGROUP_LSM_START,
CGROUP_LSM_END = CGROUP_LSM_START + CGROUP_LSM_NUM - 1,
MAX_CGROUP_BPF_ATTACH_TYPE
include/linux/bpf-cgroup.h
+41 -21
View File
@@ -100,6 +100,8 @@ struct bpf_cgroup_storage {
struct bpf_cgroup_link {
struct bpf_link link;
struct cgroup *cgroup;
struct bpf_map *map;
wait_queue_head_t wait_hup;
};
struct bpf_prog_list {
@@ -110,6 +112,18 @@ struct bpf_prog_list {
u32 flags;
};
#define bpf_cgroup_struct_ops_foreach(var, item, cgrp, atype) \
for (item = rcu_dereference((cgrp)->bpf.effective[atype])->items;\
((var) = READ_ONCE(item->kdata)); \
item++)
static inline bool cgroup_bpf_is_struct_ops_atype(enum cgroup_bpf_attach_type atype)
{
return atype == CGROUP_TCP_SOCK_OPS;
}
void cgroup_bpf_struct_ops_register(int atype, u32 type_id, void *cfi_stubs, bool mult_trace);
int cgroup_bpf_struct_ops_attach(struct bpf_map *map, const union bpf_attr *attr);
void __init cgroup_bpf_lifetime_notifier_init(void);
int __cgroup_bpf_run_filter_skb(struct sock *sk,
@@ -331,27 +345,19 @@ static inline bool cgroup_bpf_sock_enabled(struct sock *sk,
* calling bpf_setsockopt on listener-sk will not make sense anyway,
* so passing 'sock_ops->sk == req_sk' to the bpf prog is appropriate here.
*/
#define BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(sock_ops, sk) \
({ \
int __ret = 0; \
if (cgroup_bpf_enabled(CGROUP_SOCK_OPS)) \
__ret = __cgroup_bpf_run_filter_sock_ops(sk, \
sock_ops, \
CGROUP_SOCK_OPS); \
__ret; \
})
#define __BPF_CGROUP_RUN_PROG_SOCK_OPS_SK(sock_ops, sk) \
__cgroup_bpf_run_filter_sock_ops(sk, sock_ops, \
CGROUP_SOCK_OPS)
#define BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) \
({ \
int __ret = 0; \
if (cgroup_bpf_enabled(CGROUP_SOCK_OPS) && (sock_ops)->sk) { \
typeof(sk) __sk = sk_to_full_sk((sock_ops)->sk); \
if (__sk && sk_fullsock(__sk)) \
__ret = __cgroup_bpf_run_filter_sock_ops(__sk, \
sock_ops, \
CGROUP_SOCK_OPS); \
} \
__ret; \
#define __BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) \
({ \
int __ret = 0; \
typeof(sk) __sk = sk_to_full_sk((sock_ops)->sk); \
if (__sk && sk_fullsock(__sk)) \
__ret = __cgroup_bpf_run_filter_sock_ops(__sk, \
sock_ops, \
CGROUP_SOCK_OPS); \
__ret; \
})
#define BPF_CGROUP_RUN_PROG_DEVICE_CGROUP(atype, major, minor, access) \
@@ -479,6 +485,20 @@ static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,
return 0;
}
static inline bool cgroup_bpf_is_struct_ops_atype(int atype)
{
return false;
}
static inline void cgroup_bpf_struct_ops_register(int atype, u32 type_id, void *cfi_stubs,
bool mult_trace)
{
}
static inline int cgroup_bpf_struct_ops_attach(struct bpf_map *map,
const union bpf_attr *attr)
{
return -EOPNOTSUPP;
}
#define cgroup_bpf_enabled(atype) (0)
#define BPF_CGROUP_RUN_SA_PROG_LOCK(sk, uaddr, uaddrlen, atype, t_ctx) ({ 0; })
#define BPF_CGROUP_RUN_SA_PROG(sk, uaddr, uaddrlen, atype) ({ 0; })
@@ -501,7 +521,7 @@ static inline int bpf_percpu_cgroup_storage_update(struct bpf_map *map,
#define BPF_CGROUP_RUN_PROG_UDP4_RECVMSG_LOCK(sk, uaddr, uaddrlen) ({ 0; })
#define BPF_CGROUP_RUN_PROG_UDP6_RECVMSG_LOCK(sk, uaddr, uaddrlen) ({ 0; })
#define BPF_CGROUP_RUN_PROG_UNIX_RECVMSG_LOCK(sk, uaddr, uaddrlen) ({ 0; })
#define BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) ({ 0; })
#define __BPF_CGROUP_RUN_PROG_SOCK_OPS(sock_ops) ({ 0; })
#define BPF_CGROUP_RUN_PROG_DEVICE_CGROUP(atype, major, minor, access) ({ 0; })
#define BPF_CGROUP_RUN_PROG_SYSCTL(head,table,write,buf,count,pos) ({ 0; })
#define BPF_CGROUP_RUN_PROG_GETSOCKOPT(sock, level, optname, optval, \
include/linux/bpf.h
+203 -27
View File
@@ -17,6 +17,7 @@
#include <linux/numa.h>
#include <linux/mm_types.h>
#include <linux/wait.h>
#include <linux/irq_work_types.h>
#include <linux/refcount.h>
#include <linux/mutex.h>
#include <linux/module.h>
@@ -90,6 +91,7 @@ struct bpf_map_ops {
struct bpf_map *(*map_alloc)(union bpf_attr *attr);
void (*map_release)(struct bpf_map *map, struct file *map_file);
void (*map_free)(struct bpf_map *map);
void (*map_free_pre_rcu)(struct bpf_map *map);
int (*map_get_next_key)(struct bpf_map *map, void *key, void *next_key);
void (*map_release_uref)(struct bpf_map *map);
void *(*map_lookup_elem_sys_only)(struct bpf_map *map, void *key);
@@ -214,6 +216,7 @@ enum btf_field_type {
BPF_UPTR = (1 << 11),
BPF_RES_SPIN_LOCK = (1 << 12),
BPF_TASK_WORK = (1 << 13),
BPF_RCU_HEAD = (1 << 14),
};
enum bpf_cgroup_storage_type {
@@ -268,6 +271,7 @@ struct btf_record {
int wq_off;
int refcount_off;
int task_work_off;
int rcu_head_off;
struct btf_field fields[];
};
@@ -337,12 +341,23 @@ struct bpf_map {
bool frozen; /* write-once; write-protected by freeze_mutex */
bool free_after_mult_rcu_gp;
bool free_after_rcu_gp;
bool was_inner_map;
atomic64_t sleepable_refcnt;
s64 __percpu *elem_count;
u64 cookie; /* write-once */
char *excl_prog_sha;
/*
* Which programs use the map, see bpf_map_claim(): 0 - none so far,
* aux of the program - only that one, the same with BPF_MAP_USER_PATCHED
* set - only that one and it stored the addresses of its functions into
* the map, BPF_MAP_USER_MANY - more than one.
*/
unsigned long user;
};
#define BPF_MAP_USER_MANY 1UL
#define BPF_MAP_USER_PATCHED 1UL
static inline const char *btf_field_type_name(enum btf_field_type type)
{
switch (type) {
@@ -373,6 +388,8 @@ static inline const char *btf_field_type_name(enum btf_field_type type)
return "bpf_refcount";
case BPF_TASK_WORK:
return "bpf_task_work";
case BPF_RCU_HEAD:
return "bpf_rcu_head";
default:
WARN_ON_ONCE(1);
return "unknown";
@@ -413,6 +430,8 @@ static inline u32 btf_field_type_size(enum btf_field_type type)
return sizeof(struct bpf_refcount);
case BPF_TASK_WORK:
return sizeof(struct bpf_task_work);
case BPF_RCU_HEAD:
return sizeof(struct bpf_rcu_head);
default:
WARN_ON_ONCE(1);
return 0;
@@ -447,6 +466,8 @@ static inline u32 btf_field_type_align(enum btf_field_type type)
return __alignof__(struct bpf_refcount);
case BPF_TASK_WORK:
return __alignof__(struct bpf_task_work);
case BPF_RCU_HEAD:
return __alignof__(struct bpf_rcu_head);
default:
WARN_ON_ONCE(1);
return 0;
@@ -479,6 +500,7 @@ static inline void bpf_obj_init_field(const struct btf_field *field, void *addr)
case BPF_KPTR_PERCPU:
case BPF_UPTR:
case BPF_TASK_WORK:
case BPF_RCU_HEAD:
break;
default:
WARN_ON_ONCE(1);
@@ -572,7 +594,7 @@ static inline void bpf_obj_memcpy(struct btf_record *rec,
if (long_memcpy)
bpf_long_memcpy(dst, src, size);
else
memcpy(dst, src, size);
data_race(memcpy(dst, src, size));
return;
}
@@ -580,10 +602,10 @@ static inline void bpf_obj_memcpy(struct btf_record *rec,
u32 next_off = rec->fields[i].offset;
u32 sz = next_off - curr_off;
memcpy(dst + curr_off, src + curr_off, sz);
data_race(memcpy(dst + curr_off, src + curr_off, sz));
curr_off += rec->fields[i].size + sz;
}
memcpy(dst + curr_off, src + curr_off, size - curr_off);
data_race(memcpy(dst + curr_off, src + curr_off, size - curr_off));
}
static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
@@ -780,7 +802,10 @@ enum bpf_type_flag {
*/
PTR_UNTRUSTED = BIT(6 + BPF_BASE_TYPE_BITS),
/* MEM can be uninitialized. */
/*
* MEM can be uninitialized. Generic memory outputs need not be fully
* initialized by the callee.
*/
MEM_UNINIT = BIT(7 + BPF_BASE_TYPE_BITS),
/* DYNPTR points to memory local to the bpf program. */
@@ -874,7 +899,7 @@ enum bpf_type_flag {
/* function argument constraints */
enum bpf_arg_type {
ARG_DONTCARE = 0, /* unused argument in helper function */
ARG_UNUSED = 0, /* unused argument; terminates argument iteration */
/* the following constraints used to prototype
* bpf_map_lookup/update/delete_elem() functions
@@ -909,6 +934,24 @@ enum bpf_arg_type {
ARG_PTR_TO_TIMER, /* pointer to bpf_timer */
ARG_KPTR_XCHG_DEST, /* pointer to destination that kptrs are bpf_kptr_xchg'd into */
ARG_PTR_TO_DYNPTR, /* pointer to bpf_dynptr. See bpf_type_flag for dynptr type */
ARG_CONST_SCALAR, /* scalar known at verification time */
ARG_CONST_MEM_SIZE, /* ARG_MEM_SIZE that must be constant */
ARG_PTR_TO_ALLOC_BTF_ID, /* pointer to an allocated object */
ARG_PTR_TO_REFCOUNTED_KPTR, /* pointer to a refcounted local kptr */
ARG_PTR_TO_ITER, /* pointer to an iterator */
ARG_PTR_TO_LIST_HEAD, /* pointer to bpf_list_head */
ARG_PTR_TO_LIST_NODE, /* pointer to bpf_list_node */
ARG_PTR_TO_RB_ROOT, /* pointer to bpf_rb_root */
ARG_PTR_TO_RB_NODE, /* pointer to bpf_rb_node */
ARG_PTR_TO_WORKQUEUE, /* pointer to bpf_wq */
ARG_PTR_TO_TASK_WORK, /* pointer to bpf_task_work */
ARG_PTR_TO_RCU_HEAD, /* pointer to bpf_rcu_head */
ARG_PTR_TO_IRQ_FLAG, /* pointer to saved IRQ flags on the stack */
ARG_PTR_TO_RES_SPIN_LOCK, /* pointer to bpf_res_spin_lock */
ARG_PTR_TO_CTX_OUT, /* hook output argument passed through from ctx */
ARG_PTR_TO_PROG_AUX, /* pointer to the caller's bpf_prog_aux */
ARG_IGNORE, /* argument the verifier does not check at all */
__BPF_ARG_TYPE_MAX,
/* Extended arg_types. */
@@ -977,6 +1020,13 @@ static_assert(__BPF_RET_TYPE_MAX <= BPF_BASE_TYPE_LIMIT);
*/
#define MAX_BPF_FUNC_REG_ARGS 5
/* A by-value argument takes two eightbytes at most, so the maximum number of
* argument slots of any function is 2 * MAX_BPF_FUNC_ARGS. A local array may
* need that size for processing, although eventually the maximum slots will
* be capped at MAX_BPF_FUNC_ARGS.
*/
#define MAX_BPF_FUNC_ARG_SLOTS (2 * MAX_BPF_FUNC_ARGS)
/* eBPF function prototype used by verifier to allow BPF_CALLs from eBPF programs
* to in-kernel helper functions and for adjusting imm32 field in BPF_CALL
* instructions after verifying
@@ -1005,13 +1055,13 @@ struct bpf_func_proto {
};
union {
struct {
u32 *arg1_btf_id;
u32 *arg2_btf_id;
u32 *arg3_btf_id;
u32 *arg4_btf_id;
u32 *arg5_btf_id;
const u32 *arg1_btf_id;
const u32 *arg2_btf_id;
const u32 *arg3_btf_id;
const u32 *arg4_btf_id;
const u32 *arg5_btf_id;
};
u32 *arg_btf_id[MAX_BPF_FUNC_ARGS];
const u32 *arg_btf_id[MAX_BPF_FUNC_ARGS];
struct {
size_t arg1_size;
size_t arg2_size;
@@ -1114,6 +1164,7 @@ struct bpf_insn_access_aux {
u32 ref_id;
};
};
u32 mem_size;
struct bpf_verifier_log *log; /* for verbose logs */
bool is_retval; /* is accessing function return value ? */
};
@@ -1194,6 +1245,9 @@ struct bpf_prog_offload {
u32 jited_len;
};
/* The argument is aligned to 16 bytes. */
#define BTF_FMODEL_ALIGN16_ARG BIT(0)
/* The argument is signed. */
#define BTF_FMODEL_SIGNED_ARG BIT(1)
@@ -1211,6 +1265,11 @@ struct btf_func_model {
u8 arg_flags[MAX_BPF_FUNC_ARGS];
};
static inline u32 btf_func_model_arg_slots(const struct btf_func_model *m, u32 arg)
{
return (m->arg_size[arg] + sizeof(u64) - 1) / sizeof(u64);
}
/* Restore arguments before returning from trampoline to let original function
* continue executing. This flag is used for fentry progs when there are no
* fexit progs.
@@ -1684,12 +1743,21 @@ static inline void bpf_trampoline_multi_detach(struct bpf_prog *prog,
static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {}
#endif
/* The contexts that a global function is called in, and is verified for. */
enum bpf_func_context {
BPF_FUNC_CTX_NON_SLEEPABLE,
BPF_FUNC_CTX_SLEEPABLE,
/* Sleepable code has disabled preemption or interrupts: no RCU read lock. */
BPF_FUNC_CTX_NO_RCU_LOCK,
BPF_FUNC_CTX_CNT,
};
struct bpf_func_info_aux {
u16 linkage;
bool unreliable;
/* Indexed by in_sleepable. */
bool called[2];
bool verified[2];
bool called[BPF_FUNC_CTX_CNT];
bool verified[BPF_FUNC_CTX_CNT];
bool called_with_pkt_ptrs;
};
enum bpf_jit_poke_reason {
@@ -1721,6 +1789,7 @@ struct bpf_ctx_arg_aux {
struct btf *btf;
u32 btf_id;
u32 ref_id;
u32 mem_size;
bool refcounted;
};
@@ -1749,12 +1818,18 @@ enum {
};
struct bpf_stream {
atomic_t capacity;
refcount_t refcnt;
atomic_t capacity; /* bytes reserved against the stream limit */
atomic_t readable; /* published bytes available to readers */
struct llist_head log; /* list of in-flight stream elements in LIFO order */
struct mutex lock; /* lock protecting backlog_{head,tail} */
struct llist_node *backlog_head; /* list of in-flight stream elements in FIFO order */
struct llist_node *backlog_tail; /* tail of the list above */
wait_queue_head_t waitq;
struct irq_work notify_work;
bool notify_used; /* notify_work was queued at least once */
bool dead;
};
struct bpf_stream_stage {
@@ -1773,6 +1848,7 @@ enum bpf_sig_keyring {
BPF_SIG_KEYRING_SECONDARY,
BPF_SIG_KEYRING_PLATFORM,
BPF_SIG_KEYRING_USER,
BPF_SIG_KEYRING_BPF,
};
struct bpf_prog_aux {
@@ -1792,6 +1868,9 @@ struct bpf_prog_aux {
u32 ctx_arg_info_size;
u32 max_rdonly_access;
u32 max_rdwr_access;
/* [min, max): the bytes of a rdwr buffer that may be written */
u32 min_rdwr_write;
u32 max_rdwr_write;
u32 subprog_start;
struct btf *attach_btf;
struct bpf_ctx_arg_aux *ctx_arg_info;
@@ -1807,24 +1886,28 @@ struct bpf_prog_aux {
bool attach_btf_trace; /* true if attaching to BTF-enabled raw tp */
bool attach_tracing_prog; /* true if tracing another tracing program */
bool tramp_linked; /* true if it was ever linked to a trampoline */
bool sleepable_async_cb; /* has a bpf_wq or bpf_task_work callback */
bool func_proto_unreliable;
bool tail_call_reachable;
bool xdp_has_frags;
bool exception_cb;
bool exception_boundary;
bool is_extended; /* true if extended by freplace program */
bool jits_use_priv_stack;
bool priv_stack_requested;
bool changes_pkt_data;
bool has_prologue;
bool might_sleep;
bool can_unwind;
bool kprobe_write_ctx;
/* has insns that work with the arena base, see bpf_prog_has_arena_insns() */
bool uses_arena_insns;
struct {
s32 keyring_serial;
u8 keyring_type;
u8 verdict;
} sig;
u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
struct mutex ext_mutex; /* mutex for freplace_link_cnt and prog_array_member_cnt */
struct bpf_arena *arena;
void (*recursion_detected)(struct bpf_prog *prog); /* callback if recursion is detected */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
@@ -1856,6 +1939,7 @@ struct bpf_prog_aux {
char name[BPF_OBJ_NAME_LEN];
u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64);
u16 stack_arg_sp_adjust;
u16 freplace_link_cnt; /* counts freplace links extending this prog */
#ifdef CONFIG_SECURITY
void *security;
#endif
@@ -1892,8 +1976,9 @@ struct bpf_prog_aux {
union {
struct work_struct work;
struct rcu_head rcu;
struct rcu_work rwork;
};
struct bpf_stream stream[2];
struct bpf_stream *stream[2];
struct mutex st_ops_assoc_mutex;
struct bpf_map __rcu *st_ops_assoc;
};
@@ -1940,6 +2025,15 @@ struct bpf_prog {
};
};
/*
* Does the program, or a callback of it, run under RCU Tasks Trace protection
* only, outside of any RCU read side critical section?
*/
static inline bool bpf_prog_runs_sleepable(const struct bpf_prog *prog)
{
return prog->sleepable || prog->aux->sleepable_async_cb;
}
struct bpf_array_aux {
/* Programs with direct jumps into programs part of this array. */
struct list_head poke_progs;
@@ -2137,6 +2231,18 @@ struct btf_member;
* unloaded while in use.
* @name: The name of the struct bpf_struct_ops object.
* @func_models: Func models
* @cgroup_atype: A value in enum cgroup_bpf_attach_type for cgroup attachment.
* 0 means the struct_ops type does not support cgroup attachment.
* If cgroup_atype is non-zero, the @reg and @unreg must be NULL
* because the attachment/detachment will be handled by the bpf core.
* @free_after_tasks_rcu_gp: Set to true if it needs the bpf core to wait for
* a tasks_rcu gp before freeing the struct_ops map
* and its progs. It is unnecessary if the @unreg
* has waited for the correct rcu gp or the @unreg
* has ensured all struct_ops prog has finished running.
* @free_after_mult_rcu_gp: Same as @free_after_tasks_rcu_gp but waiting for
* both tasks_trace_rcu and regular rcu grace period.
* It is usually needed if the struct_ops has sleepable prog.
*/
struct bpf_struct_ops {
const struct bpf_verifier_ops *verifier_ops;
@@ -2155,6 +2261,9 @@ struct bpf_struct_ops {
struct module *owner;
const char *name;
struct btf_func_model func_models[BPF_STRUCT_OPS_MAX_NR_MEMBERS];
int cgroup_atype;
bool free_after_tasks_rcu_gp;
bool free_after_mult_rcu_gp;
};
/* Every member of a struct_ops type has an instance even a member is not
@@ -2289,6 +2398,12 @@ u32 bpf_struct_ops_id(const void *kdata);
int bpf_struct_ops_for_each_prog(const void *kdata,
int (*cb)(struct bpf_prog *prog, void *data),
void *data);
void *bpf_struct_ops_map_kdata(struct bpf_map *map);
void *bpf_struct_ops_map_cfi_stubs(struct bpf_map *map);
bool bpf_struct_ops_valid_to_reg(struct bpf_map *map);
int bpf_struct_ops_link_update_check(struct bpf_map *new_map, struct bpf_map *old_map,
struct bpf_map *expected_old_map);
int bpf_struct_ops_map_cgroup_atype(struct bpf_map *map);
#ifdef CONFIG_NET
/* Define it here to avoid the use of forward declaration */
@@ -2353,6 +2468,32 @@ static inline void bpf_map_struct_ops_info_fill(struct bpf_map_info *info, struc
static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_ops_desc)
{
}
static inline u32 bpf_struct_ops_id(void *kdata)
{
return 0;
}
static inline void *bpf_struct_ops_map_kdata(struct bpf_map *map)
{
return NULL;
}
static inline int bpf_struct_ops_map_cgroup_atype(struct bpf_map *map)
{
return 0;
}
static inline void *bpf_struct_ops_map_cfi_stubs(struct bpf_map *map)
{
return NULL;
}
static inline bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)
{
return false;
}
static inline int bpf_struct_ops_link_update_check(struct bpf_map *new_map,
struct bpf_map *old_map,
struct bpf_map *expected_old_map)
{
return -EOPNOTSUPP;
}
#endif
@@ -2528,7 +2669,10 @@ u64 bpf_event_output(struct bpf_map *map, u64 flags, void *meta, u64 meta_size,
* since other cpus are walking the array of pointers in parallel.
*/
struct bpf_prog_array_item {
struct bpf_prog *prog;
union {
struct bpf_prog *prog;
void *kdata;
};
union {
struct bpf_cgroup_storage *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE];
u64 bpf_cookie;
@@ -2570,6 +2714,7 @@ int bpf_prog_array_copy(struct bpf_prog_array *old_array,
struct bpf_prog *include_prog,
u64 bpf_cookie,
struct bpf_prog_array **new_array);
struct bpf_prog *bpf_prog_dummy(void);
struct bpf_run_ctx {};
@@ -2588,6 +2733,7 @@ struct bpf_trace_run_ctx {
struct bpf_tramp_run_ctx {
struct bpf_run_ctx run_ctx;
u64 bpf_cookie;
int retval;
struct bpf_run_ctx *saved_run_ctx;
};
@@ -2821,8 +2967,10 @@ struct bpf_map *bpf_map_get_curr_or_next(u32 *id);
struct bpf_prog *bpf_prog_get_curr_or_next(u32 *id);
int bpf_map_alloc_pages(const struct bpf_map *map, int nid,
unsigned long nr_pages, struct page **page_array);
struct page *bpf_alloc_page(int nid, bool sleepable);
int bpf_alloc_pages(int nid, unsigned long nr_pages,
struct llist_head *pages, bool sleepable);
void bpf_free_pages(struct llist_head *pages);
#ifdef CONFIG_MEMCG
void bpf_map_memcg_enter(const struct bpf_map *map, struct mem_cgroup **old_memcg,
struct mem_cgroup **new_memcg);
@@ -3008,6 +3156,8 @@ typedef int (*bpf_iter_attach_target_t)(struct bpf_prog *prog,
union bpf_iter_link_info *linfo,
struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_detach_target_t)(struct bpf_iter_aux_info *aux);
typedef int (*bpf_iter_check_prog_t)(const struct bpf_prog *prog,
const struct bpf_iter_aux_info *aux);
typedef void (*bpf_iter_show_fdinfo_t) (const struct bpf_iter_aux_info *aux,
struct seq_file *seq);
typedef int (*bpf_iter_fill_link_info_t)(const struct bpf_iter_aux_info *aux,
@@ -3017,7 +3167,8 @@ typedef const struct bpf_func_proto *
const struct bpf_prog *prog);
enum bpf_iter_feature {
BPF_ITER_RESCHED = BIT(0),
BPF_ITER_RESCHED = BIT(0), /* may cond_resched() between objects */
BPF_ITER_NOSLEEP = BIT(1), /* ->show() can't run sleepable progs */
};
#define BPF_ITER_CTX_ARG_MAX 2
@@ -3025,6 +3176,8 @@ struct bpf_iter_reg {
const char *target;
bpf_iter_attach_target_t attach_target;
bpf_iter_detach_target_t detach_target;
/* may the program run on what the link is attached to? */
bpf_iter_check_prog_t check_prog;
bpf_iter_show_fdinfo_t show_fdinfo;
bpf_iter_fill_link_info_t fill_link_info;
bpf_iter_get_func_proto_t get_func_proto;
@@ -3061,6 +3214,8 @@ void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
struct seq_file *seq);
int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
struct bpf_link_info *info);
int bpf_iter_map_check_value_write(const struct bpf_prog *prog,
const struct bpf_map *map);
int map_set_for_each_callback_args(struct bpf_verifier_env *env,
struct bpf_func_state *caller,
@@ -3218,7 +3373,7 @@ const struct btf_func_model *
bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
const struct bpf_insn *insn);
int bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
u16 btf_fd_idx, u8 **func_addr);
u16 desc_idx, u8 **func_addr);
struct bpf_core_ctx {
struct bpf_verifier_log *log;
@@ -3561,7 +3716,7 @@ bpf_jit_find_kfunc_model(const struct bpf_prog *prog,
static inline int
bpf_get_kfunc_addr(const struct bpf_prog *prog, u32 func_id,
u16 btf_fd_idx, u8 **func_addr)
u16 desc_idx, u8 **func_addr)
{
return -ENOTSUPP;
}
@@ -3667,6 +3822,7 @@ int bpf_dev_bound_kfunc_check(struct bpf_verifier_log *log,
void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id);
int bpf_prog_dev_bound_init(struct bpf_prog *prog, union bpf_attr *attr);
int bpf_prog_dev_bound_inherit(struct bpf_prog *new_prog, struct bpf_prog *old_prog);
void bpf_prog_dev_bound_moved(struct bpf_prog *prog);
void bpf_dev_bound_netdev_unregister(struct net_device *dev);
static inline bool bpf_prog_is_dev_bound(const struct bpf_prog_aux *aux)
@@ -3728,6 +3884,10 @@ static inline int bpf_prog_dev_bound_inherit(struct bpf_prog *new_prog,
return -EOPNOTSUPP;
}
static inline void bpf_prog_dev_bound_moved(struct bpf_prog *prog)
{
}
static inline void bpf_dev_bound_netdev_unregister(struct net_device *dev)
{
}
@@ -3858,6 +4018,8 @@ struct bpf_key {
#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL)
struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags);
struct bpf_key *bpf_lookup_system_key(u64 id);
struct bpf_key *bpf_lookup_keyring(void);
bool bpf_keyring_enforced(void);
void bpf_key_put(struct bpf_key *bkey);
int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p,
const struct bpf_dynptr *sig_p,
@@ -3878,6 +4040,16 @@ static inline struct bpf_key *bpf_lookup_system_key(u64 id)
return NULL;
}
static inline struct bpf_key *bpf_lookup_keyring(void)
{
return NULL;
}
static inline bool bpf_keyring_enforced(void)
{
return false;
}
static inline void bpf_key_put(struct bpf_key *bkey)
{
}
@@ -4138,9 +4310,10 @@ void bpf_bprintf_cleanup(struct bpf_bprintf_data *data);
int bpf_try_get_buffers(struct bpf_bprintf_buffers **bufs);
void bpf_put_buffers(void);
void bpf_prog_stream_init(struct bpf_prog *prog);
int bpf_prog_stream_init(struct bpf_prog *prog, gfp_t gfp_extra_flags);
void bpf_prog_stream_free(struct bpf_prog *prog);
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, int len);
int bpf_prog_stream_read(struct bpf_prog *prog, enum bpf_stream_id stream_id, void __user *buf, u32 len);
int bpf_prog_stream_new_fd(struct bpf_prog *prog, enum bpf_stream_id stream_id, u32 flags);
void bpf_stream_stage_init(struct bpf_stream_stage *ss);
void bpf_stream_stage_free(struct bpf_stream_stage *ss);
__printf(2, 3)
@@ -4203,7 +4376,7 @@ struct bpf_prog *bpf_prog_find_from_stack(void);
int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog);
int bpf_insn_array_ready(struct bpf_map *map);
void bpf_insn_array_release(struct bpf_map *map);
void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len);
void bpf_insn_array_adjust(struct bpf_map *map, u32 first, u32 len);
void bpf_insn_array_adjust_after_remove(struct bpf_map *map, u32 off, u32 len);
#ifdef CONFIG_BPF_SYSCALL
@@ -4228,6 +4401,9 @@ static inline bool bpf_map_is_percpu_map(enum bpf_map_type map_type)
}
}
#define BPF_EXIST_FLAGS (BPF_EXIST | BPF_NOEXIST)
#define BPF_CPU_FLAGS (BPF_F_CPU | BPF_F_ALL_CPUS)
static inline int bpf_map_check_op_flags(struct bpf_map *map, u64 flags, u64 allowed_flags)
{
u32 cpu;
include/linux/bpf_compiler_types.h
+21
View File
@@ -0,0 +1,21 @@
/* SPDX-License-Identifier: GPL-2.0 */
#ifndef __LINUX_BPF_COMPILER_TYPES_H
#define __LINUX_BPF_COMPILER_TYPES_H
#include <linux/compiler_types.h>
/*
* Qualifiers for the BPF verifier rather than address spaces: __arena marks a
* pointer into a BPF arena, __nullable a pointer that may be NULL. They reach
* BPF programs as BTF type tags on the members of kernel structs shared with
* them; to kernel code and to sparse the pointers are ordinary.
*/
#ifdef __CHECKER__
# define __arena
# define __nullable
#else
# define __arena BTF_TYPE_TAG(arena)
# define __nullable BTF_TYPE_TAG(nullable)
#endif
#endif /* __LINUX_BPF_COMPILER_TYPES_H */
include/linux/bpf_crypto.h
-24
View File
@@ -1,24 +0,0 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
#ifndef _BPF_CRYPTO_H
#define _BPF_CRYPTO_H
struct bpf_crypto_type {
void *(*alloc_tfm)(const char *algo);
void (*free_tfm)(void *tfm);
int (*has_algo)(const char *algo);
int (*setkey)(void *tfm, const u8 *key, unsigned int keylen);
int (*setauthsize)(void *tfm, unsigned int authsize);
int (*encrypt)(void *tfm, const u8 *src, u8 *dst, unsigned int len, u8 *iv);
int (*decrypt)(void *tfm, const u8 *src, u8 *dst, unsigned int len, u8 *iv);
unsigned int (*ivsize)(void *tfm);
unsigned int (*statesize)(void *tfm);
u32 (*get_flags)(void *tfm);
struct module *owner;
char name[14];
};
int bpf_crypto_register_type(const struct bpf_crypto_type *type);
int bpf_crypto_unregister_type(const struct bpf_crypto_type *type);
#endif /* _BPF_CRYPTO_H */
include/linux/bpf_local_storage.h
+13
View File
@@ -104,6 +104,19 @@ struct bpf_local_storage {
refcount_t owner_refcnt;/* Used to pin owner when map_free is uncharging */
};
/*
* Drop a reference got by refcount_inc_not_zero(&local_storage->owner_refcnt).
* bpf_local_storage_destroy() may have dropped the owner's reference meanwhile
* and wait for the count to reach zero, so this one can be the last.
*/
static inline void
bpf_local_storage_unpin_owner(struct bpf_local_storage *local_storage)
{
bool last __maybe_unused;
last = refcount_dec_and_test(&local_storage->owner_refcnt);
}
/* U16_MAX is much more than enough for sk local storage
* considering a tcp_sock is ~2k.
*/
include/linux/bpf_lsm.h
+9 -2
View File
@@ -21,6 +21,13 @@ extern bool bpf_lsm_initialized __ro_after_init;
#include <linux/lsm_hook_defs.h>
#undef LSM_HOOK
/*
* Number of xattr slots the BPF LSM reserves in the array handed to
* security_inode_init_security(), i.e. the maximum number of labels
* a policy may attach to an inode while it is being created.
*/
#define BPF_LSM_INODE_INIT_XATTRS 2
struct bpf_storage_blob {
struct bpf_local_storage __rcu *storage;
};
@@ -28,7 +35,7 @@ struct bpf_storage_blob {
extern struct lsm_blob_sizes bpf_lsm_blob_sizes;
int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
const struct bpf_prog *prog);
struct bpf_prog *prog);
bool bpf_lsm_is_sleepable_hook(u32 btf_id);
bool bpf_lsm_is_trusted(const struct bpf_prog *prog);
@@ -71,7 +78,7 @@ static inline bool bpf_lsm_is_trusted(const struct bpf_prog *prog)
}
static inline int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
const struct bpf_prog *prog)
struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
include/linux/bpf_verifier.h
+430 -134
View File
File diff suppressed because it is too large. Load diff
include/linux/btf.h
+62 -3
View File
@@ -5,6 +5,7 @@
#define _LINUX_BTF_H 1
#include <linux/types.h>
#include <linux/module.h>
#include <linux/bpfptr.h>
#include <linux/bsearch.h>
#include <linux/btf_ids.h>
@@ -118,6 +119,19 @@ struct btf_show;
struct btf_id_set;
struct bpf_prog;
#define BTF_LINK_SHA256_LEN 32
/*
* A .BTF.link or .BTF.inline.link section identifies the module which
* carries a vmlinux BTF section. The module name is NUL terminated and the
* field is padded to __MODULE_NAME_LEN.
*/
struct btf_link {
char module_name[__MODULE_NAME_LEN];
u8 sha256[BTF_LINK_SHA256_LEN];
u32 btf_size;
} __packed;
typedef int (*btf_kfunc_filter_t)(const struct bpf_prog *prog, u32 kfunc_id);
struct btf_kfunc_id_set {
@@ -236,6 +250,7 @@ struct btf_record *btf_parse_fields(const struct btf *btf, const struct btf_type
u32 field_mask, u32 value_size);
int btf_check_and_fixup_fields(const struct btf *btf, struct btf_record *rec);
bool btf_type_is_void(const struct btf_type *t);
bool btf_type_is_arena_ptr(const struct btf *btf, const struct btf_type *t);
s32 btf_find_by_name_kind(const struct btf *btf, const char *name, u8 kind);
s32 bpf_find_btf_id(const char *name, u32 kind, struct btf **btf_p);
struct btf *btf_get_module_btf(const struct module *module);
@@ -261,6 +276,11 @@ const char *btf_type_str(const struct btf_type *t);
i < btf_type_vlen(datasec_type); \
i++, member++)
#define for_each_loc(i, locsec_type, member) \
for (i = 0, member = btf_type_loc_secinfo(locsec_type); \
i < btf_type_vlen(locsec_type); \
i++, member++)
static inline bool btf_type_is_ptr(const struct btf_type *t)
{
return BTF_INFO_KIND(t->info) == BTF_KIND_PTR;
@@ -329,6 +349,21 @@ static inline u64 btf_enum64_value(const struct btf_enum64 *e)
return ((u64)e->val_hi32 << 32) | e->val_lo32;
}
static inline struct btf_loc_param *btf_loc_param(const struct btf_type *t)
{
return (struct btf_loc_param *)(t + 1);
}
static inline __u32 *btf_loc_proto_params(const struct btf_type *t)
{
return (__u32 *)(t + 1);
}
static inline struct btf_loc *btf_type_loc_secinfo(const struct btf_type *t)
{
return (struct btf_loc *)(t + 1);
}
static inline bool btf_is_composite(const struct btf_type *t)
{
u16 kind = btf_kind(t);
@@ -559,7 +594,7 @@ struct btf_field_desc {
/* member struct size, or zero, if no members */
int m_sz;
/* repeated per-member offsets */
int m_off_cnt, m_offs[1];
int m_off_cnt, m_offs[2];
};
struct btf_field_iter {
@@ -573,7 +608,8 @@ struct btf_field_iter {
#ifdef CONFIG_BPF_SYSCALL
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **map_ids);
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **map_ids,
__u32 **map_strs);
int btf_field_iter_init(struct btf_field_iter *it, struct btf_type *t,
enum btf_field_iter_kind iter_kind);
__u32 *btf_field_iter_next(struct btf_field_iter *it);
@@ -603,6 +639,29 @@ bool btf_types_are_same(const struct btf *btf1, u32 id1,
const struct btf *btf2, u32 id2);
int btf_check_iter_arg(struct btf *btf, const struct btf_type *func, int arg_idx);
#if IS_ENABLED(CONFIG_SYSFS)
struct bin_attribute *sysfs_btf_add(const char *name, void *data, size_t data_size,
bool mmap, const char *lazy_module_name);
bool sysfs_btf_update(struct bin_attribute *attr, void *data);
void sysfs_btf_remove(struct bin_attribute *attr);
#else
static inline struct bin_attribute *
sysfs_btf_add(const char *name, void *data, size_t data_size, bool mmap,
const char *lazy_module_name)
{
return NULL;
}
static inline bool sysfs_btf_update(struct bin_attribute *attr, void *data)
{
return false;
}
static inline void sysfs_btf_remove(struct bin_attribute *attr)
{
}
#endif
static inline bool btf_type_is_struct_ptr(struct btf *btf, const struct btf_type *t)
{
if (!btf_type_is_ptr(t))
@@ -624,7 +683,7 @@ static inline void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
}
static inline int btf_relocate(void *log, struct btf *btf, const struct btf *base_btf,
__u32 **map_ids)
__u32 **map_ids, __u32 **map_strs)
{
return -EOPNOTSUPP;
}
include/linux/cnum.h
+3
View File
@@ -39,6 +39,7 @@ u32 cnum32_umin(struct cnum32 cnum);
u32 cnum32_umax(struct cnum32 cnum);
s32 cnum32_smin(struct cnum32 cnum);
s32 cnum32_smax(struct cnum32 cnum);
struct cnum32 cnum32_union(struct cnum32 a, struct cnum32 b);
struct cnum32 cnum32_intersect(struct cnum32 a, struct cnum32 b);
void cnum32_intersect_with(struct cnum32 *dst, struct cnum32 src);
void cnum32_intersect_with_urange(struct cnum32 *dst, u32 min, u32 max);
@@ -65,6 +66,7 @@ u64 cnum64_umin(struct cnum64 cnum);
u64 cnum64_umax(struct cnum64 cnum);
s64 cnum64_smin(struct cnum64 cnum);
s64 cnum64_smax(struct cnum64 cnum);
struct cnum64 cnum64_union(struct cnum64 a, struct cnum64 b);
struct cnum64 cnum64_intersect(struct cnum64 a, struct cnum64 b);
void cnum64_intersect_with(struct cnum64 *dst, struct cnum64 src);
void cnum64_intersect_with_urange(struct cnum64 *dst, u64 min, u64 max);
@@ -78,5 +80,6 @@ bool cnum64_is_subset(struct cnum64 outer, struct cnum64 inner);
struct cnum32 cnum32_from_cnum64(struct cnum64 cnum);
struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b);
struct cnum64 cnum64_intersect_linear(struct cnum64 in, u16 base, u16 step);
#endif /* _LINUX_CNUM_H */
include/linux/filter.h
+83
View File
@@ -98,6 +98,11 @@ struct ctl_table_header;
/* BPF program can access up to 512 bytes of stack space. */
#define MAX_BPF_STACK 512
/*
* Stack budget of a program on a JIT that lays out frames of that size.
* The interpreter and JITs without such support keep MAX_BPF_STACK.
*/
#define MAX_BPF_STACK_JIT 2048
/* Helper macros for filter block array initializers. */
@@ -249,6 +254,29 @@ static inline bool insn_is_mov_percpu_addr(const struct bpf_insn *insn)
.off = 0, \
.imm = 1 })
static inline bool bpf_is_unwind_goto(const struct bpf_insn *insn)
{
return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_UNWIND_GOTO;
}
static inline bool bpf_is_unwind_exit(const struct bpf_insn *insn)
{
return insn->code == (BPF_JMP | BPF_EXIT) && insn->src_reg == BPF_UNWIND_EXIT;
}
/*
* The caller of the main prog doesn't tell unwind_exit from exit. An extension
* is different: its caller is the caller of the subprog that it replaced.
* An exception callback returns from the frame of the main prog. After freplace
* or a tail call it returns to a prog that wasn't verified with the callback.
*/
static inline bool bpf_prog_can_unwind(const struct bpf_prog *prog)
{
if (prog->aux->exception_cb)
return false;
return bpf_is_subprog(prog) || prog->type == BPF_PROG_TYPE_EXT;
}
static inline bool insn_is_zext(const struct bpf_insn *insn)
{
return insn->code == (BPF_ALU | BPF_MOV | BPF_X) && insn->imm == 1;
@@ -1105,6 +1133,19 @@ static inline unsigned int bpf_prog_size(unsigned int proglen)
offsetof(struct bpf_prog, insns[proglen]));
}
bool bpf_prog_has_arena_insns(const struct bpf_prog *prog);
/*
* Whether a JIT has to set up the arena base for @prog. A program bound to
* an arena need not touch it, libbpf binds the arena of an object to every
* program in it. The flag caches bpf_prog_has_arena_insns() for the function
* about to be JITed, see bpf_prog_jit_compile() and jit_subprogs().
*/
static inline bool bpf_prog_uses_arena_insns(const struct bpf_prog *prog)
{
return prog->aux->uses_arena_insns;
}
static inline bool bpf_prog_was_classic(const struct bpf_prog *prog)
{
/* When classic BPF programs have been loaded and the arch
@@ -1237,16 +1278,53 @@ bool bpf_jit_inlines_helper_call(s32 imm);
bool bpf_jit_supports_subprog_tailcalls(void);
bool bpf_jit_supports_percpu_insn(void);
bool bpf_jit_supports_kfunc_call(void);
bool bpf_jit_supports_callx(void);
bool bpf_jit_supports_kfunc_ret_reg_pair(void);
bool bpf_jit_supports_unwind(void);
bool bpf_jit_supports_stack_args(void);
bool bpf_jit_supports_arena_args(void);
bool bpf_jit_supports_far_kfunc_call(void);
bool bpf_jit_supports_exceptions(void);
bool bpf_jit_supports_ptr_xchg(void);
bool bpf_jit_supports_arena(void);
bool bpf_jit_supports_arena_scalar(void);
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena);
bool bpf_jit_supports_private_stack(void);
bool bpf_jit_supports_large_stack(void);
bool bpf_jit_supports_timed_may_goto(void);
bool bpf_jit_supports_fsession(void);
struct bpf_jit_arg_abi {
/* Argument registers of the kernel convention. */
u8 nr_arg_regs;
/* Round the register number up to an even one for 16-byte alignment. */
bool even_reg_align;
/* Round the stack slot up to an even one for 16-byte alignment. */
bool even_stack_align;
/* An argument may straddle the last register and the stack. */
bool split_at_boundary;
/* A later argument may reuse a register a stack-passed one skipped. */
bool backfill_after_stack;
};
const struct bpf_jit_arg_abi *bpf_jit_arg_abi(void);
u32 bpf_jit_place_args(const struct bpf_jit_arg_abi *abi,
const struct btf_func_model *fm, u8 *pos_of_slot);
/* The JIT's scratch register, in place of an argument slot. */
#define BPF_JIT_ARG_TMP 0xff
/* Every argument slot moves at most once, and the scratch goes out and back. */
#define BPF_JIT_MAX_ARG_MOVES (MAX_BPF_FUNC_ARG_SLOTS + 2)
struct bpf_jit_arg_move {
u8 dst;
u8 src;
};
u32 bpf_jit_plan_arg_moves(const struct bpf_jit_arg_abi *abi,
const struct btf_func_model *fm,
struct bpf_jit_arg_move *moves);
u64 bpf_arch_uaddress_limit(void);
void arch_bpf_stack_walk(bool (*consume_fn)(void *cookie, u64 ip, u64 sp, u64 bp), void *cookie);
u64 arch_bpf_timed_may_goto(void);
@@ -1897,6 +1975,11 @@ static __always_inline long __bpf_xdp_redirect_map(struct bpf_map *map, u64 inde
return XDP_REDIRECT;
}
int __bpf_sock_ops_load_hdr_opt(struct bpf_sock_ops_kern *bpf_sock,
void *search_res, u32 len, u64 flags);
int __bpf_sock_ops_store_hdr_opt(struct bpf_sock_ops_kern *bpf_sock,
const void *from, u32 len, u64 flags);
#ifdef CONFIG_NET
int __bpf_skb_load_bytes(const struct sk_buff *skb, u32 offset, void *to, u32 len);
int __bpf_skb_store_bytes(struct sk_buff *skb, u32 offset, const void *from,
include/linux/ftrace.h
+1
View File
@@ -416,6 +416,7 @@ void free_ftrace_hash(struct ftrace_hash *hash);
struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash,
unsigned long ip, unsigned long direct);
void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry);
struct ftrace_func_entry *ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip);
void ftrace_hash_remove(struct ftrace_hash *hash);
/* The hash used to know what functions callbacks trace */
include/linux/key.h
+2
View File
@@ -445,6 +445,8 @@ extern key_ref_t keyring_search(key_ref_t keyring,
extern int keyring_restrict(key_ref_t keyring, const char *type,
const char *restriction);
extern void key_register_bpf_keyring(struct key *keyring);
extern struct key *key_lookup(key_serial_t id);
static inline key_serial_t key_serial(const struct key *key)
include/linux/mm.h
+4 -4
View File
@@ -3314,10 +3314,10 @@ extern int access_process_vm(struct task_struct *tsk, unsigned long addr,
extern int access_remote_vm(struct mm_struct *mm, unsigned long addr,
void *buf, int len, unsigned int gup_flags);
#ifdef CONFIG_BPF_SYSCALL
extern int copy_remote_vm_str(struct task_struct *tsk, unsigned long addr,
void *buf, int len, unsigned int gup_flags);
#endif
int copy_remote_mm_str(struct mm_struct *mm, unsigned long addr,
void *buf, int len, unsigned int gup_flags);
int copy_remote_vm_str(struct task_struct *tsk, unsigned long addr,
void *buf, int len, unsigned int gup_flags);
long get_user_pages_remote(struct mm_struct *mm,
unsigned long start, unsigned long nr_pages,
include/linux/module.h
+4
View File
@@ -503,6 +503,10 @@ struct module {
void *btf_data;
void *btf_base_data;
#endif
#if IS_ENABLED(CONFIG_DEBUG_INFO_BTF_INLINE)
unsigned int btf_inline_data_size;
void *btf_inline_data;
#endif
#ifdef CONFIG_JUMP_LABEL
struct jump_entry *jump_entries;
unsigned int num_jump_entries;
include/linux/tcp.h
+7
View File
@@ -233,6 +233,13 @@ struct tcp_sock {
is_sack_reneg:1, /* in recovery from loss with SACK reneg? */
is_cwnd_limited:1,/* forward progress limited by snd_cwnd? */
recvmsg_inq : 1;/* Indicate # of bytes in queue upon recvmsg */
#ifdef CONFIG_BPF
u32 bpf_tcp_ops_flags;
#define BPF_TCP_OPS_TEST_FLAG(TP, ARG) \
(READ_ONCE((TP)->bpf_tcp_ops_flags) & BPF_TCP_OPS_FLAG_ ## ARG)
#else
#define BPF_TCP_OPS_TEST_FLAG(TP, ARG) (0)
#endif
__cacheline_group_end(tcp_sock_read_txrx);
/* RX read-mostly hotpath cache lines */
include/linux/tnum.h
+2
View File
@@ -134,4 +134,6 @@ static inline bool tnum_subreg_is_const(struct tnum a)
/* Returns the smallest member of t larger than z */
u64 tnum_step(struct tnum t, u64 z);
u32 tnum_alignment(struct tnum a);
#endif /* _LINUX_TNUM_H */
include/net/tcp.h
+213 -33
View File
@@ -512,6 +512,7 @@ void tcp_set_keepalive(struct sock *sk, int val);
void tcp_syn_ack_timeout(const struct request_sock *req);
int tcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
int flags);
int __tcp_set_rcvlowat(struct sock *sk, int val, bool wakeup);
int tcp_set_rcvlowat(struct sock *sk, int val);
void tcp_set_rcvbuf(struct sock *sk, int val);
int tcp_set_window_clamp(struct sock *sk, int val);
@@ -2930,7 +2931,7 @@ static inline void bpf_skops_init_skb(struct bpf_sock_ops_kern *skops,
* program loaded).
*/
#ifdef CONFIG_BPF
static inline int tcp_call_bpf(struct sock *sk, int op, u32 nargs, u32 *args)
static inline int __tcp_call_bpf(struct sock *sk, int op, u32 nargs, u32 *args)
{
struct bpf_sock_ops_kern sock_ops;
int ret;
@@ -2947,7 +2948,7 @@ static inline int tcp_call_bpf(struct sock *sk, int op, u32 nargs, u32 *args)
if (nargs > 0)
memcpy(sock_ops.args, args, nargs * sizeof(*args));
ret = BPF_CGROUP_RUN_PROG_SOCK_OPS(&sock_ops);
ret = __BPF_CGROUP_RUN_PROG_SOCK_OPS(&sock_ops);
if (ret == 0)
ret = sock_ops.reply;
else
@@ -2955,42 +2956,37 @@ static inline int tcp_call_bpf(struct sock *sk, int op, u32 nargs, u32 *args)
return ret;
}
static inline int tcp_call_bpf_2arg(struct sock *sk, int op, u32 arg1, u32 arg2)
{
u32 args[2] = {arg1, arg2};
#define tcp_call_bpf(sk, op) \
({ \
int __ret = 0; \
if (cgroup_bpf_enabled(CGROUP_SOCK_OPS)) { \
__ret = __tcp_call_bpf(sk, op, 0, NULL); \
} \
__ret; \
})
return tcp_call_bpf(sk, op, 2, args);
}
static inline int tcp_call_bpf_3arg(struct sock *sk, int op, u32 arg1, u32 arg2,
u32 arg3)
{
u32 args[3] = {arg1, arg2, arg3};
return tcp_call_bpf(sk, op, 3, args);
}
#define tcp_call_bpf_flag(sk, op, ...) \
do { \
if (cgroup_bpf_enabled(CGROUP_SOCK_OPS) && \
BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk), op ## _FLAG)) { \
u32 __args[] = { __VA_ARGS__ }; \
__tcp_call_bpf(sk, op, ARRAY_SIZE(__args), __args); \
} \
} while (0)
static inline void tcp_clear_sock_ops_cb_flags(struct sock *sk)
{
tcp_sk(sk)->bpf_sock_ops_cb_flags = 0;
WRITE_ONCE(tcp_sk(sk)->bpf_tcp_ops_flags, 0);
}
#else
static inline int tcp_call_bpf(struct sock *sk, int op, u32 nargs, u32 *args)
static inline int tcp_call_bpf(struct sock *sk, int op)
{
return -EPERM;
}
static inline int tcp_call_bpf_2arg(struct sock *sk, int op, u32 arg1, u32 arg2)
{
return -EPERM;
}
static inline int tcp_call_bpf_3arg(struct sock *sk, int op, u32 arg1, u32 arg2,
u32 arg3)
{
return -EPERM;
}
#define tcp_call_bpf_flag(sk, op, ...) do { } while (0)
static inline void tcp_clear_sock_ops_cb_flags(struct sock *sk)
{
@@ -2998,12 +2994,196 @@ static inline void tcp_clear_sock_ops_cb_flags(struct sock *sk)
#endif
#if defined(CONFIG_BPF_JIT) && defined(CONFIG_CGROUP_BPF)
struct bpf_tcp_ops {
/* Should return the initial SYN (active open) or SYN-ACK (passive open)
* retransmission timeout. Return the timeout in jiffies, or <= 0 for
* the kernel default.
*
* @req: request_sock on the passive (synack) path; NULL otherwise.
*/
int (*timeout_init)(struct sock *sk, struct request_sock *req);
/* Should return the initial advertised receive window, in packets,
* or < 0 for the kernel default. @req as in timeout_init().
*/
int (*rwnd_init)(struct sock *sk, struct request_sock *req);
/* Called when an active connection becomes established.
* @skb is the SYNACK that completed the 3WHS, or NULL for a
* TCP_REPAIR socket (tcp_finish_connect() with no skb).
*/
void (*active_established)(struct sock *sk, struct sk_buff *skb__nullable);
/* Called when a passive connection becomes established.
* @skb is the ACK that completed the 3WHS.
*/
void (*passive_established)(struct sock *sk, struct sk_buff *skb);
/* Called when the retransmission timer fires. */
void (*rto)(struct sock *sk);
/* Called on every RTT sample if BPF_TCP_OPS_FLAG_RTT is enabled.
* @mrtt: the measured RTT, in microseconds.
* @srtt: the updated smoothed RTT.
*/
void (*rtt)(struct sock *sk, long mrtt, u32 srtt);
/* Called when the connection changes TCP state.
* @state: the new state (one of the TCP_* states).
*/
void (*set_state)(struct sock *sk, int state);
/* Called when an skb is retransmitted.
* @skb: the retransmitted skb.
* @err: tcp_transmit_skb() return value (0 on success).
*/
void (*retrans)(struct sock *sk, struct sk_buff *skb, int err);
/* Called right before an active connection is initialized. */
void (*connect)(struct sock *sk);
/* Called on listen(2), right after the socket enters TCP_LISTEN. */
void (*listen)(struct sock *sk);
/*
* Parse the TCP header options of an incoming skb received on an
* established connection. Use bpf_dynptr_from_skb()/bpf_skb_load_bytes()
* to access the options.
*
* Called if BPF_TCP_OPS_FLAG_PARSE_HDR_OPT_ALL is enabled, or if
* BPF_TCP_OPS_FLAG_PARSE_HDR_OPT_UNKNOWN is enabled and an unknown
* option is received.
*/
void (*parse_hdr)(struct sock *sk, struct sk_buff *skb);
/*
* Reserve space in the outgoing TCP header for options to be written
* later by write_hdr_opt(). Call bpf_reserve_hdr_opt() to reserve bytes.
*
* Called if BPF_TCP_OPS_FLAG_WRITE_HDR_OPT is enabled.
*
* @skb: outgoing packet. NULL when called from tcp_current_mss()
* (MSS sizing).
* @req: request_sock on the synack path; NULL otherwise.
* @syn_skb: incoming SYN on the synack path; NULL otherwise.
* @synack_type: TCP_SYNACK_COOKIE indicates a stateless syncookie.
* @remaining: pointer to the size of space still available; cast it
* using bpf_rdonly_cast() before dereferencing.
*/
void (*hdr_opt_len)(struct sock *sk, struct sk_buff *skb,
struct request_sock *req, struct sk_buff *syn_skb,
enum tcp_synack_type synack_type,
unsigned int *remaining);
/*
* Write header options into the space reserved earlier by hdr_opt_len().
* Use bpf_store_hdr_opt() to write; it appends within the reserved window
* shared with legacy SOCKOPS.
*
* Called if BPF_TCP_OPS_FLAG_WRITE_HDR_OPT is enabled.
*
* @skb: outgoing packet.
* @req: request_sock on the synack path; NULL otherwise.
* @syn_skb: incoming SYN on the synack path; NULL otherwise.
* @synack_type: TCP_SYNACK_COOKIE indicates a stateless syncookie.
* @opt_off: offset in the outgoing @skb's TCP header where the
* bpf_tcp_ops portion of the reserved window begins, i.e. after
* the kernel and legacy options.
*/
void (*write_hdr_opt)(struct sock *sk, struct sk_buff *skb,
struct request_sock *req, struct sk_buff *syn_skb,
enum tcp_synack_type synack_type,
u32 opt_off);
/*
* Called when an incoming skb is enqueued to sk->sk_receive_queue
* if BPF_TCP_OPS_FLAG_RCVQ is enabled.
*/
void (*enqueue_rcvq)(struct sock *sk, struct sk_buff *skb);
/*
* Called after data is dequeued from sk->sk_receive_queue
* if BPF_TCP_OPS_FLAG_RCVQ is enabled.
*/
void (*dequeue_rcvq)(struct sock *sk);
};
#define __bpf_tcp_ops_call(op, sk, ...) \
do { \
const struct bpf_prog_array_item *item; \
const struct bpf_tcp_ops *tcp_ops; \
struct cgroup *cgrp; \
\
cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data); \
rcu_read_lock_dont_migrate(); \
bpf_cgroup_struct_ops_foreach(tcp_ops, item, cgrp, \
CGROUP_TCP_SOCK_OPS) { \
if (tcp_ops->op) \
tcp_ops->op(sk, ##__VA_ARGS__); \
} \
rcu_read_unlock_migrate(); \
} while (0)
#define bpf_tcp_ops_call(op, sk, ...) \
do { \
if (cgroup_bpf_enabled(CGROUP_TCP_SOCK_OPS)) \
__bpf_tcp_ops_call(op, sk, ##__VA_ARGS__); \
} while (0)
#define bpf_tcp_ops_call_flag(op, flag, sk, ...) \
do { \
if (cgroup_bpf_enabled(CGROUP_TCP_SOCK_OPS) && \
BPF_TCP_OPS_TEST_FLAG(tcp_sk(sk), flag)) \
__bpf_tcp_ops_call(op, sk, ##__VA_ARGS__); \
} while (0)
#define bpf_tcp_ops_call_int(op, init_retval, sk, ...) \
({ \
int __retval = (init_retval); \
if (cgroup_bpf_enabled(CGROUP_TCP_SOCK_OPS)) { \
const struct bpf_prog_array_item *item; \
const struct bpf_tcp_ops *tcp_ops; \
struct bpf_tramp_run_ctx run_ctx; \
struct bpf_run_ctx *old_run_ctx; \
struct sock *__sk = sk_to_full_sk(sk); \
struct request_sock *req = NULL; \
struct cgroup *cgrp; \
\
if (__sk) { \
run_ctx.retval = (init_retval); \
cgrp = sock_cgroup_ptr(&__sk->sk_cgrp_data); \
if (!sk_fullsock(sk)) \
req = (struct request_sock *)sk; \
rcu_read_lock_dont_migrate(); \
old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx);\
bpf_cgroup_struct_ops_foreach(tcp_ops, item, cgrp, \
CGROUP_TCP_SOCK_OPS) { \
if (tcp_ops->op) \
run_ctx.retval = tcp_ops->op(__sk, req, ##__VA_ARGS__); \
} \
bpf_reset_run_ctx(old_run_ctx); \
rcu_read_unlock_migrate(); \
__retval = run_ctx.retval; \
} \
} \
__retval; \
})
#else
#define __bpf_tcp_ops_call(op, sk, ...) do { } while (0)
#define bpf_tcp_ops_call(op, sk, ...) do { } while (0)
#define bpf_tcp_ops_call_flag(op, flag, sk, ...) do { } while (0)
#define bpf_tcp_ops_call_int(op, init_retval, sk, ...) (init_retval)
#endif
static inline u32 tcp_timeout_init(struct sock *sk)
{
int timeout;
timeout = tcp_call_bpf(sk, BPF_SOCK_OPS_TIMEOUT_INIT, 0, NULL);
timeout = tcp_call_bpf(sk, BPF_SOCK_OPS_TIMEOUT_INIT);
timeout = bpf_tcp_ops_call_int(timeout_init, timeout, sk);
if (timeout <= 0)
timeout = TCP_TIMEOUT_INIT;
return min_t(int, timeout, TCP_RTO_MAX);
@@ -3013,8 +3193,8 @@ static inline u32 tcp_rwnd_init_bpf(struct sock *sk)
{
int rwnd;
rwnd = tcp_call_bpf(sk, BPF_SOCK_OPS_RWND_INIT, 0, NULL);
rwnd = tcp_call_bpf(sk, BPF_SOCK_OPS_RWND_INIT);
rwnd = bpf_tcp_ops_call_int(rwnd_init, rwnd, sk);
if (rwnd < 0)
rwnd = 0;
return rwnd;
@@ -3022,13 +3202,13 @@ static inline u32 tcp_rwnd_init_bpf(struct sock *sk)
static inline bool tcp_bpf_ca_needs_ecn(struct sock *sk)
{
return (tcp_call_bpf(sk, BPF_SOCK_OPS_NEEDS_ECN, 0, NULL) == 1);
return (tcp_call_bpf(sk, BPF_SOCK_OPS_NEEDS_ECN) == 1);
}
static inline void tcp_bpf_rtt(struct sock *sk, long mrtt, u32 srtt)
{
if (BPF_SOCK_OPS_TEST_FLAG(tcp_sk(sk), BPF_SOCK_OPS_RTT_CB_FLAG))
tcp_call_bpf_2arg(sk, BPF_SOCK_OPS_RTT_CB, mrtt, srtt);
tcp_call_bpf_flag(sk, BPF_SOCK_OPS_RTT_CB, mrtt, srtt);
bpf_tcp_ops_call_flag(rtt, RTT, sk, mrtt, srtt);
}
#if IS_ENABLED(CONFIG_SMC)
include/uapi/linux/bpf.h
+112 -14
View File
@@ -56,6 +56,19 @@
enum bpf_cond_pseudo_jmp {
BPF_MAY_GOTO = 0,
/*
* Follows a call of a bpf subprog. Taken when the subprog left with
* BPF_EXIT that has src_reg == BPF_UNWIND_EXIT.
*/
BPF_UNWIND_GOTO = 1,
};
enum bpf_pseudo_exit {
/*
* When bpf_exit->src_reg == BPF_UNWIND_EXIT, it's the same as bpf_exit,
* except that BPF_UNWIND_GOTO after the call is taken.
*/
BPF_UNWIND_EXIT = 1,
};
/* Register numbers */
@@ -936,6 +949,33 @@ union bpf_iter_link_info {
* 0 on success or -1 if an error occurred (in which case,
* *errno* is set appropriately).
*
* BPF_PROG_STREAM_OPEN
* Description
* Open a file descriptor for one of the BPF streams associated
* with the program identified by *prog_fd*. The stream is selected
* by *stream_id*.
*
* The returned file descriptor supports **read**\ (2) and
* **poll**\ (2). Reads block while the stream is empty unless
* **BPF_F_STREAM_NONBLOCK** is specified in *flags*. A non-blocking
* read of an empty stream fails with **EAGAIN**.
*
* **poll**\ (2) reports **POLLIN** when data is available and
* **POLLHUP** once the program has been freed, that is, after every
* reference to it, including links and other file descriptors, has
* been dropped. Hangup may lag the final release because program
* teardown is deferred. Buffered data remains readable after
* **POLLHUP** and a read returns zero after all such data has been
* consumed.
*
* The file descriptor is read-only and has the close-on-exec flag
* set. It is not seekable and **lseek**\ (2) fails with **ESPIPE**.
* *flags* may only contain **BPF_F_STREAM_NONBLOCK**.
*
* Return
* A new file descriptor (a nonnegative integer), or -1 if an
* error occurred (in which case, *errno* is set appropriately).
*
* NOTES
* eBPF objects (maps and programs) can be shared between processes.
*
@@ -993,6 +1033,7 @@ enum bpf_cmd {
BPF_TOKEN_CREATE,
BPF_PROG_STREAM_READ_BY_FD,
BPF_PROG_ASSOC_STRUCT_OPS,
BPF_PROG_STREAM_OPEN,
__MAX_BPF_CMD,
BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */
};
@@ -1316,6 +1357,12 @@ enum bpf_perf_event_type {
/* The verifier internal test flag. Behavior is undefined */
#define BPF_F_TEST_REG_INVARIANTS (1U << 7)
/*
* Load and store through a number is an access to the arena of the program
* at the low 32 bits of the number. It's for programs written in Rust.
*/
#define BPF_F_ARENA_SCALAR (1U << 8)
/* link_create.kprobe_multi.flags used in LINK_CREATE command for
* BPF_TRACE_KPROBE_MULTI attach type to create return probe.
*/
@@ -1524,6 +1571,11 @@ enum {
BPF_STREAM_STDERR = 2,
};
/* flags for BPF_PROG_STREAM_OPEN command */
enum {
BPF_F_STREAM_NONBLOCK = (1U << 0),
};
union bpf_attr {
struct { /* anonymous struct used by BPF_MAP_CREATE command */
__u32 map_type; /* one of enum bpf_map_type */
@@ -1756,7 +1808,7 @@ union bpf_attr {
__u32 prog_cnt;
__u32 count;
};
__u32 :32;
__u32 type_id;
/* output: per-program attach_flags.
* not allowed to be set during effective query.
*/
@@ -1950,6 +2002,12 @@ union bpf_attr {
__u32 flags;
} prog_assoc_struct_ops;
struct {
__u32 prog_fd;
__u32 stream_id;
__u32 flags;
} prog_stream_open;
} __attribute__((aligned(8)));
/* The description below is an attempt at providing documentation to eBPF
@@ -4867,15 +4925,18 @@ union bpf_attr {
* The non-negative copied *buf* length equal to or less than
* *size* on success, or a negative error in case of failure.
*
* long bpf_load_hdr_opt(struct bpf_sock_ops *skops, void *searchby_res, u32 len, u64 flags)
* long bpf_load_hdr_opt(void *ctx, void *searchby_res, u32 len, u64 flags)
* Description
* Load header option. Support reading a particular TCP header
* option for bpf program (**BPF_PROG_TYPE_SOCK_OPS**).
* option for bpf program (**BPF_PROG_TYPE_SOCK_OPS**). For the
* **bpf_tcp_ops** struct_ops, this helper can be called from the
* **parse_hdr**\ () and **write_hdr_opt**\ () operators.
*
* If *flags* is 0, it will search the option from the
* *skops*\ **->skb_data**. The comment in **struct bpf_sock_ops**
* has details on what skb_data contains under different
* *skops*\ **->op**.
* If *flags* is 0, it will search the option from the packet
* associated with the current operation. For
* **BPF_PROG_TYPE_SOCK_OPS**, the comment in
* **struct bpf_sock_ops** has details on what skb_data
* contains under different *op*.
*
* The first byte of the *searchby_res* specifies the
* kind that it wants to search.
@@ -4908,6 +4969,8 @@ union bpf_attr {
*
* * **BPF_LOAD_HDR_OPT_TCP_SYN** to search from the
* saved_syn packet or the just-received syn packet.
* Not supported by the **bpf_tcp_ops** struct_ops, which
* rejects all flags.
*
* Return
* > 0 when found, the header option is copied to *searchby_res*.
@@ -4928,9 +4991,9 @@ union bpf_attr {
* packet.
*
* **-EPERM** if the helper cannot be used under the current
* *skops*\ **->op**.
* operation.
*
* long bpf_store_hdr_opt(struct bpf_sock_ops *skops, const void *from, u32 len, u64 flags)
* long bpf_store_hdr_opt(void *ctx, const void *from, u32 len, u64 flags)
* Description
* Store header option. The data will be copied
* from buffer *from* with length *len* to the TCP header.
@@ -4946,7 +5009,9 @@ union bpf_attr {
* by searching the same option in the outgoing skb.
*
* This helper can only be called during
* **BPF_SOCK_OPS_WRITE_HDR_OPT_CB**.
* **BPF_SOCK_OPS_WRITE_HDR_OPT_CB**, or from the
* **write_hdr_opt**\ () operator of the **bpf_tcp_ops**
* struct_ops.
*
* Return
* 0 on success, or negative error in case of failure:
@@ -4961,9 +5026,9 @@ union bpf_attr {
* **-EFAULT** on failure to parse the existing header options.
*
* **-EPERM** if the helper cannot be used under the current
* *skops*\ **->op**.
* operation.
*
* long bpf_reserve_hdr_opt(struct bpf_sock_ops *skops, u32 len, u64 flags)
* long bpf_reserve_hdr_opt(void *ctx, u32 len, u64 flags)
* Description
* Reserve *len* bytes for the bpf header option. The
* space will be used by **bpf_store_hdr_opt**\ () later in
@@ -4973,7 +5038,9 @@ union bpf_attr {
* the total number of bytes will be reserved.
*
* This helper can only be called during
* **BPF_SOCK_OPS_HDR_OPT_LEN_CB**.
* **BPF_SOCK_OPS_HDR_OPT_LEN_CB**, or from the
* **hdr_opt_len**\ () operator of the **bpf_tcp_ops**
* struct_ops.
*
* Return
* 0 on success, or negative error in case of failure:
@@ -4983,7 +5050,7 @@ union bpf_attr {
* **-ENOSPC** if there is not enough space in the header.
*
* **-EPERM** if the helper cannot be used under the current
* *skops*\ **->op**.
* operation.
*
* void *bpf_inode_storage_get(struct bpf_map *map, void *inode, void *value, u64 flags)
* Description
@@ -6890,6 +6957,8 @@ struct bpf_link_info {
} xdp;
struct {
__u32 map_id;
__u32 :32;
__u64 cgroup_id;
} struct_ops;
struct {
__u32 pf;
@@ -7284,6 +7353,31 @@ enum {
*/
};
/*
* Most callbacks of struct bpf_tcp_ops are placed in the slow
* path (e.g., one-shot connection setup or unlikely events like
* timers) and are invoked simply by defining non-NULL callbacks.
*
* Callbacks in the fast path, however, would incur noticeable
* overhead even when set to NULL, so they are disabled by default
* and must be explicitly enabled per socket via bpf_tcp_ops_set_flags().
*
* The flags are per-socket and shared by all effective bpf_tcp_ops
* programs.
*/
enum {
/* .rtt() */
BPF_TCP_OPS_FLAG_RTT = (1 << 0),
/* .parse_hdr() */
BPF_TCP_OPS_FLAG_PARSE_HDR_OPT_ALL = (1 << 1),
BPF_TCP_OPS_FLAG_PARSE_HDR_OPT_UNKNOWN = (1 << 2),
/* .hdr_opt_len() and .write_hdr_opt() */
BPF_TCP_OPS_FLAG_WRITE_HDR_OPT = (1 << 3),
/* .enqueue_rcvq() and .dequeue_rcvq() */
BPF_TCP_OPS_FLAG_RCVQ = (1 << 4),
BPF_TCP_OPS_FLAG_ALL = (1 << 5) - 1,
};
/* List of TCP states. There is a build check in net/ipv4/tcp.c to detect
* changes between the TCP and BPF versions. Ideally this should never happen.
* If it does, we need to add code to convert them before calling
@@ -7600,6 +7694,10 @@ struct bpf_task_work {
__u64 __opaque;
} __attribute__((aligned(8)));
struct bpf_rcu_head {
__u64 __opaque[6];
} __attribute__((aligned(8)));
struct bpf_wq {
__u64 __opaque[2];
} __attribute__((aligned(8)));
include/uapi/linux/btf.h
+80 -3
View File
@@ -54,8 +54,8 @@ struct btf_type {
* decl_tag and type_tag
*/
__u32 info;
/* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC and ENUM64.
* "size" tells the size of the type it is describing.
/* "size" is used by INT, ENUM, STRUCT, UNION, DATASEC, ENUM64
* and LOC_PARAM. "size" tells the size of the type it is describing.
*
* "type" is used by PTR, TYPEDEF, VOLATILE, CONST, RESTRICT,
* FUNC, FUNC_PROTO, VAR, DECL_TAG and TYPE_TAG.
@@ -92,7 +92,9 @@ enum {
BTF_KIND_DECL_TAG = 17, /* Decl Tag */
BTF_KIND_TYPE_TAG = 18, /* Type Tag */
BTF_KIND_ENUM64 = 19, /* Enumeration up to 64-bit values */
BTF_KIND_LOC_PARAM = 20, /* Location parameter information */
BTF_KIND_LOC_PROTO = 21, /* Location prototype for site */
BTF_KIND_LOCSEC = 22, /* Location section */
NR_BTF_KINDS,
BTF_KIND_MAX = NR_BTF_KINDS - 1,
};
@@ -212,4 +214,79 @@ struct btf_enum64 {
__u32 val_hi32;
};
/*
* BTF_KIND_LOC_PARAM is followed by a single "struct btf_loc_param"
* that contains flags specifying the contents of the vlen-specified
* number of 4-byte values that follow.
*/
struct btf_loc_param {
__u32 flags;
__u32 values[];
};
/*
* The combination of size, vlen and flags gives us the means to interpret
* the following vlen-specified set of 4-byte values:
*
* - a BTF_LOC_PARAM_CONST is a constant value; combination
* of size, vlen and _SIGNED flag determines it. If the value requires
* 64 bits it is stored in {lo,hi} order.
* - a BTF_LOC_PARAM_ADDR|BTF_LOC_PARAM_CONST is an address that should be
* normalized with respect to kernel/module base address.
* - a BTF_LOC_PARAM_REG with vlen 1 is a simple register number;
* with vlen 2 it is a multi-register parameter. Register numbers are
* numbers derived from DW_OP_reg values, i.e. 0 is DW_OP_reg0; since
* DW_OP_fbreg has its own special DW_OP_value and DW_OP_regx can refer
* to an arbitrary register number, we reserve BTF_LOC_PARAM_FBREG for
* the frame base register to avoid collisions.
* - a _REG | OFFSET describes an address without dereferencing it.
* - a _REG | DEREF with vlen 1 dereferences the value in the register
* number specified.
* - a REG | DEREF | OFFSET with vlen > 1 specifies the register number
* in the first 4-byte value and the offset in the remainder.
* In the case of REG and OFFSET combinations, the OFFSET has the width
* of the value words while the type size describes the represented parameter,
* so for example a REG | DEREF | OFFSET with size 8 and vlen 2 would be
* an 8-byte register dereference with signed 4-byte offset, since the vlen 2
* values consist of a register value and the signed value.
*/
enum btf_loc_param_flags {
BTF_LOC_PARAM_SIGNED = 0x1,
BTF_LOC_PARAM_CONST = 0x2,
BTF_LOC_PARAM_ADDR = 0x4,
BTF_LOC_PARAM_REG = 0x8,
BTF_LOC_PARAM_DEREF = 0x10,
BTF_LOC_PARAM_OFFSET = 0x20,
};
enum {
BTF_LOC_PARAM_FBREG = 0xffffffff
};
/*
* BTF_KIND_LOC_PROTO specifies location prototypes; i.e. how locations relate
* to parameters; a struct btf_type of BTF_KIND_LOC_PROTO is followed by a
* vlen-specified number of __u32 BTF type ids which specify the associated
* BTF_KIND_LOC_PARAM for each function parameter associated with the
* location. The type should either be 0 (no location info) or point at
* a BTF_KIND_LOC_PARAM.
*/
/*
* BTF_KIND_LOCSEC consists of vlen-specified number of "struct btf_loc"
* containing location site-specific information for a specific ELF section;
* for example locations in ".text" are in a LOCSEC named "inline.text".
*
* - function (func)
* - location prototype type id (loc_proto)
* - address offset (offset) relative to the runtime base address of the
* ELF section associated with the LOCSEC
*/
struct btf_loc {
__u32 func;
__u32 loc_proto;
__u32 offset;
};
#endif /* _UAPI__LINUX_BTF_H__ */
include/uapi/linux/keyctl.h
+1
View File
@@ -24,6 +24,7 @@
#define KEY_SPEC_GROUP_KEYRING -6 /* - key ID for GID-specific keyring */
#define KEY_SPEC_REQKEY_AUTH_KEY -7 /* - key ID for assumed request_key auth key */
#define KEY_SPEC_REQUESTOR_KEYRING -8 /* - key ID for request_key() dest keyring */
#define KEY_SPEC_BPF_KEYRING -9 /* - key ID for the BPF-specific keyring */
/* request-key default keyrings */
#define KEY_REQKEY_DEFL_NO_CHANGE -1
kernel/bpf/Kconfig
+26
View File
@@ -17,6 +17,10 @@ config HAVE_CBPF_JIT
config HAVE_EBPF_JIT
bool
# KASAN support for JIT compiler
config HAVE_EBPF_JIT_KASAN
bool
# Used by archs to tell that they want the BPF JIT compiler enabled by
# default for kernels that were compiled with BPF JIT support.
config ARCH_WANT_DEFAULT_BPF_JIT
@@ -87,6 +91,15 @@ config BPF_UNPRIV_DEFAULT_OFF
If you are unsure how to answer this question, answer Y.
config BPF_CRYPTO
def_bool y
depends on BPF_SYSCALL
select CRYPTO_LIB_AES_CBC
select CRYPTO_LIB_AES_ECB
help
Provide the kfuncs needed for BPF programs to encrypt and decrypt
data. The supported algorithms are AES-CBC and AES-ECB.
source "kernel/bpf/preload/Kconfig"
config BPF_LSM
@@ -101,4 +114,17 @@ config BPF_LSM
If you are unsure how to answer this question, answer N.
config BPF_JIT_KASAN
bool
depends on HAVE_EBPF_JIT_KASAN
depends on KASAN_GENERIC
depends on KASAN_VMALLOC
depends on BPF_JIT
default y
help
Makes JIT compiler insert generic outline KASAN checks in BPF
programs when they are inserted in the kernel. This feature is
automatically enabled if the needed set of KASAN and BPF
configuration options is enabled.
endmenu # "BPF subsystem"
kernel/bpf/Makefile
+6 -4
View File
@@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse
endif
CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy)
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o
obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o diagnostics.o loops.o scev.o
obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o
obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o
obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o
@@ -27,6 +27,9 @@ obj-$(CONFIG_BPF_SYSCALL) += offload.o
obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
obj-$(CONFIG_BPF_SYSCALL) += tcx.o
endif
ifeq ($(CONFIG_KEYS),y)
obj-$(CONFIG_BPF_SYSCALL) += keys.o
endif
ifeq ($(CONFIG_PERF_EVENTS),y)
obj-$(CONFIG_BPF_SYSCALL) += stackmap.o
endif
@@ -39,6 +42,7 @@ obj-$(CONFIG_BPF_SYSCALL) += reuseport_array.o
endif
ifeq ($(CONFIG_SYSFS),y)
obj-$(CONFIG_DEBUG_INFO_BTF) += sysfs_btf.o
obj-$(CONFIG_DEBUG_INFO_BTF_INLINE) += btf_vmlinux_inline.o
endif
ifeq ($(CONFIG_BPF_JIT),y)
obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
@@ -55,9 +59,7 @@ obj-$(CONFIG_BPF_SYSCALL) += cpumask.o
# semantics within pahole are revisited accordingly.
obj-${CONFIG_BPF_LSM} += bpf_lsm_proto.o bpf_lsm.o
endif
ifneq ($(CONFIG_CRYPTO),)
obj-$(CONFIG_BPF_SYSCALL) += crypto.o
endif
obj-$(CONFIG_BPF_CRYPTO) += crypto.o
obj-$(CONFIG_BPF_PRELOAD) += preload/
obj-$(CONFIG_BPF_SYSCALL) += relo_core.o
kernel/bpf/arena.c
+229 -181
View File
@@ -146,7 +146,7 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr)
struct apply_range_data {
struct bpf_arena *arena;
struct page **pages;
struct llist_head *pages;
int i;
};
@@ -158,13 +158,17 @@ struct clear_range_data {
static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
{
struct apply_range_data *d = data;
struct llist_node *node;
struct page *page;
pte_t pteval;
if (!data)
return 0;
page = d->pages[d->i];
node = READ_ONCE(d->pages->first);
if (WARN_ON_ONCE(!node))
return -EINVAL;
page = llist_entry(node, struct page, pcp_llist);
/* paranoia, similar to vmap_pages_pte_range() */
if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page))))
return -EINVAL;
@@ -197,6 +201,7 @@ static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data)
return -EBUSY;
set_pte_at(&init_mm, addr, pte, pteval);
#endif
WARN_ON_ONCE(llist_del_first(d->pages) != node);
d->i++;
WRITE_ONCE(d->arena->nr_pages, d->arena->nr_pages + 1);
return 0;
@@ -312,8 +317,8 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)
INIT_WORK(&arena->free_work, arena_free_worker);
bpf_map_init_from_attr(&arena->map, attr);
err = bpf_map_alloc_pages(&arena->map, NUMA_NO_NODE, 1, &arena->scratch_page);
if (err)
arena->scratch_page = bpf_alloc_page(NUMA_NO_NODE, true);
if (!arena->scratch_page)
goto err_free_arena;
range_tree_init(&arena->rt);
@@ -481,7 +486,10 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
struct bpf_map *map = vmf->vma->vm_file->private_data;
struct bpf_arena *arena = container_of(map, struct bpf_arena, map);
struct mem_cgroup *new_memcg, *old_memcg;
struct page *page;
LLIST_HEAD(pages);
struct page *page, *new_page = NULL;
struct apply_range_data data;
vm_fault_t fault_ret;
long kbase, kaddr;
unsigned long flags;
int ret;
@@ -489,59 +497,123 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf)
kbase = bpf_arena_get_kern_vm_start(arena);
kaddr = kbase + (u32)(vmf->address);
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
page = vmalloc_to_page((void *)kaddr);
if (!page && !(arena->map.map_flags & BPF_F_SEGV_ON_FAULT)) {
/*
* Preallocate outside the lock so the allocation can reclaim;
* __GFP_RETRY_MAYFAIL keeps the OOM killer out of it.
*/
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
new_page = alloc_pages_node(map->numa_node,
GFP_KERNEL | __GFP_ZERO |
__GFP_ACCOUNT | __GFP_NOWARN |
__GFP_RETRY_MAYFAIL, 0);
bpf_map_memcg_exit(old_memcg, new_memcg);
}
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) {
/*
* A failed lock means a possible deadlock was detected. Don't
* return VM_FAULT_RETRY: this handler never took mmap_lock, but
* the fault path would re-take it on retry and deadlock. Fail.
*/
if (new_page)
free_pages_nolock(new_page, 0);
return VM_FAULT_SIGBUS;
}
page = vmalloc_to_page((void *)kaddr);
if (page) {
if (page == arena->scratch_page)
if (page == arena->scratch_page) {
/* BPF triggered scratch here; don't lazy-alloc over it */
goto out_sigsegv;
fault_ret = (arena->map.map_flags & BPF_F_SEGV_ON_FAULT)
? VM_FAULT_SIGSEGV : VM_FAULT_SIGBUS;
goto out_err_locked;
}
/* already have a page vmap-ed */
goto out;
}
if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT) {
/*
* User space requested to segfault when page is not allocated
* by bpf prog
*/
fault_ret = VM_FAULT_SIGSEGV;
goto out_err_locked;
}
if (is_range_tree_set(&arena->rt, vmf->pgoff, 1)) {
/* Allocated without a page: reserved or being freed */
fault_ret = VM_FAULT_SIGBUS;
goto out_err_locked;
}
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT)
/* User space requested to segfault when page is not allocated by bpf prog */
goto out_sigsegv_memcg;
if (!new_page) {
/*
* The probed page was freed meanwhile or preallocation failed;
* try the non-blocking allocator, we cannot sleep here.
*/
new_page = bpf_alloc_page(map->numa_node, false);
if (!new_page) {
fault_ret = VM_FAULT_SIGBUS;
goto out_err_locked_memcg;
}
}
ret = range_tree_clear(&arena->rt, vmf->pgoff, 1);
if (ret)
goto out_sigsegv_memcg;
struct apply_range_data data = { .arena = arena, .pages = &page, .i = 0 };
/* Account into memcg of the process that created bpf_arena */
ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page);
if (ret) {
range_tree_set(&arena->rt, vmf->pgoff, 1);
goto out_sigsegv_memcg;
fault_ret = VM_FAULT_SIGBUS;
goto out_err_locked_memcg;
}
llist_add(&new_page->pcp_llist, &pages);
data.arena = arena;
data.pages = &pages;
data.i = 0;
ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data);
if (ret) {
llist_del_first(&pages);
range_tree_set(&arena->rt, vmf->pgoff, 1);
free_pages_nolock(page, 0);
goto out_sigsegv_memcg;
fault_ret = VM_FAULT_SIGBUS;
goto out_err_locked_memcg;
}
flush_vmap_cache(kaddr, PAGE_SIZE);
bpf_map_memcg_exit(old_memcg, new_memcg);
/* new_page was consumed */
page = new_page;
new_page = NULL;
out:
page_ref_add(page, 1);
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
vmf->page = page;
return 0;
out_sigsegv_memcg:
if (new_page)
free_pages_nolock(new_page, 0);
/*
* A free clears the kernel pte and then takes the page lock before it
* zaps user vma-s. Either it waits for this fault and zaps its pte, or
* the page is gone from here.
*/
lock_page(page);
if (vmalloc_to_page((void *)kaddr) == page && !PageHWPoison(page)) {
vmf->page = page;
return VM_FAULT_LOCKED;
}
/* The page was freed meanwhile. Let the fault start over. */
fault_ret = PageHWPoison(page) ? VM_FAULT_HWPOISON : VM_FAULT_NOPAGE;
unlock_page(page);
put_page(page);
return fault_ret;
out_err_locked_memcg:
bpf_map_memcg_exit(old_memcg, new_memcg);
out_sigsegv:
out_err_locked:
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
return VM_FAULT_SIGSEGV;
if (new_page)
free_pages_nolock(new_page, 0);
return fault_ret;
}
static const struct vm_operations_struct arena_vm_ops = {
@@ -661,6 +733,27 @@ static u64 clear_lo32(u64 val)
return val & ~(u64)~0U;
}
static int arena_adjust_tree(struct bpf_arena *arena, long uaddr, long page_cnt, long *pgoff)
{
int ret;
/* Special case where user is requesting specific range. */
if (uaddr) {
ret = is_range_tree_set(&arena->rt, *pgoff, page_cnt);
if (ret)
return ret;
return range_tree_clear(&arena->rt, *pgoff, page_cnt);
}
ret = range_tree_find(&arena->rt, page_cnt);
if (ret < 0)
return ret;
*pgoff = ret;
return range_tree_clear(&arena->rt, *pgoff, page_cnt);
}
/*
* Allocate pages and vmap them into kernel vmalloc area.
* Later the pages will be mmaped into user space vma.
@@ -673,13 +766,13 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
u64 kern_vm_start = bpf_arena_get_kern_vm_start(arena);
struct mem_cgroup *new_memcg, *old_memcg;
struct apply_range_data data;
struct page **pages = NULL;
long remaining, mapped = 0;
long alloc_pages;
LLIST_HEAD(pages);
long mapped = 0;
unsigned long flags;
long pgoff = 0;
u32 uaddr32;
int ret, i;
long addr = 0;
int ret;
if (node_id != NUMA_NO_NODE &&
((unsigned int)node_id >= nr_node_ids || !node_online(node_id)))
@@ -696,89 +789,63 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt
/* requested address will be outside of user VMA */
return 0;
}
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
/* Cap allocation size to KMALLOC_MAX_CACHE_SIZE so kmalloc_nolock() can succeed. */
alloc_pages = min(page_cnt, KMALLOC_MAX_CACHE_SIZE / sizeof(struct page *));
pages = kmalloc_nolock(alloc_pages * sizeof(struct page *), __GFP_ACCOUNT, NUMA_NO_NODE);
if (!pages) {
bpf_map_memcg_exit(old_memcg, new_memcg);
return 0;
}
ret = bpf_alloc_pages(node_id, page_cnt, &pages, sleepable);
if (ret)
goto out_memcg;
data.arena = arena;
data.pages = pages;
data.pages = &pages;
data.i = 0;
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
goto out_free_pages;
if (uaddr) {
ret = is_range_tree_set(&arena->rt, pgoff, page_cnt);
if (ret)
goto out_unlock_free_pages;
ret = range_tree_clear(&arena->rt, pgoff, page_cnt);
} else {
ret = pgoff = range_tree_find(&arena->rt, page_cnt);
if (pgoff >= 0)
ret = range_tree_clear(&arena->rt, pgoff, page_cnt);
ret = arena_adjust_tree(arena, uaddr, page_cnt, &pgoff);
if (ret) {
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
goto out_free_pages;
}
if (ret)
goto out_unlock_free_pages;
remaining = page_cnt;
uaddr32 = (u32)(arena->user_vm_start + pgoff * PAGE_SIZE);
while (remaining) {
long this_batch = min(remaining, alloc_pages);
/*
* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1
* will not overflow 32-bit. Lower 32-bit need to represent
* contiguous user address range.
* Map these pages at kern_vm_start base.
* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow
* lower 32-bit and it's ok.
*/
ret = apply_to_page_range(&init_mm, kern_vm_start + uaddr32,
page_cnt << PAGE_SHIFT, apply_range_set_cb, &data);
mapped = data.i;
if (ret)
goto out_unmap;
/* zeroing is needed, since alloc_pages_bulk() only fills in non-zero entries */
memset(pages, 0, this_batch * sizeof(struct page *));
ret = bpf_map_alloc_pages(&arena->map, node_id, this_batch, pages);
if (ret)
goto out;
/*
* Earlier checks made sure that uaddr32 + page_cnt * PAGE_SIZE - 1
* will not overflow 32-bit. Lower 32-bit need to represent
* contiguous user address range.
* Map these pages at kern_vm_start base.
* kern_vm_start + uaddr32 + page_cnt * PAGE_SIZE - 1 can overflow
* lower 32-bit and it's ok.
*/
data.i = 0;
ret = apply_to_page_range(&init_mm,
kern_vm_start + uaddr32 + (mapped << PAGE_SHIFT),
this_batch << PAGE_SHIFT, apply_range_set_cb, &data);
if (ret) {
/* data.i pages were mapped, account them and free the remaining */
mapped += data.i;
for (i = data.i; i < this_batch; i++)
free_pages_nolock(pages[i], 0);
goto out;
}
mapped += this_batch;
remaining -= this_batch;
}
flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
kfree_nolock(pages);
bpf_map_memcg_exit(old_memcg, new_memcg);
return clear_lo32(arena->user_vm_start) + uaddr32;
out:
addr = clear_lo32(arena->user_vm_start) + uaddr32;
goto out_memcg;
out_unmap:
/* Error handling: Undo partial mappings. */
flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
range_tree_set(&arena->rt, pgoff + mapped, page_cnt - mapped);
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
if (mapped) {
flush_vmap_cache(kern_vm_start + uaddr32, mapped << PAGE_SHIFT);
if (mapped)
arena_free_pages(arena, uaddr32, mapped, sleepable);
}
goto out_free_pages;
out_unlock_free_pages:
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
out_free_pages:
kfree_nolock(pages);
/* Error handling: Free back any unmapped pages. */
bpf_free_pages(&pages);
out_memcg:
bpf_map_memcg_exit(old_memcg, new_memcg);
return 0;
return addr;
}
/*
@@ -807,7 +874,7 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
* otherwise call B's mark would make call A skip a vml that A has
* not yet zapped for A's uaddr range.
*/
mutex_lock(&arena->zap_mutex);
lockdep_assert_held(&arena->zap_mutex);
mutex_lock(&arena->lock);
my_gen = ++arena->zap_gen;
for (;;) {
@@ -841,49 +908,41 @@ static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt)
mutex_lock(&arena->lock);
}
mutex_unlock(&arena->lock);
mutex_unlock(&arena->zap_mutex);
}
static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)
/*
* The range stays allocated in the range tree until the pages are unmapped
* from vmalloc area and from all user space vma-s, so that nobody can
* populate it while stale TLB entries or user ptes are still around.
*/
static int arena_free_range(struct bpf_arena *arena, long uaddr, long page_cnt)
{
struct mem_cgroup *new_memcg, *old_memcg;
u64 full_uaddr, uaddr_end;
long kaddr, pgoff;
struct page *page;
struct llist_head free_pages;
struct llist_node *pos, *t;
struct arena_free_span *s;
u64 full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
long kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr;
long pgoff = compute_pgoff(arena, uaddr);
struct clear_range_data cdata;
struct llist_node *pos, *t;
LLIST_HEAD(free_pages);
unsigned long flags;
int ret = 0;
struct page *page;
int ret;
/* only aligned lower 32-bit are relevant */
uaddr = (u32)uaddr;
uaddr &= PAGE_MASK;
kaddr = bpf_arena_get_kern_vm_start(arena) + uaddr;
full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
if (full_uaddr < arena->user_vm_start)
return;
uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT));
if (full_uaddr >= uaddr_end)
return;
page_cnt = (uaddr_end - full_uaddr) >> PAGE_SHIFT;
pgoff = compute_pgoff(arena, uaddr);
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
if (!sleepable)
goto defer;
/* An overlapping free must not release the range before this one is done */
guard(mutex)(&arena->zap_mutex);
ret = raw_res_spin_lock_irqsave(&arena->spinlock, flags);
/* Can't proceed without holding the spinlock so defer the free */
if (ret)
goto defer;
return ret;
range_tree_set(&arena->rt, pgoff, page_cnt);
/*
* Take the free pages of the range too. It allocates, and can fail,
* only when the whole range is free. Then there is nothing to do.
*/
if (range_tree_clear(&arena->rt, pgoff, page_cnt)) {
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
return 0;
}
init_llist_head(&free_pages);
cdata.arena = arena;
cdata.free_pages = &free_pages;
/* clear ptes and collect struct pages */
@@ -893,6 +952,12 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
/* drop the lock to do the tlb flush and zap pages */
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
/* wait for arena_vm_fault() that found the page before its pte was cleared */
llist_for_each_entry(page, free_pages.first, pcp_llist) {
lock_page(page);
unlock_page(page);
}
/* ensure no stale TLB entries */
flush_tlb_kernel_range(kaddr, kaddr + (page_cnt * PAGE_SIZE));
@@ -911,8 +976,39 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt,
zap_pages(arena, full_uaddr, 1);
__free_page(page);
}
bpf_map_memcg_exit(old_memcg, new_memcg);
/* The callers hold no other rqspinlock, so this one can only time out */
while (raw_res_spin_lock_irqsave(&arena->spinlock, flags))
cond_resched();
range_tree_set(&arena->rt, pgoff, page_cnt);
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
return 0;
}
static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable)
{
struct mem_cgroup *new_memcg, *old_memcg;
u64 full_uaddr, uaddr_end;
struct arena_free_span *s;
/* only aligned lower 32-bit are relevant */
uaddr = (u32)uaddr;
uaddr &= PAGE_MASK;
full_uaddr = clear_lo32(arena->user_vm_start) + uaddr;
if (full_uaddr < arena->user_vm_start)
return;
uaddr_end = min(arena->user_vm_end, full_uaddr + (page_cnt << PAGE_SHIFT));
if (full_uaddr >= uaddr_end)
return;
page_cnt = (uaddr_end - full_uaddr) >> PAGE_SHIFT;
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
/* Can't proceed without holding the spinlock so defer the free */
if (!sleepable || arena_free_range(arena, uaddr, page_cnt))
goto defer;
bpf_map_memcg_exit(old_memcg, new_memcg);
return;
defer:
@@ -974,66 +1070,18 @@ static void arena_free_worker(struct work_struct *work)
{
struct bpf_arena *arena = container_of(work, struct bpf_arena, free_work);
struct mem_cgroup *new_memcg, *old_memcg;
struct llist_node *list, *pos, *t;
struct llist_node *pos, *t;
struct arena_free_span *s;
u64 arena_vm_start, user_vm_start;
struct llist_head free_pages;
struct clear_range_data cdata;
struct page *page;
unsigned long full_uaddr;
long kaddr, page_cnt, pgoff;
unsigned long flags;
if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) {
schedule_work(work);
return;
}
bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg);
init_llist_head(&free_pages);
cdata.arena = arena;
cdata.free_pages = &free_pages;
arena_vm_start = bpf_arena_get_kern_vm_start(arena);
user_vm_start = bpf_arena_get_user_vm_start(arena);
list = llist_del_all(&arena->free_spans);
llist_for_each(pos, list) {
llist_for_each_safe(pos, t, llist_del_all(&arena->free_spans)) {
s = llist_entry(pos, struct arena_free_span, node);
page_cnt = s->page_cnt;
kaddr = arena_vm_start + s->uaddr;
pgoff = compute_pgoff(arena, s->uaddr);
/* clear ptes and collect pages in free_pages llist */
apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT,
apply_range_clear_cb, &cdata);
range_tree_set(&arena->rt, pgoff, page_cnt);
}
raw_res_spin_unlock_irqrestore(&arena->spinlock, flags);
/* Iterate the list again without holding spinlock to do the tlb flush and zap_pages */
llist_for_each_safe(pos, t, list) {
s = llist_entry(pos, struct arena_free_span, node);
page_cnt = s->page_cnt;
full_uaddr = clear_lo32(user_vm_start) + s->uaddr;
kaddr = arena_vm_start + s->uaddr;
/* ensure no stale TLB entries */
flush_tlb_kernel_range(kaddr, kaddr + (page_cnt * PAGE_SIZE));
/* remove pages from user vmas */
zap_pages(arena, full_uaddr, page_cnt);
while (arena_free_range(arena, s->uaddr, s->page_cnt))
cond_resched();
kfree_nolock(s);
}
/* free all pages collected by apply_to_existing_page_range() in the first loop */
llist_for_each_safe(pos, t, __llist_del_all(&free_pages)) {
page = llist_entry(pos, struct page, pcp_llist);
__free_page(page);
}
bpf_map_memcg_exit(old_memcg, new_memcg);
}
kernel/bpf/arraymap.c
+4 -4
View File
@@ -438,15 +438,15 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value,
u32 size;
int cpu, off = 0;
if (unlikely((map_flags & BPF_F_LOCK) || (u32)map_flags > BPF_F_ALL_CPUS))
/* unknown flags */
if (unlikely(((map_flags & BPF_EXIST_FLAGS) == BPF_EXIST_FLAGS) ||
((u32)map_flags & ~(BPF_EXIST_FLAGS | BPF_CPU_FLAGS))))
return -EINVAL;
if (unlikely(index >= array->map.max_entries))
/* all elements were pre-allocated, cannot insert a new one */
return -E2BIG;
if (unlikely(map_flags == BPF_NOEXIST))
if (unlikely(map_flags & BPF_NOEXIST))
/* all elements already exist */
return -EEXIST;
@@ -1035,7 +1035,7 @@ static void *prog_fd_array_get_ptr(struct bpf_map *map,
}
mutex_lock(&prog->aux->ext_mutex);
is_extended = prog->aux->is_extended;
is_extended = prog->aux->freplace_link_cnt > 0;
if (!is_extended)
prog->aux->prog_array_member_cnt++;
mutex_unlock(&prog->aux->ext_mutex);
kernel/bpf/backtrack.c
+103 -55
View File
@@ -9,7 +9,8 @@
/* for any branch, call, exit record the history of jmps in the given state */
int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur,
int insn_flags, int spi, int frame, u64 linked_regs)
int insn_flags, int spi, int frame, const u16 *linked_regs,
u8 linked_regs_cnt)
{
u32 cnt = cur->jmp_history_cnt;
struct bpf_jmp_history_entry *p;
@@ -27,10 +28,13 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
env->cur_hist_ent->flags |= insn_flags;
env->cur_hist_ent->spi = spi;
env->cur_hist_ent->frame = frame;
verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env,
"insn history: insn_idx %d linked_regs: %#llx",
env->insn_idx, env->cur_hist_ent->linked_regs);
env->cur_hist_ent->linked_regs = linked_regs;
verifier_bug_if(env->cur_hist_ent->linked_regs_cnt != 0, env,
"insn history: insn_idx %d has %u linked regs",
env->insn_idx, env->cur_hist_ent->linked_regs_cnt);
if (linked_regs_cnt)
memcpy(env->cur_hist_ent->linked_regs, linked_regs,
linked_regs_cnt * sizeof(*linked_regs));
env->cur_hist_ent->linked_regs_cnt = linked_regs_cnt;
return 0;
}
@@ -47,7 +51,9 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state
p->flags = insn_flags;
p->spi = spi;
p->frame = frame;
p->linked_regs = linked_regs;
if (linked_regs_cnt)
memcpy(p->linked_regs, linked_regs, linked_regs_cnt * sizeof(*linked_regs));
p->linked_regs_cnt = linked_regs_cnt;
cur->jmp_history_cnt = cnt;
env->cur_hist_ent = p;
@@ -123,13 +129,26 @@ static inline void bt_reset(struct backtrack_state *bt)
bt->env = env;
}
static inline u32 bt_empty(struct backtrack_state *bt)
static inline bool bt_frame_stack_empty(struct backtrack_state *bt, u32 frame)
{
u64 mask = 0;
return bitmap_empty(bt->stack_masks[frame], MAX_BPF_STACK_SLOTS);
}
static inline bool bt_stack_empty(struct backtrack_state *bt)
{
return bt_frame_stack_empty(bt, bt->frame);
}
static inline bool bt_empty(struct backtrack_state *bt)
{
u32 mask = 0;
int i;
for (i = 0; i <= bt->frame; i++)
mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i];
for (i = 0; i <= bt->frame; i++) {
mask |= bt->reg_masks[i] | bt->stack_arg_masks[i];
if (!bt_frame_stack_empty(bt, i))
return false;
}
return mask == 0;
}
@@ -181,7 +200,7 @@ static inline void bt_clear_reg(struct backtrack_state *bt, u32 reg)
static inline void bt_clear_frame_slot(struct backtrack_state *bt, u32 frame, u32 slot)
{
bt->stack_masks[frame] &= ~(1ull << slot);
__clear_bit(slot, bt->stack_masks[frame]);
}
static inline u32 bt_frame_reg_mask(struct backtrack_state *bt, u32 frame)
@@ -194,14 +213,14 @@ static inline u32 bt_reg_mask(struct backtrack_state *bt)
return bt->reg_masks[bt->frame];
}
static inline u64 bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
static inline unsigned long *bt_frame_stack_mask(struct backtrack_state *bt, u32 frame)
{
return bt->stack_masks[frame];
}
static inline u64 bt_stack_mask(struct backtrack_state *bt)
static inline unsigned long *bt_stack_mask(struct backtrack_state *bt)
{
return bt->stack_masks[bt->frame];
return bt_frame_stack_mask(bt, bt->frame);
}
static inline u8 bt_stack_arg_mask(struct backtrack_state *bt)
@@ -233,17 +252,16 @@ static void fmt_reg_mask(char *buf, ssize_t buf_sz, u32 reg_mask)
break;
}
}
/* format stack slots bitmask, e.g., "-8,-24,-40" for 0x15 mask */
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, u64 stack_mask)
/* format stack slots bitmask, e.g., "-8,-24,-40" for slots 0, 2 and 4 */
void bpf_fmt_stack_mask(char *buf, ssize_t buf_sz, const unsigned long *stack_mask)
{
DECLARE_BITMAP(mask, 64);
bool first = true;
int i, n;
buf[0] = '\0';
bitmap_from_u64(mask, stack_mask);
for_each_set_bit(i, mask, 64) {
for_each_set_bit(i, stack_mask, MAX_BPF_STACK_SLOTS) {
n = snprintf(buf, buf_sz, "%s%d", first ? "" : ",", -(i + 1) * 8);
first = false;
buf += n;
@@ -406,15 +424,18 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
if (class == BPF_STX)
bt_set_reg(bt, sreg);
} else if (class == BPF_JMP || class == BPF_JMP32) {
if (bpf_pseudo_call(insn)) {
int subprog_insn_idx, subprog;
if (bpf_pseudo_call(insn) || bpf_is_callx(insn)) {
int subprog_insn_idx, subprog = -1;
subprog_insn_idx = idx + insn->imm + 1;
subprog = bpf_find_subprog(env, subprog_insn_idx);
if (subprog < 0)
return -EFAULT;
if (bpf_pseudo_call(insn)) {
subprog_insn_idx = idx + insn->imm + 1;
subprog = bpf_find_subprog(env, subprog_insn_idx);
if (subprog < 0)
return -EFAULT;
}
if (bpf_subprog_is_global(env, subprog)) {
/* callx calls static subprogs only */
if (subprog >= 0 && bpf_subprog_is_global(env, subprog)) {
/* check that jump history doesn't have any
* extra instructions from subprog; the next
* instruction after call to global subprog
@@ -423,6 +444,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
*/
verifier_bug_if(idx + 1 != subseq_idx, env,
"extra insn from subprog");
/* global subprog always sets R0 */
bt_clear_reg(bt, BPF_REG_0);
/* and if it does not set R2, main pass would catch it */
bt_clear_reg(bt, BPF_REG_2);
/* r1-r5 are invalidated after subprog call,
* so for global func call it shouldn't be set
* anymore
@@ -432,8 +457,6 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_reg_mask(bt));
return -EFAULT;
}
/* global subprog always sets R0 */
bt_clear_reg(bt, BPF_REG_0);
return 0;
} else {
/* static subprog call instruction, which
@@ -450,10 +473,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
/* we are now tracking register spills correctly,
* so any instance of leftover slots is a bug
*/
if (bt_stack_mask(bt) != 0) {
verifier_bug(env,
"static subprog leftover stack slots %llx",
bt_stack_mask(bt));
if (!bt_stack_empty(bt)) {
bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
bt_stack_mask(bt));
verifier_bug(env, "static subprog leftover stack slots %s",
env->tmp_str_buf);
return -EFAULT;
}
/* propagate r1-r5 to the caller */
@@ -486,9 +510,11 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
bt_reg_mask(bt));
return -EFAULT;
}
if (bt_stack_mask(bt) != 0) {
verifier_bug(env, "callback leftover stack slots %llx",
bt_stack_mask(bt));
if (!bt_stack_empty(bt)) {
bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN,
bt_stack_mask(bt));
verifier_bug(env, "callback leftover stack slots %s",
env->tmp_str_buf);
return -EFAULT;
}
/* clear r1-r5 in callback subprog's mask */
@@ -506,6 +532,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -ENOTSUPP;
/* regular helper call sets R0 */
bt_clear_reg(bt, BPF_REG_0);
/* kfunc might also set R2 */
bt_clear_reg(bt, BPF_REG_2);
if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) {
/* if backtracking was looking for registers R1-R5
* they should have been found already.
@@ -520,40 +548,51 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx,
return -EFAULT;
}
} else if (opcode == BPF_EXIT) {
bool from_subprog_call, r0_precise;
bool from_subprog_call, r0_precise, r2_precise;
/* BPF_EXIT in subprog or callback always returns
* right after the call instruction, so by checking
* whether the instruction at subseq_idx-1 is subprog
* call or not we can distinguish actual exit from
* *subprog* from exit from *callback*. In the former
* case, we need to propagate r0 precision, if
* necessary. In the former we never do that.
* case, we need to propagate the precision of the
* return registers, if necessary. In the latter we
* never do that.
*/
from_subprog_call = subseq_idx - 1 >= 0 &&
bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]);
(bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) ||
bpf_is_callx(&env->prog->insnsi[subseq_idx - 1]));
r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0);
r2_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_2);
/* Backtracking to a nested function call, 'idx' is a part of
* the inner frame 'subseq_idx' is a part of the outer frame.
* In case of a regular function call, instructions giving
* precision to registers R1-R5 should have been found already.
* In case of a regular function call, the callee defines the
* return registers R0 and R2, so clear them before checking
* that instructions giving precision to registers R1-R5 have
* been found already. R2 is also an argument register, hence
* it has to be cleared before that check.
* In case of a callback from bpf_loop(), R{1,4} in the calling
* frame would be set as precise and that is correct.
* frame would be set as precise and that is correct, and R2
* might be a precise helper argument as well, so leave it set.
*/
bt_clear_reg(bt, BPF_REG_0);
if (from_subprog_call)
bt_clear_reg(bt, BPF_REG_2);
if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) {
verifier_bug(env, "backtracking exit unexpected regs %x",
bt_reg_mask(bt));
return -EFAULT;
}
bt_clear_reg(bt, BPF_REG_0);
if (bt_subprog_enter(bt))
return -EFAULT;
if (r0_precise)
bt_set_reg(bt, BPF_REG_0);
if (r2_precise)
bt_set_reg(bt, BPF_REG_2);
/* r6-r9 and stack slots will stay set in caller frame
* bitmasks until we return back from callee(s)
*/
@@ -693,10 +732,12 @@ void bpf_mark_all_scalars_precise(struct bpf_verifier_env *env,
i, j);
}
}
for (j = 0; j < func->allocated_stack / BPF_REG_SIZE; j++) {
if (!bpf_is_spilled_reg(&func->stack[j]))
for (j = 0; j < bpf_stack_nr_slots(func); j++) {
struct bpf_stack_state *ss = bpf_stack_slot(func, j);
if (!bpf_is_spilled_reg(ss))
continue;
reg = &func->stack[j].spilled_ptr;
reg = &ss->spilled_ptr;
if (reg->type != SCALAR_VALUE || reg->precise)
continue;
reg->precise = true;
@@ -808,6 +849,7 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
int subseq_idx = -1;
struct bpf_func_state *func;
bool tmp, skip_first = true;
struct bpf_stack_state *ss;
struct bpf_reg_state *reg;
int i, fr, err;
@@ -846,6 +888,10 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
}
if (last_idx < 0) {
/* Consecutive checkpoints can have no instructions between them. */
if (st->parent)
goto parent;
/* we are at the entry into subprog, which
* is expected for global funcs, but only if
* requested precise registers are R1-R5
@@ -854,7 +900,7 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
if (st->curframe == 0 &&
st->frame[0]->subprogno > 0 &&
st->frame[0]->callsite == BPF_MAIN_FUNC &&
bt_stack_mask(bt) == 0 &&
bt_stack_empty(bt) &&
(bt_reg_mask(bt) & ~BPF_REGMASK_ARGS) == 0) {
bitmap_from_u64(mask, bt_reg_mask(bt));
for_each_set_bit(i, mask, 32) {
@@ -868,8 +914,9 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
return 0;
}
verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %llx",
st->frame[0]->subprogno, bt_reg_mask(bt), bt_stack_mask(bt));
bpf_fmt_stack_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, bt_stack_mask(bt));
verifier_bug(env, "backtracking func entry subprog %d reg_mask %x stack_mask %s",
st->frame[0]->subprogno, bt_reg_mask(bt), env->tmp_str_buf);
return -EFAULT;
}
@@ -909,6 +956,7 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
return -EFAULT;
}
}
parent:
st = st->parent;
if (!st)
break;
@@ -930,18 +978,18 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env,
}
}
bitmap_from_u64(mask, bt_frame_stack_mask(bt, fr));
for_each_set_bit(i, mask, 64) {
if (verifier_bug_if(i >= func->allocated_stack / BPF_REG_SIZE,
for_each_set_bit(i, bt_frame_stack_mask(bt, fr), MAX_BPF_STACK_SLOTS) {
if (verifier_bug_if(i >= bpf_stack_nr_slots(func),
env, "stack slot %d, total slots %d",
i, func->allocated_stack / BPF_REG_SIZE))
i, bpf_stack_nr_slots(func)))
return -EFAULT;
if (!bpf_is_spilled_scalar_reg(&func->stack[i])) {
ss = bpf_stack_slot(func, i);
if (!bpf_is_spilled_scalar_reg(ss)) {
bt_clear_frame_slot(bt, fr, i);
continue;
}
reg = &func->stack[i].spilled_ptr;
reg = &ss->spilled_ptr;
if (reg->precise) {
bt_clear_frame_slot(bt, fr, i);
} else {
kernel/bpf/bpf_insn_array.c
+12 -7
View File
@@ -199,12 +199,16 @@ int bpf_insn_array_init(struct bpf_map *map, const struct bpf_prog *prog)
return -EBUSY;
/*
* Reset all the map indexes to the original values. This is needed,
* e.g., when a replay of verification with different log level should
* be performed.
* Reset the map to its pre-verification state. The xlated and jitted
* offsets and the jitted target pointers are recomputed by the verifier
* and the JIT for this program, so any values left by a previous owner
* must be cleared here.
*/
for (i = 0; i < map->max_entries; i++)
for (i = 0; i < map->max_entries; i++) {
values[i].xlated_off = values[i].orig_off;
values[i].jitted_off = 0;
insn_array->ips[i] = 0;
}
return 0;
}
@@ -228,10 +232,11 @@ void bpf_insn_array_release(struct bpf_map *map)
{
struct bpf_insn_array *insn_array = cast_insn_array(map);
atomic_set(&insn_array->used, 0);
/* Paired with atomic_xchg() in bpf_insn_array_init(). */
atomic_set_release(&insn_array->used, 0);
}
void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len)
void bpf_insn_array_adjust(struct bpf_map *map, u32 first, u32 len)
{
struct bpf_insn_array *insn_array = cast_insn_array(map);
int i;
@@ -240,7 +245,7 @@ void bpf_insn_array_adjust(struct bpf_map *map, u32 off, u32 len)
return;
for (i = 0; i < map->max_entries; i++) {
if (insn_array->values[i].xlated_off <= off)
if (insn_array->values[i].xlated_off < first)
continue;
if (insn_array->values[i].xlated_off == INSN_DELETED)
continue;
kernel/bpf/bpf_iter.c
+28 -3
View File
@@ -72,6 +72,15 @@ static inline bool bpf_iter_target_support_resched(const struct bpf_iter_target_
return tinfo->reg_info->feature & BPF_ITER_RESCHED;
}
/* Only allow sleepable program for resched-able iterator that can run one */
static bool bpf_iter_target_support_prog(const struct bpf_iter_target_info *tinfo,
const struct bpf_prog *prog)
{
return !prog->sleepable ||
(bpf_iter_target_support_resched(tinfo) &&
!(tinfo->reg_info->feature & BPF_ITER_NOSLEEP));
}
static bool bpf_iter_support_resched(struct seq_file *seq)
{
struct bpf_iter_priv_data *iter_priv;
@@ -342,6 +351,13 @@ int bpf_iter_prog_supported(struct bpf_prog *prog)
const char *prefix = BPF_ITER_FUNC_PREFIX;
int prefix_len = strlen(prefix);
/*
* The attach function of a program that has a target program is a
* function of that program, whatever its name is.
*/
if (prog->aux->dst_prog)
return -EINVAL;
if (strncmp(attach_fname, prefix, prefix_len))
return -EINVAL;
@@ -409,6 +425,8 @@ static int bpf_iter_link_replace(struct bpf_link *link,
struct bpf_prog *new_prog,
struct bpf_prog *old_prog)
{
struct bpf_iter_link *iter_link =
container_of(link, struct bpf_iter_link, link);
int ret = 0;
mutex_lock(&link_mutex);
@@ -419,11 +437,19 @@ static int bpf_iter_link_replace(struct bpf_link *link,
if (link->prog->type != new_prog->type ||
link->prog->expected_attach_type != new_prog->expected_attach_type ||
link->prog->aux->attach_btf_id != new_prog->aux->attach_btf_id) {
link->prog->aux->attach_btf_id != new_prog->aux->attach_btf_id ||
!bpf_iter_target_support_prog(iter_link->tinfo, new_prog)) {
ret = -EINVAL;
goto out_unlock;
}
if (iter_link->tinfo->reg_info->check_prog) {
ret = iter_link->tinfo->reg_info->check_prog(new_prog,
&iter_link->aux);
if (ret)
goto out_unlock;
}
old_prog = xchg(&link->prog, new_prog);
bpf_prog_put(old_prog);
@@ -544,8 +570,7 @@ int bpf_iter_link_attach(const union bpf_attr *attr, bpfptr_t uattr,
if (!tinfo)
return -ENOENT;
/* Only allow sleepable program for resched-able iterator */
if (prog->sleepable && !bpf_iter_target_support_resched(tinfo))
if (!bpf_iter_target_support_prog(tinfo, prog))
return -EINVAL;
link = kzalloc_obj(*link, GFP_USER | __GFP_NOWARN);
kernel/bpf/bpf_local_storage.c
+48 -18
View File
@@ -199,7 +199,7 @@ static void bpf_selem_unlink_storage_nolock_misc(struct bpf_local_storage_elem *
}
if (pin_owner)
refcount_dec(&local_storage->owner_refcnt);
bpf_local_storage_unpin_owner(local_storage);
}
/* local_storage->lock must be held and selem->local_storage == local_storage.
@@ -240,24 +240,26 @@ void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
hlist_add_head_rcu(&selem->snode, &local_storage->list);
}
/* Must be called with the owning local_storage->lock held. */
static int bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
{
struct bpf_local_storage *local_storage;
struct bpf_local_storage_map *smap;
struct bpf_local_storage_map_bucket *b;
unsigned long flags;
int err;
lockdep_assert_irqs_disabled();
local_storage = rcu_dereference_check(selem->local_storage,
bpf_rcu_lock_held());
smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
b = select_bucket(smap, local_storage);
err = raw_res_spin_lock_irqsave(&b->lock, flags);
err = raw_res_spin_lock(&b->lock);
if (err)
return err;
hlist_del_init_rcu(&selem->map_node);
raw_res_spin_unlock_irqrestore(&b->lock, flags);
raw_res_spin_unlock(&b->lock);
return 0;
}
@@ -350,7 +352,8 @@ out:
static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem,
struct bpf_local_storage_map_bucket *b)
{
bool in_map_free = !!b, free_storage = false;
bool in_map_free = !!b, free_storage = false, sleepable_users = false;
bool keep_fields = false;
struct bpf_local_storage *local_storage;
struct bpf_local_storage_map *smap;
unsigned long flags;
@@ -360,17 +363,26 @@ static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem,
smap = rcu_dereference_check(SDATA(selem)->smap, bpf_rcu_lock_held());
if (smap) {
/* A map that is freed, or NULL here, has no program left. */
sleepable_users = !in_map_free &&
(atomic64_read(&smap->map.sleepable_refcnt) ||
READ_ONCE(smap->map.was_inner_map));
b = b ? : select_bucket(smap, local_storage);
err = raw_res_spin_lock_irqsave(&b->lock, flags);
if (!err) {
/*
* Call bpf_obj_free_fields() under b->lock to make sure it is done
* exactly once for an selem. Safe to free special fields immediately
* as no BPF program should be referencing the selem.
* as no BPF program should be referencing the selem, unless it
* is sleepable: see below.
*/
if (likely(selem_linked_to_map(selem))) {
hlist_del_init_rcu(&selem->map_node);
bpf_obj_free_fields(smap->map.record, SDATA(selem)->data);
if (sleepable_users)
keep_fields = true;
else
bpf_obj_free_fields(smap->map.record,
SDATA(selem)->data);
unlink++;
}
raw_res_spin_unlock_irqrestore(&b->lock, flags);
@@ -385,7 +397,15 @@ static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem,
* with selem1 as raw_res_spin_lock_irqsave() returns -ETIMEDOUT.
*/
WARN_ON_ONCE(err && in_map_free);
if (!err || in_map_free)
/*
* A sleepable program can still use, and fill, the fields of a value
* that it got before it let go of the owner. As for bpf_selem_unlink(),
* bpf_selem_free_trace_rcu() frees them, and takes smap for that:
* map_free() waits for this RCU read side critical section and then
* for the callback. It must not take smap away from an selem that
* destroy() has unlinked from the map.
*/
if (in_map_free ? err || unlink : !err && !keep_fields)
RCU_INIT_POINTER(SDATA(selem)->smap, NULL);
}
@@ -429,13 +449,18 @@ static void bpf_selem_unlink_nofail(struct bpf_local_storage_elem *selem,
* then freed after an RCU grace period. However, if destroy() and map_free() are
* racing or rqspinlock returns errors in unlikely situations (unlink != 2), free
* the selem only after both map_free() and destroy() see the selem.
*
* The owner that map_free() takes an selem from is alive: sleepable code, which
* is in no RCU read side critical section, may be walking its list for another
* map. And it can still use a value that it got before it let go of an owner
* that destroy() is called for.
*/
if (unlink == 2 ||
atomic_cmpxchg(&selem->state, SELEM_UNLINKED, SELEM_TOFREE) == SELEM_UNLINKED)
bpf_selem_free(selem, true);
bpf_selem_free(selem, !in_map_free && !sleepable_users);
if (free_storage)
bpf_local_storage_free(local_storage, true);
bpf_local_storage_free(local_storage, !in_map_free);
}
void __bpf_local_storage_insert_cache(struct bpf_local_storage *local_storage,
@@ -507,8 +532,6 @@ int bpf_local_storage_alloc(void *owner,
if (err)
goto uncharge;
bpf_selem_link_map_nolock(b, first_selem);
owner_storage_ptr =
(struct bpf_local_storage **)owner_storage(smap, owner);
/* Publish storage to the owner.
@@ -523,11 +546,17 @@ int bpf_local_storage_alloc(void *owner,
*/
prev_storage = cmpxchg(owner_storage_ptr, NULL, storage);
if (unlikely(prev_storage)) {
bpf_selem_unlink_map_nolock(first_selem);
raw_res_spin_unlock_irqrestore(&b->lock, flags);
err = -EAGAIN;
goto uncharge;
}
/*
* Not before the storage is published: the owner does not wait for
* users of a storage that it knows nothing of, so that nobody may find
* such a storage through the map. b->lock holds off those who find
* first_selem through the owner and want to unlink it.
*/
bpf_selem_link_map_nolock(b, first_selem);
raw_res_spin_unlock_irqrestore(&b->lock, flags);
return 0;
@@ -552,7 +581,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
struct bpf_local_storage *local_storage;
struct bpf_local_storage_map_bucket *b;
HLIST_HEAD(old_selem_free_list);
unsigned long flags, b_flags;
unsigned long flags;
int err;
/* BPF_EXIST and BPF_NOEXIST cannot be both set */
@@ -637,7 +666,8 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
b = select_bucket(smap, local_storage);
err = raw_res_spin_lock_irqsave(&b->lock, b_flags);
/* local_storage->lock is held, so IRQs are already disabled. */
err = raw_res_spin_lock(&b->lock);
if (err)
goto unlock;
@@ -655,7 +685,7 @@ bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
&old_selem_free_list);
}
raw_res_spin_unlock_irqrestore(&b->lock, b_flags);
raw_res_spin_unlock(&b->lock);
unlock:
raw_res_spin_unlock_irqrestore(&local_storage->lock, flags);
free_selem:
@@ -750,8 +780,8 @@ u32 bpf_local_storage_destroy(struct bpf_local_storage *local_storage)
while (refcount_read(&local_storage->owner_refcnt))
cpu_relax();
/*
* Paired with refcount_dec() in bpf_selem_unlink_nofail()
* to make sure destroy() sees the correct local_storage->mem_charge.
* Paired with bpf_local_storage_unpin_owner() to make sure
* destroy() sees the correct local_storage->mem_charge.
*/
smp_mb();
}
kernel/bpf/bpf_lru_list.c
+3 -6
View File
@@ -71,7 +71,7 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l,
return;
bpf_lru_list_count_inc(l, tgt_type);
node->type = tgt_type;
WRITE_ONCE(node->type, tgt_type);
bpf_lru_node_clear_ref(node);
/* Reset pending_free only when moving to the free list */
if (tgt_type == BPF_LRU_LIST_T_FREE)
@@ -94,7 +94,7 @@ static void __bpf_lru_node_move(struct bpf_lru_list *l,
if (node->type != tgt_type) {
bpf_lru_list_count_dec(l, node->type);
bpf_lru_list_count_inc(l, tgt_type);
node->type = tgt_type;
WRITE_ONCE(node->type, tgt_type);
}
bpf_lru_node_clear_ref(node);
@@ -298,9 +298,6 @@ static void bpf_lru_list_push_free(struct bpf_lru_list *l,
{
unsigned long flags;
if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type)))
return;
if (raw_res_spin_lock_irqsave(&l->lock, flags)) {
WRITE_ONCE(node->pending_free, 1);
return;
@@ -542,7 +539,7 @@ static void bpf_common_lru_push_free(struct bpf_lru *lru,
return;
}
if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) {
if (unlikely(READ_ONCE(node->type) != BPF_LRU_LOCAL_LIST_T_PENDING)) {
raw_res_spin_unlock_irqrestore(&loc_l->lock,
flags);
goto check_lru_list;
kernel/bpf/bpf_lsm.c
+23 -3
View File
@@ -116,8 +116,11 @@ void bpf_lsm_find_cgroup_shim(const struct bpf_prog *prog,
}
#endif
BTF_ID_LIST_SINGLE(bpf_lsm_inode_init_security_btf_id, func,
bpf_lsm_inode_init_security)
int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
const struct bpf_prog *prog)
struct bpf_prog *prog)
{
u32 btf_id = prog->aux->attach_btf_id;
const char *func_name = prog->aux->attach_func_name;
@@ -140,6 +143,23 @@ int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
return -EINVAL;
}
if (btf_id == bpf_lsm_inode_init_security_btf_id[0]) {
/*
* inode, dir, qstr, xattrs, xattr_count
*
* The trusted pointer this hands the program is only as
* trustworthy as the context it is loaded from, which
* lsm_verifier_ops keeps read-only.
*/
static const struct bpf_ctx_arg_aux xattr_count_arg_info = {
.offset = 4 * sizeof(u64),
.reg_type = PTR_TO_MEM | MEM_RDONLY | PTR_TRUSTED,
.mem_size = sizeof(int),
};
return bpf_prog_ctx_arg_info_init(prog, &xattr_count_arg_info, 1);
}
return 0;
}
@@ -414,8 +434,8 @@ const struct bpf_prog_ops lsm_prog_ops = {
};
const struct bpf_verifier_ops lsm_verifier_ops = {
.get_func_proto = bpf_lsm_func_proto,
.is_valid_access = btf_ctx_access,
.get_func_proto = bpf_lsm_func_proto,
.is_valid_access = bpf_tracing_btf_ctx_access,
};
/* hooks return 0 or 1 */
kernel/bpf/bpf_lsm_proto.c
+17 -2
View File
@@ -4,6 +4,7 @@
*/
#include <linux/fs.h>
#include <linux/xattr.h>
#include <linux/bpf_lsm.h>
/*
@@ -12,8 +13,22 @@
* explicitly enforces that BPF LSM programs check for NULL before attempting to
* dereference it.
*/
int bpf_lsm_mmap_file(struct file *file__nullable, unsigned long reqprot,
unsigned long prot, unsigned long flags)
noinline int bpf_lsm_mmap_file(struct file *file__nullable, unsigned long reqprot,
unsigned long prot, unsigned long flags)
{
return 0;
}
/*
* Strong definition of the inode_init_security() BPF LSM hook. Both the
* qstr and the xattr array are NULL for some callers, so the __nullable
* suffix marks it as PTR_MAYBE_NULL. BPF LSM programs have to check before
* dereferencing them or handing them to bpf_inode_init_xattr().
*/
noinline int bpf_lsm_inode_init_security(struct inode *inode, struct inode *dir,
const struct qstr *qstr__nullable,
struct xattr *xattrs__nullable,
int *xattr_count)
{
return -EOPNOTSUPP;
}
kernel/bpf/bpf_struct_ops.c
+95 -47
View File
@@ -13,6 +13,7 @@
#include <linux/btf_ids.h>
#include <linux/rcupdate_wait.h>
#include <linux/poll.h>
#include <linux/bpf-cgroup.h>
struct bpf_struct_ops_value {
struct bpf_struct_ops_common_value common;
@@ -57,7 +58,7 @@ struct bpf_struct_ops_map {
struct bpf_struct_ops_link {
struct bpf_link link;
struct bpf_map __rcu *map;
struct bpf_map *map;
wait_queue_head_t wait_hup;
};
@@ -1024,9 +1025,18 @@ static void __bpf_struct_ops_map_free(struct bpf_map *map)
bpf_map_area_free(st_map);
}
static void bpf_struct_ops_map_free_pre_rcu(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
bpf_struct_ops_map_del_ksyms(st_map);
}
static void bpf_struct_ops_map_free(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
struct bpf_struct_ops *st_ops = st_map->st_ops_desc->st_ops;
bool tasks_rcu = st_ops->free_after_tasks_rcu_gp;
/* st_ops->owner was acquired during map_alloc to implicitly holds
* the btf's refcnt. The acquire was only done when btf_is_module()
@@ -1037,24 +1047,8 @@ static void bpf_struct_ops_map_free(struct bpf_map *map)
bpf_struct_ops_map_dissoc_progs(st_map);
bpf_struct_ops_map_del_ksyms(st_map);
/* The struct_ops's function may switch to another struct_ops.
*
* For example, bpf_tcp_cc_x->init() may switch to
* another tcp_cc_y by calling
* setsockopt(TCP_CONGESTION, "tcp_cc_y").
* During the switch, bpf_struct_ops_put(tcp_cc_x) is called
* and its refcount may reach 0 which then free its
* trampoline image while tcp_cc_x is still running.
*
* A vanilla rcu gp is to wait for all bpf-tcp-cc prog
* to finish. bpf-tcp-cc prog is non sleepable.
* A rcu_tasks gp is to wait for the last few insn
* in the tramopline image to finish before releasing
* the trampoline image.
*/
synchronize_rcu_mult(call_rcu, call_rcu_tasks);
if (tasks_rcu && IS_ENABLED(CONFIG_TASKS_RCU))
synchronize_rcu_tasks();
__bpf_struct_ops_map_free(map);
}
@@ -1123,6 +1117,11 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr)
goto errout;
}
if (st_ops_desc->st_ops->cgroup_atype && !(attr->map_flags & BPF_F_LINK)) {
ret = -EOPNOTSUPP;
goto errout;
}
vt = st_ops_desc->value_type;
if (attr->value_size != vt->size) {
ret = -EINVAL;
@@ -1163,6 +1162,8 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr)
mutex_init(&st_map->lock);
bpf_map_init_from_attr(map, attr);
map->free_after_mult_rcu_gp = st_ops_desc->st_ops->free_after_mult_rcu_gp;
map->free_after_rcu_gp = true;
return map;
@@ -1195,6 +1196,7 @@ const struct bpf_map_ops bpf_struct_ops_map_ops = {
.map_alloc_check = bpf_struct_ops_map_alloc_check,
.map_alloc = bpf_struct_ops_map_alloc,
.map_free = bpf_struct_ops_map_free,
.map_free_pre_rcu = bpf_struct_ops_map_free_pre_rcu,
.map_get_next_key = bpf_struct_ops_map_get_next_key,
.map_lookup_elem = bpf_struct_ops_map_lookup_elem,
.map_delete_elem = bpf_struct_ops_map_delete_elem,
@@ -1281,7 +1283,31 @@ int bpf_struct_ops_for_each_prog(const void *kdata,
}
EXPORT_SYMBOL_GPL(bpf_struct_ops_for_each_prog);
static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)
void *bpf_struct_ops_map_kdata(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map;
st_map = container_of(map, struct bpf_struct_ops_map, map);
return st_map->kvalue.data;
}
int bpf_struct_ops_map_cgroup_atype(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map;
st_map = container_of(map, struct bpf_struct_ops_map, map);
return st_map->st_ops_desc->st_ops->cgroup_atype;
}
void *bpf_struct_ops_map_cfi_stubs(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map;
st_map = container_of(map, struct bpf_struct_ops_map, map);
return st_map->st_ops_desc->st_ops->cfi_stubs;
}
bool bpf_struct_ops_valid_to_reg(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
@@ -1297,8 +1323,7 @@ static void bpf_struct_ops_map_link_dealloc(struct bpf_link *link)
struct bpf_struct_ops_map *st_map;
st_link = container_of(link, struct bpf_struct_ops_link, link);
st_map = (struct bpf_struct_ops_map *)
rcu_dereference_protected(st_link->map, true);
st_map = (struct bpf_struct_ops_map *)st_link->map;
if (st_map) {
st_map->st_ops_desc->st_ops->unreg(&st_map->kvalue.data, link);
bpf_map_put(&st_map->map);
@@ -1313,11 +1338,11 @@ static void bpf_struct_ops_map_link_show_fdinfo(const struct bpf_link *link,
struct bpf_map *map;
st_link = container_of(link, struct bpf_struct_ops_link, link);
rcu_read_lock();
map = rcu_dereference(st_link->map);
mutex_lock(&update_mutex);
map = st_link->map;
if (map)
seq_printf(seq, "map_id:\t%d\n", map->id);
rcu_read_unlock();
mutex_unlock(&update_mutex);
}
static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
@@ -1327,11 +1352,31 @@ static int bpf_struct_ops_map_link_fill_link_info(const struct bpf_link *link,
struct bpf_map *map;
st_link = container_of(link, struct bpf_struct_ops_link, link);
rcu_read_lock();
map = rcu_dereference(st_link->map);
mutex_lock(&update_mutex);
map = st_link->map;
if (map)
info->struct_ops.map_id = map->id;
rcu_read_unlock();
mutex_unlock(&update_mutex);
return 0;
}
int bpf_struct_ops_link_update_check(struct bpf_map *new_map,
struct bpf_map *old_map,
struct bpf_map *expected_old_map)
{
struct bpf_struct_ops_map *st_map, *old_st_map;
if (!old_map)
return -ENOLINK;
if (expected_old_map && old_map != expected_old_map)
return -EPERM;
st_map = container_of(new_map, struct bpf_struct_ops_map, map);
old_st_map = container_of(old_map, struct bpf_struct_ops_map, map);
/* The new and old struct_ops must be the same type. */
if (st_map->st_ops_desc != old_st_map->st_ops_desc)
return -EINVAL;
return 0;
}
@@ -1353,30 +1398,19 @@ static int bpf_struct_ops_map_link_update(struct bpf_link *link, struct bpf_map
return -EOPNOTSUPP;
mutex_lock(&update_mutex);
old_map = rcu_dereference_protected(st_link->map, lockdep_is_held(&update_mutex));
if (!old_map) {
err = -ENOLINK;
old_map = st_link->map;
err = bpf_struct_ops_link_update_check(new_map, old_map, expected_old_map);
if (err)
goto err_out;
}
if (expected_old_map && old_map != expected_old_map) {
err = -EPERM;
goto err_out;
}
old_st_map = container_of(old_map, struct bpf_struct_ops_map, map);
/* The new and old struct_ops must be the same type. */
if (st_map->st_ops_desc != old_st_map->st_ops_desc) {
err = -EINVAL;
goto err_out;
}
err = st_map->st_ops_desc->st_ops->update(st_map->kvalue.data, old_st_map->kvalue.data, link);
if (err)
goto err_out;
bpf_map_inc(new_map);
rcu_assign_pointer(st_link->map, new_map);
WRITE_ONCE(st_link->map, new_map);
bpf_map_put(old_map);
err_out:
@@ -1393,7 +1427,7 @@ static int bpf_struct_ops_map_link_detach(struct bpf_link *link)
mutex_lock(&update_mutex);
map = rcu_dereference_protected(st_link->map, lockdep_is_held(&update_mutex));
map = st_link->map;
if (!map) {
mutex_unlock(&update_mutex);
return 0;
@@ -1402,7 +1436,7 @@ static int bpf_struct_ops_map_link_detach(struct bpf_link *link)
st_map->st_ops_desc->st_ops->unreg(&st_map->kvalue.data, link);
RCU_INIT_POINTER(st_link->map, NULL);
WRITE_ONCE(st_link->map, NULL);
/* Pair with bpf_map_get() in bpf_struct_ops_link_create() or
* bpf_map_inc() in bpf_struct_ops_map_link_update().
*/
@@ -1422,7 +1456,7 @@ static __poll_t bpf_struct_ops_map_link_poll(struct file *file,
poll_wait(file, &st_link->wait_hup, pts);
return rcu_access_pointer(st_link->map) ? 0 : EPOLLHUP;
return READ_ONCE(st_link->map) ? 0 : EPOLLHUP;
}
static const struct bpf_link_ops bpf_struct_ops_map_lops = {
@@ -1440,6 +1474,7 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
struct bpf_link_primer link_primer;
struct bpf_struct_ops_map *st_map;
struct bpf_map *map;
int cgroup_atype;
int err;
map = bpf_map_get(attr->link_create.map_fd);
@@ -1453,6 +1488,19 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
goto err_out;
}
cgroup_atype = st_map->st_ops_desc->st_ops->cgroup_atype;
if (cgroup_atype) {
err = cgroup_bpf_struct_ops_attach(map, attr);
bpf_map_put(map);
return err;
}
if (memchr_inv(&attr->link_create.cgroup, 0, sizeof(attr->link_create.cgroup)) ||
attr->link_create.target_fd) {
err = -EINVAL;
goto err_out;
}
link = kzalloc_obj(*link, GFP_USER);
if (!link) {
err = -ENOMEM;
@@ -1478,7 +1526,7 @@ int bpf_struct_ops_link_create(union bpf_attr *attr)
link = NULL;
goto err_out;
}
RCU_INIT_POINTER(link->map, map);
link->map = map;
mutex_unlock(&update_mutex);
return bpf_link_settle(&link_primer);
kernel/bpf/btf.c
+1111 -210
View File
File diff suppressed because it is too large. Load diff
kernel/bpf/btf_vmlinux_inline.c
+29
View File
@@ -0,0 +1,29 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2026, Oracle and/or its affiliates. */
/*
* Provide kernel BTF inline function information for use by BPF tools.
*/
#include <linux/btf.h>
#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#if IS_BUILTIN(CONFIG_DEBUG_INFO_BTF_INLINE)
extern char __start_BTF_inline[];
extern char __stop_BTF_inline[];
#endif
static int __init btf_vmlinux_inline_init(void)
{
#if IS_BUILTIN(CONFIG_DEBUG_INFO_BTF_INLINE)
size_t data_size = __stop_BTF_inline - __start_BTF_inline;
if (data_size)
sysfs_btf_add("vmlinux.inline", __start_BTF_inline, data_size, true, NULL);
#endif
return 0;
}
subsys_initcall(btf_vmlinux_inline_init);
MODULE_DESCRIPTION("BTF inline information for vmlinux");
MODULE_LICENSE("GPL");
kernel/bpf/cfg.c
+242 -129
View File
@@ -9,6 +9,8 @@
#define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args)
#define BPF_MAX_GOTOX_EDGES BPF_COMPLEXITY_LIMIT_INSNS
/* non-recursive DFS pseudo code
* 1 procedure DFS-iterative(G,v):
* 2 label v as discovered
@@ -284,15 +286,17 @@ err_free:
}
/*
* Find and collect all maps which fit in the subprog. Return the result as one
* combined jump table in jt->items (allocated with kvcalloc)
* Collect the jump table of every subprogram that has one, as the combined
* table of all maps whose targets land inside that subprogram. All gotox
* instructions of a subprogram share the same table, so this is done in a
* single pass over the maps rather than once per gotox.
*/
static struct bpf_iarray *jt_from_subprog(struct bpf_verifier_env *env,
int subprog_start, int subprog_end)
static int __compute_subprog_jts(struct bpf_verifier_env *env)
{
struct bpf_iarray *jt = NULL;
struct bpf_subprog_info *subprog;
struct bpf_iarray *jt, *jt_cur;
struct bpf_map *map;
struct bpf_iarray *jt_cur;
u32 old_cnt;
int i;
for (i = 0; i < env->insn_array_map_cnt; i++) {
@@ -303,73 +307,100 @@ static struct bpf_iarray *jt_from_subprog(struct bpf_verifier_env *env,
map = env->insn_array_maps[i];
jt_cur = jt_from_map(map);
if (IS_ERR(jt_cur)) {
kvfree(jt);
return jt_cur;
if (IS_ERR(jt_cur))
return PTR_ERR(jt_cur);
subprog = bpf_find_containing_subprog(env, jt_cur->items[0]);
if (verifier_bug_if(!subprog, env, "no subprog contains insn %u of an insn array map",
jt_cur->items[0])) {
kvfree(jt_cur);
return -EFAULT;
}
if (jt_cur->items[jt_cur->cnt - 1] >= (subprog + 1)->start) {
verbose(env, "jump table of subprog starting at %u spans multiple subprogs, map id=%d\n",
subprog->start, map->id);
bpf_diag_program_structure(
env, subprog->start, "jump table spans subprograms",
"Keep every entry of a jump table inside one subprogram.",
"The jump table in map id %d for the subprogram that starts at instruction %u reaches past its end at instruction %u.",
map->id, subprog->start, (subprog + 1)->start);
kvfree(jt_cur);
return -EINVAL;
}
/*
* This is enough to check one element. The full table is
* checked to fit inside the subprog later in create_jt()
*/
if (jt_cur->items[0] >= subprog_start && jt_cur->items[0] < subprog_end) {
u32 old_cnt = jt ? jt->cnt : 0;
jt = bpf_iarray_realloc(jt, old_cnt + jt_cur->cnt);
if (!jt) {
kvfree(jt_cur);
return ERR_PTR(-ENOMEM);
}
memcpy(jt->items + old_cnt, jt_cur->items, jt_cur->cnt << 2);
old_cnt = subprog->jt ? subprog->jt->cnt : 0;
jt = bpf_iarray_realloc(subprog->jt, old_cnt + jt_cur->cnt);
if (!jt) {
subprog->jt = NULL;
kvfree(jt_cur);
return -ENOMEM;
}
memcpy(jt->items + old_cnt, jt_cur->items, jt_cur->cnt << 2);
subprog->jt = jt;
kvfree(jt_cur);
}
if (!jt) {
for (i = 0; i < env->subprog_cnt; i++) {
jt = env->subprog_info[i].jt;
if (jt)
jt->cnt = sort_insn_array_uniq(jt->items, jt->cnt);
}
env->cfg.subprog_jts_ready = true;
return 0;
}
/* Leave nothing behind on failure. */
static int compute_subprog_jts(struct bpf_verifier_env *env)
{
int err;
err = __compute_subprog_jts(env);
if (err)
bpf_free_subprog_jts(env);
return err;
}
void bpf_free_subprog_jts(struct bpf_verifier_env *env)
{
int i;
for (i = 0; i < ARRAY_SIZE(env->subprog_info); i++) {
kvfree(env->subprog_info[i].jt);
env->subprog_info[i].jt = NULL;
}
env->cfg.subprog_jts_ready = false;
}
static struct bpf_iarray *
subprog_jt(int t, struct bpf_verifier_env *env)
{
struct bpf_subprog_info *subprog;
int subprog_start, err;
if (!env->cfg.subprog_jts_ready) {
err = compute_subprog_jts(env);
if (err)
return ERR_PTR(err);
}
subprog = bpf_find_containing_subprog(env, t);
if (verifier_bug_if(!subprog, env, "no subprog contains insn %d", t))
return ERR_PTR(-EFAULT);
subprog_start = subprog->start;
if (!subprog->jt) {
verbose(env, "no jump tables found for subprog starting at %u\n", subprog_start);
bpf_diag_program_structure(
env, subprog_start, "missing jump table",
env, t, "missing jump table",
"Make sure subprograms containing gotox instructions are accompanied by jump tables referencing these subprograms.",
"No jump table was found for the subprogram that starts at instruction %u.",
subprog_start);
return ERR_PTR(-EINVAL);
}
jt->cnt = sort_insn_array_uniq(jt->items, jt->cnt);
return jt;
}
static struct bpf_iarray *
create_jt(int t, struct bpf_verifier_env *env)
{
struct bpf_subprog_info *subprog;
int subprog_start, subprog_end;
struct bpf_iarray *jt;
int i;
subprog = bpf_find_containing_subprog(env, t);
subprog_start = subprog->start;
subprog_end = (subprog + 1)->start;
jt = jt_from_subprog(env, subprog_start, subprog_end);
if (IS_ERR(jt))
return jt;
/* Check that the every element of the jump table fits within the given subprogram */
for (i = 0; i < jt->cnt; i++) {
if (jt->items[i] < subprog_start || jt->items[i] >= subprog_end) {
verbose(env, "jump table for insn %d points outside of the subprog [%u,%u]\n",
t, subprog_start, subprog_end);
bpf_diag_program_structure(
env, t, "jump table target out of range",
"Keep every jump-table target inside the same subprogram.",
"The jump table for instruction %d points outside subprogram range [%u,%u).",
t, subprog_start, subprog_end);
kvfree(jt);
return ERR_PTR(-EINVAL);
}
}
return jt;
return subprog->jt;
}
/* "conditional jump with N edges" */
@@ -381,27 +412,39 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
struct bpf_iarray *jt;
int i, w;
jt = env->insn_aux_data[t].jt;
if (!jt) {
jt = create_jt(t, env);
if (IS_ERR(jt))
return PTR_ERR(jt);
/*
* The first visit pushes every target that still needs exploring, and
* those are all done by the time this insn is revisited, so there is
* nothing left to walk the jump table for.
*/
if (insn_state[t] & BRANCH)
return DONE_EXPLORING;
env->insn_aux_data[t].jt = jt;
jt = subprog_jt(t, env);
if (IS_ERR(jt))
return PTR_ERR(jt);
insn_state[t] |= BRANCH;
if (check_add_overflow(env->cfg.gotox_edges, jt->cnt,
&env->cfg.gotox_edges) ||
env->cfg.gotox_edges > BPF_MAX_GOTOX_EDGES) {
verbose(env, "number of indirect jump edges in the program exceeds %u\n",
BPF_MAX_GOTOX_EDGES);
bpf_diag_program_structure(
env, t, "too many indirect jump edges",
"Reduce the number of indirect jumps, or the number of distinct targets they can reach.",
"The program has more than %u indirect jump edges in total, counted over every gotox instruction.",
BPF_MAX_GOTOX_EDGES);
return -E2BIG;
}
mark_prune_point(env, t);
for (i = 0; i < jt->cnt; i++) {
w = jt->items[i];
if (w < 0 || w >= env->prog->len) {
verbose(env, "indirect jump out of range from insn %d to %d\n", t, w);
bpf_diag_program_structure(
env, t, "indirect jump out of range",
"Keep indirect jump targets inside the program.",
"Instruction %d can jump indirectly to instruction %d, but the program only contains instructions 0 through %d.",
t, w, env->prog->len - 1);
return -EINVAL;
}
if (verifier_bug_if(w < 0 || w >= env->prog->len, env,
"indirect jump from insn %d to %d out of range", t, w))
return -EFAULT;
mark_jmp_point(env, w);
mark_jump_target(env, w);
@@ -422,27 +465,73 @@ static int visit_gotox_insn(int t, struct bpf_verifier_env *env)
}
/*
* Instructions that can abnormally return from a subprog (tail_call
* upon success, ld_{abs,ind} upon load failure) have a hidden exit
* that the verifier must account for.
* Return pointers to functions in the read-only map that ld_imm64 instruction
* 't' loads the address of, or of its value, if there are any.
*/
static int visit_abnormal_return_insn(struct bpf_verifier_env *env, int t)
static struct bpf_func_ptr *insn_func_ptrs(struct bpf_verifier_env *env, int t, u32 *cnt)
{
struct bpf_subprog_info *subprog;
struct bpf_iarray *jt;
struct bpf_insn *insn = &env->prog->insnsi[t];
if (env->insn_aux_data[t].jt)
return 0;
*cnt = 0;
if (!env->func_ptr_cnt || !bpf_is_ldimm64(insn))
return NULL;
if (insn->src_reg != BPF_PSEUDO_MAP_VALUE &&
insn->src_reg != BPF_PSEUDO_MAP_IDX_VALUE &&
insn->src_reg != BPF_PSEUDO_MAP_FD &&
insn->src_reg != BPF_PSEUDO_MAP_IDX)
return NULL;
jt = bpf_iarray_realloc(NULL, 2);
if (!jt)
return -ENOMEM;
return bpf_map_func_ptrs(env, env->used_maps[env->insn_aux_data[t].map_index], cnt);
}
subprog = bpf_find_containing_subprog(env, t);
jt->items[0] = t + 1;
jt->items[1] = subprog->exit_idx;
env->insn_aux_data[t].jt = jt;
return 0;
/*
* ld_imm64 that loads the address of a map that has pointers to functions
* is similar to ld_imm64 with BPF_PSEUDO_FUNC that loads the address of one
* function: any of them may be read from the map and called via callx later.
* Treat it as a call of all of them.
*/
static int visit_func_ptrs_insn(int t, struct bpf_verifier_env *env,
struct bpf_func_ptr *ptrs, u32 cnt)
{
int *insn_stack = env->cfg.insn_stack;
int *insn_state = env->cfg.insn_state;
int ret, w;
u32 i;
ret = push_insn(t, t + 2, FALLTHROUGH, env);
if (ret)
return ret;
/*
* Explore one function at a time like push_insn() does, so that
* DISCOVERED function is a caller of 't' and not the one that waits in
* insn_stack[], which merge_callee_effects() relies on.
*/
for (i = 0; i < cnt; i++) {
w = ptrs[i].xlated_off;
if (insn_state[w])
continue;
if (env->cfg.cur_stack >= env->prog->len)
return -E2BIG;
insn_stack[env->cfg.cur_stack++] = w;
insn_state[w] = DISCOVERED;
return KEEP_EXPLORING;
}
/* all functions are explored, their effects are known */
mark_prune_point(env, t);
for (i = 0; i < cnt; i++) {
w = ptrs[i].xlated_off;
merge_callee_effects(env, t, w);
/* the same marks as push_insn() leaves on a branch target */
mark_prune_point(env, w);
mark_jmp_point(env, w);
mark_jump_target(env, w);
}
return DONE_EXPLORING;
}
/* Visits the instruction at index t and returns one of the following:
@@ -453,21 +542,20 @@ static int visit_abnormal_return_insn(struct bpf_verifier_env *env, int t)
static int visit_insn(int t, struct bpf_verifier_env *env)
{
struct bpf_insn *insns = env->prog->insnsi, *insn = &insns[t];
struct bpf_func_ptr *ptrs;
int ret, off, insn_sz;
u32 cnt;
if (bpf_pseudo_func(insn))
return visit_func_call_insn(t, insns, env, true);
ptrs = insn_func_ptrs(env, t, &cnt);
if (ptrs)
return visit_func_ptrs_insn(t, env, ptrs, cnt);
/* All non-branch instructions have a single fall-through edge. */
if (BPF_CLASS(insn->code) != BPF_JMP &&
BPF_CLASS(insn->code) != BPF_JMP32) {
if (BPF_CLASS(insn->code) == BPF_LD &&
(BPF_MODE(insn->code) == BPF_ABS ||
BPF_MODE(insn->code) == BPF_IND)) {
ret = visit_abnormal_return_insn(env, t);
if (ret)
return ret;
}
insn_sz = bpf_is_ldimm64(insn) ? 2 : 1;
return push_insn(t, t + insn_sz, FALLTHROUGH, env);
}
@@ -510,13 +598,10 @@ static int visit_insn(int t, struct bpf_verifier_env *env)
*/
if (ret == 0 && fp->might_sleep)
mark_subprog_might_sleep(env, t);
if (bpf_helper_changes_pkt_data(insn->imm))
/* nobody knows yet whether the dynptr is that of an skb */
if (bpf_helper_changes_pkt_data(insn->imm) ||
insn->imm == BPF_FUNC_dynptr_write)
mark_subprog_changes_pkt_data(env, t);
if (insn->imm == BPF_FUNC_tail_call) {
ret = visit_abnormal_return_insn(env, t);
if (ret)
return ret;
}
} else if (insn->src_reg == BPF_PSEUDO_KFUNC_CALL) {
struct bpf_call_arg_meta meta;
@@ -684,6 +769,11 @@ err_free:
return ret;
}
struct dfs_state {
u32 traversed:1;
u32 next_succ:31;
};
/*
* For each subprogram 'i' fill array env->cfg.insn_subprogram sub-range
* [env->subprog_info[i].postorder_start, env->subprog_info[i+1].postorder_start)
@@ -691,43 +781,52 @@ err_free:
*/
int bpf_compute_postorder(struct bpf_verifier_env *env)
{
u32 cur_postorder, i, top, stack_sz, s;
int *stack = NULL, *postorder = NULL, *state = NULL;
struct bpf_iarray *succ;
int *stack = NULL, *postorder = NULL, *postorder_nums = NULL;
int subprog_idx, stack_sz, cur, s, cur_postorder, start;
struct dfs_state *state = NULL;
struct bpf_iarray *succ = NULL;
postorder_nums = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
postorder = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
state = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
stack = kvzalloc_objs(int, env->prog->len, GFP_KERNEL_ACCOUNT);
if (!postorder || !state || !stack) {
state = kvzalloc_objs(struct dfs_state, env->prog->len, GFP_KERNEL_ACCOUNT);
if (!postorder_nums || !postorder || !stack || !state) {
kvfree(postorder_nums);
kvfree(postorder);
kvfree(state);
kvfree(stack);
kvfree(state);
return -ENOMEM;
}
cur_postorder = 0;
for (i = 0; i < env->subprog_cnt; i++) {
env->subprog_info[i].postorder_start = cur_postorder;
stack[0] = env->subprog_info[i].start;
for (subprog_idx = 0; subprog_idx < env->subprog_cnt; subprog_idx++) {
start = env->subprog_info[subprog_idx].start;
env->subprog_info[subprog_idx].postorder_start = cur_postorder;
stack[0] = start;
stack_sz = 1;
state[start].traversed = true;
do {
top = stack[stack_sz - 1];
state[top] |= DISCOVERED;
if (state[top] & EXPLORED) {
postorder[cur_postorder++] = top;
cur = stack[stack_sz - 1];
succ = bpf_insn_successors(env, cur);
if (state[cur].next_succ == succ->cnt) {
postorder_nums[cur] = cur_postorder;
postorder[cur_postorder] = cur;
cur_postorder++;
stack_sz--;
continue;
}
succ = bpf_insn_successors(env, top);
for (s = 0; s < succ->cnt; ++s) {
if (!state[succ->items[s]]) {
stack[stack_sz++] = succ->items[s];
state[succ->items[s]] |= DISCOVERED;
}
s = succ->items[state[cur].next_succ];
if (!state[s].traversed) {
state[s].traversed = true;
state[s].next_succ = 0;
stack[stack_sz] = s;
stack_sz++;
continue;
}
state[top] |= EXPLORED;
state[cur].next_succ++;
} while (stack_sz);
}
env->subprog_info[i].postorder_start = cur_postorder;
env->subprog_info[subprog_idx].postorder_start = cur_postorder;
env->cfg.postorder_nums = postorder_nums;
env->cfg.insn_postorder = postorder;
env->cfg.cur_postorder = cur_postorder;
kvfree(stack);
@@ -749,7 +848,7 @@ int bpf_compute_scc(struct bpf_verifier_env *env)
struct bpf_insn_aux_data *aux = env->insn_aux_data;
const u32 insn_cnt = env->prog->len;
int stack_sz, dfs_sz, err = 0;
u32 *stack, *pre, *low, *dfs;
u32 *stack, *pre, *low, *dfs, *dfs_pos;
u32 i, j, t, w;
u32 next_preorder_num;
u32 next_scc_id;
@@ -762,13 +861,16 @@ int bpf_compute_scc(struct bpf_verifier_env *env)
* - 'stack' accumulates vertices in DFS order, see invariant comment below;
* - 'pre[t] == p' => preorder number of vertex 't' is 'p';
* - 'low[t] == n' => smallest preorder number of the vertex reachable from 't' is 'n';
* - 'dfs' DFS traversal stack, used to emulate explicit recursion.
* - 'dfs' DFS traversal stack, used to emulate explicit recursion;
* - 'dfs_pos[k] == j' => the frame 'dfs[k]' resumes visiting its
* successors at index 'j'.
*/
stack = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
pre = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
low = kvcalloc(insn_cnt, sizeof(int), GFP_KERNEL_ACCOUNT);
dfs = kvcalloc(insn_cnt, sizeof(*dfs), GFP_KERNEL_ACCOUNT);
if (!stack || !pre || !low || !dfs) {
dfs_pos = kvcalloc(insn_cnt, sizeof(*dfs_pos), GFP_KERNEL_ACCOUNT);
if (!stack || !pre || !low || !dfs || !dfs_pos) {
err = -ENOMEM;
goto exit;
}
@@ -851,6 +953,7 @@ int bpf_compute_scc(struct bpf_verifier_env *env)
stack_sz = 0;
dfs_sz = 1;
dfs[0] = i;
dfs_pos[0] = 0;
dfs_continue:
while (dfs_sz) {
w = dfs[dfs_sz - 1];
@@ -860,13 +963,22 @@ dfs_continue:
next_preorder_num++;
stack[stack_sz++] = w;
}
/* Visit 'w' successors */
/* Visit remaining 'w' successors */
succ = bpf_insn_successors(env, w);
for (j = 0; j < succ->cnt; ++j) {
for (j = dfs_pos[dfs_sz - 1]; j < succ->cnt; ++j) {
if (pre[succ->items[j]]) {
low[w] = min(low[w], low[succ->items[j]]);
} else {
dfs[dfs_sz++] = succ->items[j];
/*
* Once DFS for succ->items[j] is complete,
* pre[succ->items[j]] is non-zero, hence
* resuming at 'j' allows to follow the
* low[w] = min(...) update branch above.
*/
dfs_pos[dfs_sz - 1] = j;
dfs_pos[dfs_sz] = 0;
dfs[dfs_sz] = succ->items[j];
dfs_sz++;
goto dfs_continue;
}
}
@@ -916,5 +1028,6 @@ exit:
kvfree(pre);
kvfree(low);
kvfree(dfs);
kvfree(dfs_pos);
return err;
}
kernel/bpf/cgroup.c
+464 -36
View File
@@ -24,6 +24,34 @@
DEFINE_STATIC_KEY_ARRAY_FALSE(cgroup_bpf_enabled_key, MAX_CGROUP_BPF_ATTACH_TYPE);
EXPORT_SYMBOL(cgroup_bpf_enabled_key);
struct cgroup_struct_ops {
u32 type_id;
void *cfi_stubs;
bool mult_rcu;
};
static struct cgroup_struct_ops cgroup_struct_ops[MAX_CGROUP_BPF_ATTACH_TYPE];
void cgroup_bpf_struct_ops_register(int atype, u32 type_id, void *cfi_stubs, bool mult_rcu)
{
cgroup_struct_ops[atype].type_id = type_id;
cgroup_struct_ops[atype].cfi_stubs = cfi_stubs;
cgroup_struct_ops[atype].mult_rcu = mult_rcu;
}
static enum cgroup_bpf_attach_type find_atype_by_struct_ops_id(u32 type_id)
{
enum cgroup_bpf_attach_type atype;
for (atype = 0; atype < MAX_CGROUP_BPF_ATTACH_TYPE; atype++) {
if (cgroup_bpf_is_struct_ops_atype(atype) &&
cgroup_struct_ops[atype].type_id &&
cgroup_struct_ops[atype].type_id == type_id)
return atype;
}
return CGROUP_BPF_ATTACH_TYPE_INVALID;
}
/*
* cgroup bpf destruction makes heavy use of work items and there can be a lot
* of concurrent destructions. Use a separate workqueue so that cgroup bpf
@@ -306,6 +334,19 @@ static void bpf_cgroup_storages_link(struct bpf_cgroup_storage *storages[],
bpf_cgroup_storage_link(storages[stype], cgrp, attach_type);
}
static void cgroup_struct_ops_link_detach_wake(struct bpf_cgroup_link *link, bool wake_poll)
{
cgroup_put(link->cgroup);
link->cgroup = NULL;
bpf_map_put(link->map);
/* READ_ONCE in cgroup_struct_ops_link_poll */
WRITE_ONCE(link->map, NULL);
if (wake_poll)
wake_up_interruptible_poll(&link->wait_hup, EPOLLHUP);
}
/* Called when bpf_cgroup_link is auto-detached from dying cgroup.
* It drops cgroup and bpf_prog refcounts, and marks bpf_link as defunct. It
* doesn't free link memory, which will eventually be done by bpf_link's
@@ -313,10 +354,96 @@ static void bpf_cgroup_storages_link(struct bpf_cgroup_storage *storages[],
*/
static void bpf_cgroup_link_auto_detach(struct bpf_cgroup_link *link)
{
if (link->map) {
cgroup_struct_ops_link_detach_wake(link, true);
return;
}
if (link->link.prog->expected_attach_type == BPF_LSM_CGROUP)
bpf_trampoline_unlink_cgroup_shim(link->link.prog);
cgroup_put(link->cgroup);
link->cgroup = NULL;
}
static void bpf_cgroup_array_free_rcu(struct rcu_head *rcu)
{
kfree(container_of(rcu, struct bpf_prog_array, rcu));
}
static void bpf_cgroup_array_free(struct bpf_prog_array *array,
enum cgroup_bpf_attach_type atype)
{
if (!array || array == &bpf_empty_prog_array)
return;
if (cgroup_struct_ops[atype].mult_rcu)
/* RCU tasks trace grace period implies RCU grace period. */
call_rcu_tasks_trace(&array->rcu, bpf_cgroup_array_free_rcu);
else
kfree_rcu(array, rcu);
}
static void *bpf_cgroup_array_dummy(enum cgroup_bpf_attach_type atype)
{
if (cgroup_bpf_is_struct_ops_atype(atype))
return cgroup_struct_ops[atype].cfi_stubs;
return bpf_prog_dummy();
}
static int bpf_cgroup_array_length(struct bpf_prog_array *array,
enum cgroup_bpf_attach_type atype)
{
struct bpf_prog_array_item *item;
int cnt = 0;
for (item = array->items; item->prog; item++) {
if (item->prog != bpf_cgroup_array_dummy(atype))
cnt++;
}
return cnt;
}
static int bpf_cgroup_array_copy_to_user(struct bpf_prog_array *array,
__u32 __user *prog_ids, int cnt,
enum cgroup_bpf_attach_type atype)
{
struct bpf_prog_array_item *item;
int i = 0;
u32 id;
for (item = array->items; item->prog && i < cnt; item++) {
if (item->prog == bpf_cgroup_array_dummy(atype))
continue;
if (cgroup_bpf_is_struct_ops_atype(atype))
id = bpf_struct_ops_id(item->kdata);
else
id = item->prog->aux->id;
if (copy_to_user(prog_ids + i, &id, sizeof(id)))
return -EFAULT;
i++;
}
return item->prog ? -ENOSPC : 0;
}
static int bpf_cgroup_array_delete_safe_at(struct bpf_prog_array *array,
int index, enum cgroup_bpf_attach_type atype)
{
struct bpf_prog_array_item *item;
for (item = array->items; item->prog; item++) {
if (item->prog == bpf_cgroup_array_dummy(atype))
continue;
if (!index) {
WRITE_ONCE(item->prog, bpf_cgroup_array_dummy(atype));
return 0;
}
index--;
}
return -ENOENT;
}
/**
* cgroup_bpf_release() - put references of all bpf programs and
* release all cgroup bpf data
@@ -346,18 +473,15 @@ static void cgroup_bpf_release(struct work_struct *work)
bpf_trampoline_unlink_cgroup_shim(pl->prog);
bpf_prog_put(pl->prog);
}
if (pl->link) {
if (pl->link->link.prog->expected_attach_type == BPF_LSM_CGROUP)
bpf_trampoline_unlink_cgroup_shim(pl->link->link.prog);
if (pl->link)
bpf_cgroup_link_auto_detach(pl->link);
}
kfree(pl);
static_branch_dec(&cgroup_bpf_enabled_key[atype]);
}
old_array = rcu_dereference_protected(
cgrp->bpf.effective[atype],
lockdep_is_held(&cgroup_mutex));
bpf_prog_array_free(old_array);
bpf_cgroup_array_free(old_array, atype);
}
list_for_each_entry_safe(storage, stmp, storages, list_cg) {
@@ -399,6 +523,37 @@ static struct bpf_prog *prog_list_prog(struct bpf_prog_list *pl)
return NULL;
}
static bool prog_list_is_struct_ops(const struct bpf_prog_list *pl)
{
return pl->link && pl->link->map;
}
static void prog_list_init_item(struct bpf_prog_list *pl, struct bpf_prog_array_item *item)
{
if (prog_list_is_struct_ops(pl)) {
item->kdata = bpf_struct_ops_map_kdata(pl->link->map);
return;
}
item->prog = prog_list_prog(pl);
bpf_cgroup_storages_assign(item->cgroup_storage, pl->storage);
}
static void prog_list_replace_item(struct bpf_prog_list *pl, struct bpf_prog_array_item *item)
{
if (prog_list_is_struct_ops(pl))
WRITE_ONCE(item->kdata, bpf_struct_ops_map_kdata(pl->link->map));
else
WRITE_ONCE(item->prog, pl->link->link.prog);
}
static u32 prog_list_id(struct bpf_prog_list *pl)
{
if (prog_list_is_struct_ops(pl))
return pl->link->map->id;
return prog_list_prog(pl)->aux->id;
}
/* count number of elements in the list.
* it's slow but the list cannot be long
*/
@@ -408,7 +563,7 @@ static u32 prog_list_length(struct hlist_head *head, int *preorder_cnt)
u32 cnt = 0;
hlist_for_each_entry(pl, head, node) {
if (!prog_list_prog(pl))
if (!pl->prog && !pl->link)
continue;
if (preorder_cnt && (pl->flags & BPF_F_PREORDER))
(*preorder_cnt)++;
@@ -482,7 +637,7 @@ static int compute_effective_progs(struct cgroup *cgrp,
init_bstart = bstart;
hlist_for_each_entry(pl, &p->bpf.progs[atype], node) {
if (!prog_list_prog(pl))
if (!pl->prog && !pl->link)
continue;
if (pl->flags & BPF_F_PREORDER) {
@@ -492,9 +647,7 @@ static int compute_effective_progs(struct cgroup *cgrp,
item = &progs->items[fstart];
fstart++;
}
item->prog = prog_list_prog(pl);
bpf_cgroup_storages_assign(item->cgroup_storage,
pl->storage);
prog_list_init_item(pl, item);
cnt++;
}
@@ -517,7 +670,7 @@ static void activate_effective_progs(struct cgroup *cgrp,
/* free prog array after grace period, since __cgroup_bpf_run_*()
* might be still walking the array
*/
bpf_prog_array_free(old_array);
bpf_cgroup_array_free(old_array, atype);
}
/**
@@ -557,7 +710,7 @@ static int cgroup_bpf_inherit(struct cgroup *cgrp)
return 0;
cleanup:
for (i = 0; i < NR; i++)
bpf_prog_array_free(arrays[i]);
bpf_cgroup_array_free(arrays[i], i);
for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))
cgroup_bpf_put(p);
@@ -612,7 +765,7 @@ static int update_effective_progs(struct cgroup *cgrp,
if (percpu_ref_is_zero(&desc->bpf.refcnt)) {
if (unlikely(desc->bpf.inactive)) {
bpf_prog_array_free(desc->bpf.inactive);
bpf_cgroup_array_free(desc->bpf.inactive, atype);
desc->bpf.inactive = NULL;
}
continue;
@@ -631,7 +784,7 @@ cleanup:
css_for_each_descendant_pre(css, &cgrp->self) {
struct cgroup *desc = container_of(css, struct cgroup, self);
bpf_prog_array_free(desc->bpf.inactive);
bpf_cgroup_array_free(desc->bpf.inactive, atype);
desc->bpf.inactive = NULL;
}
@@ -870,7 +1023,7 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
old_pl_flags = pl->flags;
bpf_cgroup_storages_assign(old_storage, pl->storage);
} else {
pl = kmalloc_obj(*pl);
pl = kzalloc_obj(*pl);
if (!pl) {
bpf_cgroup_storages_free(new_storage);
return -ENOMEM;
@@ -973,9 +1126,10 @@ static int effective_prog_pos(struct cgroup *cgrp,
init_bstart = bstart;
hlist_for_each_entry(pl, &p->bpf.progs[atype], node) {
if (!prog_list_prog(pl))
continue;
/*
* No detaching pl (NULL prog and link) is visible to the callers,
* so skip the check compute_effective_progs() needs.
*/
if (pl->flags & BPF_F_PREORDER) {
if (pl == target_pl)
pos = bstart;
@@ -1022,7 +1176,7 @@ static void replace_effective_prog(struct cgroup *cgrp,
desc->bpf.effective[atype],
lockdep_is_held(&cgroup_mutex));
item = &progs->items[pos];
WRITE_ONCE(item->prog, pl->link->link.prog);
prog_list_replace_item(pl, item);
}
}
@@ -1177,7 +1331,7 @@ static void purge_effective_progs(struct cgroup *cgrp, struct bpf_prog_list *pl,
lockdep_is_held(&cgroup_mutex));
/* Remove the program from the array */
WARN_ONCE(bpf_prog_array_delete_safe_at(progs, pos),
WARN_ONCE(bpf_cgroup_array_delete_safe_at(progs, pos, atype),
"Failed to purge a prog from array at index %d", pos);
}
}
@@ -1287,7 +1441,15 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
if (effective_query && prog_attach_flags)
return -EINVAL;
if (type == BPF_LSM_CGROUP) {
if (type == BPF_STRUCT_OPS) {
u32 type_id = attr->query.type_id;
atype = find_atype_by_struct_ops_id(type_id);
if (atype == CGROUP_BPF_ATTACH_TYPE_INVALID)
return -ENOENT;
from_atype = to_atype = atype;
flags = 0;
} else if (type == BPF_LSM_CGROUP) {
if (!effective_query && attr->query.prog_cnt &&
prog_ids && !prog_attach_flags)
return -EINVAL;
@@ -1307,7 +1469,7 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
if (effective_query) {
effective = rcu_dereference_protected(cgrp->bpf.effective[atype],
lockdep_is_held(&cgroup_mutex));
total_cnt += bpf_prog_array_length(effective);
total_cnt += bpf_cgroup_array_length(effective, atype);
} else {
total_cnt += prog_list_length(&cgrp->bpf.progs[atype], NULL);
}
@@ -1337,35 +1499,33 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
if (effective_query) {
effective = rcu_dereference_protected(cgrp->bpf.effective[atype],
lockdep_is_held(&cgroup_mutex));
cnt = min_t(int, bpf_prog_array_length(effective), total_cnt);
ret = bpf_prog_array_copy_to_user(effective, prog_ids, cnt);
cnt = min_t(int, bpf_cgroup_array_length(effective, atype), total_cnt);
ret = bpf_cgroup_array_copy_to_user(effective, prog_ids, cnt, atype);
} else {
struct hlist_head *progs;
struct bpf_prog_list *pl;
struct bpf_prog *prog;
u32 id;
progs = &cgrp->bpf.progs[atype];
cnt = min_t(int, prog_list_length(progs, NULL), total_cnt);
i = 0;
hlist_for_each_entry(pl, progs, node) {
prog = prog_list_prog(pl);
id = prog->aux->id;
id = prog_list_id(pl);
if (copy_to_user(prog_ids + i, &id, sizeof(id)))
return -EFAULT;
if (prog_attach_flags) {
flags = cgrp->bpf.flags[atype] |
(pl->flags & BPF_F_PREORDER);
if (copy_to_user(prog_attach_flags + i,
&flags, sizeof(flags)))
return -EFAULT;
}
if (++i == cnt)
break;
}
if (prog_attach_flags) {
flags = cgrp->bpf.flags[atype];
for (i = 0; i < cnt; i++)
if (copy_to_user(prog_attach_flags + i,
&flags, sizeof(flags)))
return -EFAULT;
if (prog_attach_flags)
prog_attach_flags += cnt;
}
}
prog_ids += cnt;
@@ -2329,7 +2489,7 @@ static const struct bpf_func_proto bpf_sysctl_get_name_proto = {
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_WRITE,
.arg2_type = ARG_PTR_TO_MEM | MEM_UNINIT | MEM_WRITE,
.arg3_type = ARG_MEM_SIZE,
.arg4_type = ARG_ANYTHING,
};
@@ -2772,6 +2932,274 @@ const struct bpf_verifier_ops cg_sockopt_verifier_ops = {
const struct bpf_prog_ops cg_sockopt_prog_ops = {
};
static int __cgroup_struct_ops_link_detach(struct bpf_link *link, bool wake_poll)
{
struct bpf_cgroup_link *cg_link = container_of(link, struct bpf_cgroup_link, link);
enum cgroup_bpf_attach_type atype;
struct bpf_prog_list *pl;
struct bpf_map *map;
struct cgroup *cgrp;
cgroup_lock();
cgrp = cg_link->cgroup;
if (!cgrp) {
cgroup_unlock();
return 0;
}
map = cg_link->map;
atype = bpf_struct_ops_map_cgroup_atype(map);
hlist_for_each_entry(pl, &cgrp->bpf.progs[atype], node) {
if (pl->link == cg_link)
break;
}
if (WARN_ON_ONCE(!pl)) {
cgroup_unlock();
return -ENOENT;
}
/* mark deleted so compute_effective_progs() skips it */
pl->link = NULL;
if (update_effective_progs(cgrp, atype)) {
pl->link = cg_link;
purge_effective_progs(cgrp, pl, atype);
}
hlist_del(&pl->node);
cgroup_struct_ops_link_detach_wake(cg_link, wake_poll);
cgrp->bpf.revisions[atype]++;
kfree(pl);
static_branch_dec(&cgroup_bpf_enabled_key[atype]);
cgroup_unlock();
return 0;
}
static int cgroup_struct_ops_link_detach(struct bpf_link *link)
{
return __cgroup_struct_ops_link_detach(link, true);
}
static void cgroup_struct_ops_link_dealloc(struct bpf_link *link)
{
struct bpf_cgroup_link *cg_link = container_of(link, struct bpf_cgroup_link, link);
__cgroup_struct_ops_link_detach(link, false);
kfree(cg_link);
}
static void cgroup_struct_ops_link_show_fdinfo(const struct bpf_link *link, struct seq_file *seq)
{
struct bpf_cgroup_link *cg_link =
container_of(link, struct bpf_cgroup_link, link);
cgroup_lock();
if (!cg_link->cgroup) {
cgroup_unlock();
return;
}
seq_printf(seq, "map_id:\t%u\n", cg_link->map->id);
seq_printf(seq, "cgroup_id:\t%llu\n", cgroup_id(cg_link->cgroup));
cgroup_unlock();
}
static int cgroup_struct_ops_link_fill_link_info(const struct bpf_link *link,
struct bpf_link_info *info)
{
struct bpf_cgroup_link *cg_link = container_of(link, struct bpf_cgroup_link, link);
cgroup_lock();
if (!cg_link->cgroup) {
cgroup_unlock();
return 0;
}
info->struct_ops.map_id = cg_link->map->id;
info->struct_ops.cgroup_id = cgroup_id(cg_link->cgroup);
cgroup_unlock();
return 0;
}
static int cgroup_struct_ops_link_update(struct bpf_link *link, struct bpf_map *new_map,
struct bpf_map *expected_old_map)
{
struct bpf_cgroup_link *cg_link = container_of(link, struct bpf_cgroup_link, link);
enum cgroup_bpf_attach_type atype;
struct bpf_prog_list *pl;
struct bpf_map *old_map;
struct cgroup *cgrp;
bool found = false;
int err;
if (!bpf_struct_ops_valid_to_reg(new_map))
return -EINVAL;
cgroup_lock();
cgrp = cg_link->cgroup;
if (!cgrp) {
err = -ENOLINK;
goto out;
}
old_map = cg_link->map;
err = bpf_struct_ops_link_update_check(new_map, old_map, expected_old_map);
if (err)
goto out;
atype = bpf_struct_ops_map_cgroup_atype(new_map);
hlist_for_each_entry(pl, &cgrp->bpf.progs[atype], node) {
if (pl->link == cg_link) {
found = true;
break;
}
}
if (!found) {
err = -ENOENT;
goto out;
}
bpf_map_inc(new_map);
WRITE_ONCE(cg_link->map, new_map);
replace_effective_prog(cgrp, atype, pl);
bpf_map_put(old_map);
cgrp->bpf.revisions[atype]++;
out:
cgroup_unlock();
return err;
}
static __poll_t cgroup_struct_ops_link_poll(struct file *file, struct poll_table_struct *pts)
{
struct bpf_cgroup_link *link = file->private_data;
poll_wait(file, &link->wait_hup, pts);
return READ_ONCE(link->map) ? 0 : EPOLLHUP;
}
static const struct bpf_link_ops cgroup_struct_ops_link_ops = {
.dealloc = cgroup_struct_ops_link_dealloc,
.detach = cgroup_struct_ops_link_detach,
.show_fdinfo = cgroup_struct_ops_link_show_fdinfo,
.fill_link_info = cgroup_struct_ops_link_fill_link_info,
.update_map = cgroup_struct_ops_link_update,
.poll = cgroup_struct_ops_link_poll,
};
int cgroup_bpf_struct_ops_attach(struct bpf_map *map, const union bpf_attr *attr)
{
u32 flags = attr->link_create.flags;
u32 pl_flags = (flags & BPF_F_PREORDER) | BPF_F_ALLOW_MULTI;
enum cgroup_bpf_attach_type atype;
struct bpf_link_primer link_primer;
struct bpf_cgroup_link *link;
struct bpf_prog_list *pl = NULL;
struct hlist_head *progs;
struct cgroup *cgrp;
int err;
if (flags & ~BPF_F_LINK_ATTACH_MASK)
return -EINVAL;
/*
* Attaching struct_ops to cgroup is through link only. All relative
* position must be corresponding to a link id or fd.
*/
if (attr->link_create.cgroup.relative_fd && !(flags & BPF_F_LINK))
return -EINVAL;
link = kzalloc_obj(*link, GFP_USER);
if (!link)
return -ENOMEM;
bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS,
&cgroup_struct_ops_link_ops, NULL,
attr->link_create.attach_type);
err = bpf_link_prime(&link->link, &link_primer);
if (err) {
kfree(link);
return err;
}
cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
if (IS_ERR(cgrp)) {
err = PTR_ERR(cgrp);
goto cleanup;
}
bpf_map_inc(map);
link->map = map;
link->cgroup = cgrp;
init_waitqueue_head(&link->wait_hup);
atype = bpf_struct_ops_map_cgroup_atype(map);
progs = &cgrp->bpf.progs[atype];
cgroup_lock();
if (attr->link_create.cgroup.expected_revision &&
attr->link_create.cgroup.expected_revision != cgrp->bpf.revisions[atype]) {
err = -ESTALE;
goto unlock;
}
if (prog_list_length(progs, NULL) >= BPF_CGROUP_MAX_PROGS) {
err = -E2BIG;
goto unlock;
}
pl = kzalloc_obj(*pl);
if (!pl) {
err = -ENOMEM;
goto unlock;
}
pl->link = link;
pl->flags = pl_flags;
cgrp->bpf.flags[atype] = BPF_F_ALLOW_MULTI;
err = insert_pl_to_hlist(pl, progs, NULL, link,
flags | BPF_F_ALLOW_MULTI, attr->link_create.cgroup.relative_fd);
if (err)
goto unlock;
err = update_effective_progs(cgrp, atype);
if (err) {
hlist_del(&pl->node);
goto unlock;
}
cgrp->bpf.revisions[atype]++;
static_branch_inc(&cgroup_bpf_enabled_key[atype]);
cgroup_unlock();
return bpf_link_settle(&link_primer);
unlock:
cgroup_unlock();
cleanup:
kfree(pl);
if (link->cgroup) {
cgroup_put(link->cgroup);
link->cgroup = NULL;
bpf_map_put(link->map);
link->map = NULL;
}
bpf_link_cleanup(&link_primer);
return err;
}
/* Common helpers for cgroup hooks. */
const struct bpf_func_proto *
cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
kernel/bpf/cnum.c
+36
View File
@@ -2,6 +2,7 @@
/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */
#include <linux/bits.h>
#include <linux/bpf_verifier.h> /* for imod() */
#define T 32
#include "cnum_defs.h"
@@ -118,3 +119,38 @@ struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b)
}
return t;
}
/* Intersect 'in' with the set of integers defined by equation 'base + step * k'. */
struct cnum64 cnum64_intersect_linear(struct cnum64 in, u16 base, u16 step)
{
s64 smin = cnum64_smin(in);
s64 smax = cnum64_smax(in);
s64 lo, hi;
u16 d;
if (step <= 1 || cnum64_is_empty(in) || cnum64_srange_overflow(in))
return in;
/*
* Round smin up to the next value congruent to 'base' modulo 'step',
* i.e. increase smin by d = (base - smin) mod step:
*
* |<---- d ---->|
* |-----------|=============|...
* base+step*k smin base+step*(k+1)
*/
d = imod(base - imod(smin, step), step);
if ((u64)smax - (u64)smin < d)
return CNUM64_EMPTY;
lo = smin + d;
/*
* Round smax down to the previous value congruent to 'base' modulo 'step',
* i.e. decrease smax by d = (smax - base) mod step:
*
* |<--- d --->|
* ...|===========|-------------|
* base+step*k smax base+step*(k+1)
*/
d = imod(imod(smax, step) - base, step);
hi = smax - d;
return cnum64_from_srange(lo, hi);
}
kernel/bpf/cnum_defs.h
+66 -15
View File
@@ -33,7 +33,12 @@ struct cnum_t FN(from_srange)(st min, st max)
return (struct cnum_t){ .base = base, .size = size };
}
/* True if this cnum represents two unsigned ranges. */
/*
* True if this cnum represents two unsigned ranges.
*
* The caller should ensure !is_empty(cnum) holds when calling
* this function.
*/
static inline bool FN(urange_overflow)(struct cnum_t cnum)
{
/* Same as cnum.base + cnum.size > UT_MAX but avoids overflow */
@@ -44,6 +49,9 @@ static inline bool FN(urange_overflow)(struct cnum_t cnum)
* cnum{T}_umin / cnum{T}_umax query an unsigned range represented by this cnum.
* If cnum represents a range crossing the UT_MAX/0 boundary, the unbound range
* [0..UT_MAX] is returned.
*
* The caller should ensure !is_empty(cnum) holds when calling
* cnum{T}_umin / cnum{T}_umax.
*/
ut FN(umin)(struct cnum_t cnum)
{
@@ -57,7 +65,12 @@ ut FN(umax)(struct cnum_t cnum)
}
EXPORT_SYMBOL_GPL(FN(umax));
/* True if this cnum represents two signed ranges. */
/*
* True if this cnum represents two signed ranges.
*
* The caller should ensure !is_empty(cnum) holds when calling
* this function.
*/
static inline bool FN(srange_overflow)(struct cnum_t cnum)
{
return FN(contains)(cnum, (ut)ST_MAX) && FN(contains)(cnum, (ut)ST_MIN);
@@ -67,19 +80,18 @@ static inline bool FN(srange_overflow)(struct cnum_t cnum)
* cnum{T}_smin / cnum{T}_smax query a signed range represented by this cnum.
* If cnum represents a range crossing the ST_MAX/ST_MIN boundary, the unbound range
* [ST_MIN..ST_MAX] is returned.
*
* The caller should ensure !is_empty(cnum) holds when calling
* cnum{T}_smin / cnum{T}_smax.
*/
st FN(smin)(struct cnum_t cnum)
{
return FN(srange_overflow)(cnum)
? ST_MIN
: min((st)cnum.base, (st)(cnum.base + cnum.size));
return FN(srange_overflow)(cnum) ? ST_MIN : (st)cnum.base;
}
st FN(smax)(struct cnum_t cnum)
{
return FN(srange_overflow)(cnum)
? ST_MAX
: max((st)cnum.base, (st)(cnum.base + cnum.size));
return FN(srange_overflow)(cnum) ? ST_MAX : (st)(cnum.base + cnum.size);
}
/*
@@ -181,11 +193,55 @@ void FN(intersect_with_srange)(struct cnum_t *dst, st min, st max)
static inline struct cnum_t FN(normalize)(struct cnum_t cnum)
{
if (cnum.size == UT_MAX && cnum.base != 0 && cnum.base != (ut)ST_MAX)
if (cnum.size == UT_MAX)
cnum.base = 0;
return cnum;
}
/*
* Return a smallest arc containing both 'a' and 'b'.
* Break equal-size ties by choosing the smaller base.
*/
struct cnum_t FN(union)(struct cnum_t a, struct cnum_t b)
{
struct cnum_t b1, ab, ba;
ut end;
if (FN(is_empty)(a))
return b;
if (FN(is_empty)(b))
return a;
/*
* Rotate so that a1.base == 0 and a1.end == a.size.
* Normalize b1 to preserve the full-circle representation.
*/
b1 = FN(normalize)((struct cnum_t){ b.base - a.base, b.size });
end = max(a.size, (ut)(b1.base + b1.size));
if (FN(urange_overflow)(b1)) {
/* a1 reaches b1's main arc: together they cover the circle. */
if (b1.base <= a.size)
return (struct cnum_t){ 0, UT_MAX };
/* Extend b1's tail through a1's end, then rotate back. */
return FN(normalize)((struct cnum_t){ b.base, end - b1.base });
}
/* ab, rotated back, covers both nonwrapping arcs. */
ab = (struct cnum_t){ a.base, end };
if (b1.base <= a.size)
return FN(normalize)(ab);
/* The arcs are disjoint; ba is the other possible covering arc. */
ba = (struct cnum_t){ b.base, a.size - b1.base };
if (ba.size < ab.size ||
(ba.size == ab.size && ba.base < ab.base))
ab = ba;
return FN(normalize)(ab);
}
struct cnum_t FN(add)(struct cnum_t a, struct cnum_t b)
{
if (FN(is_empty)(a) || FN(is_empty)(b))
@@ -210,12 +266,7 @@ bool FN(is_empty)(struct cnum_t cnum)
bool FN(contains)(struct cnum_t cnum, ut v)
{
if (FN(is_empty)(cnum))
return false;
if (FN(urange_overflow)(cnum))
return v >= cnum.base || v <= (ut)cnum.base + cnum.size;
else
return v >= cnum.base && v <= (ut)cnum.base + cnum.size;
return !FN(is_empty)(cnum) && v - cnum.base <= cnum.size;
}
bool FN(is_const)(struct cnum_t cnum)
Loaded 100 of 570 files, more files were not shown because too many files have changed in this diff. Show more