mirror of
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
synced 2026-10-26 10:57:38 +08:00
master
2369
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Linus Torvalds
|
fa899d500a |
Merge tag 'for-7.4/io_uring-20261021' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull io_uring updates from Jens Axboe:
"The bulk of this is a big robustness pass over cancelation, io-wq,
timeouts, and poll. Outside of that:
- Large series from Josef hardening the cancelation, io-wq, timeout,
poll, and wait paths. Fixes for lost wakeups and lost cancels as
io-wq arms poll/timeouts, work stranding on worker exit, CQE32
resize handling, napi entry timeouts, and a pile of smaller
correctness issues across the board
- Rework of how io-wq posts completions, posting from the last
request reference and dropping the request file before the
completion, and running ring release cancelations synchronously -
canceling and waiting for all in-flight requests and dropping
registered files and buffers at release time
- Don't charge the SQ/CQ rings or provided buffer rings to
RLIMIT_MEMLOCK, which the region API conversion had started doing
- Split io_uring_cmd_set_res() out from io_uring_cmd_done() so a
command can set the result from its ->end_io() handler, used by
nvme
- CQE32 ring fixes for the free entry check, aux CQE big_cqe zeroing,
and the dummy skip CQE
- Timeout fixes for sequence handling and multishot updates
- net fixes for MSG_TRUNC buffer overconsumption and keeping recv
buffer select length within MAX_RW_COUNT
- Wake real-time waiters from hard interrupt context
- Don't spin on local work an exiting task can't run, reachable by an
unprivileged user
- Various other fixes and cleanups"
* tag 'for-7.4/io_uring-20261021' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (75 commits)
io_uring: end the drain when the drain request completes
io_uring/futex: return a wait's data to the cache when it fails to arm
io_uring/wait: wake real-time waiters from hard interrupt context
io_uring/waitid: return -EAGAIN for a non-blocking pidfd
io_uring/net: keep recv buffer select length within MAX_RW_COUNT
io_uring/waitid: prevent cancel reference wrap
io_uring/poll: don't loop on HUP with multiple references
io_uring/timeout: don't lose a cancel as io-wq arms a timeout
io_uring/timeout: don't flush with the sequence behind the last flush
io_uring/timeout: don't put a sequence target behind the last flush
io_uring/poll: avoid retry wakeup loss when dropping ownership
io_uring: look up the file before queueing to io-wq
io_uring/timeout: don't end a linked timeout with 1
io_uring/timeout: no ABS update of multishot timeouts
io_uring/net: don't retry recv on uncommitted buffers
io_uring: don't pass restart error codes to userspace
io_uring/openclose: keep the lock in io_pipe_fixed()
io_uring/sqpoll: let the thread sleep on disabled rings
io_uring/kbuf: handle index wrap in last buffer check
io_uring/zcrx: fix buffer offsets in areas above 4GB
...
|
||
|
Linus Torvalds
|
aaf3d47e53 |
Merge tag 'timers-core-2026-10-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
Pull timer core updates from Thomas Gleixner:
"Updates for the core timekeeping and timers subsystem:
- Futher cleanups for the POSIX timer exit/exec related code to
simplify the code flow and to make it more robust against future
changes.
- Prevent tearing of timekeeper::ktime_sec which can be accessed
without the sequence count protection.
The timekeeper is updated by memcpy() with an uneven sequence count
to force readers into the retry path. On 64-bit machine the
ktime_sec member can be accessed atomically without the sequence
count protection. That is only correct when the update side
memcpy() guarantees atomicity at the 64-bit granularity. That's not
guaranteed due to debug options, compilers and instruction
availability. If the memcpy() does not provide the atomicity then
the unprotected readers can observe store tearing.
To prevent this write the ktime_sec member before the memcpy() so
that a non-atomic memcpy() will write the same data. A similar fix
has been done for the xtime_sec member, but the ktime_sec
equivalent was missed.
- Prevent time64_to_tm() day truncation on 32-bit kernels
The function receives a 64-bit seconds value but stores the
quotient of the computation in a long, which truncates the result
for large input values. Cure this by using div_s64_rem() which
stores the quotient in s64 and only leaves the seconds-within-day
reminder in a long, which is guaranteed to fit. This fix comes with
a corresponding kunit test to prevent further mishaps.
- Mark data-racy accesses to the hrtimer_sleeper::task member.
The recent updates to KCSAN flagged these benign races. Annotate
them with new helper functions, convert all existing usage sites
and mark the task member private after that so that future users
which fail to use the helpers can be detected automatically.
- Address similar benign KCSAN flagged issues in the hrtimer, timer
and timer migration code.
- Fixes for a few selftest issues"
* tag 'timers-core-2026-10-20' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip: (26 commits)
selftests/timers: clocksource-switch: Fix unchecked open()/read()
selftests: timers: Measure the CPU timers on the clock they count
selftests: timers: Count what tick is worth in the drift estimate
time/kunit: Add time64_to_tm() case beyond 32-bit day count
time: Prevent time64_to_tm() day truncation on 32-bit
timekeeping: Use READ_ONCE/WRITE_ONCE() for ktime_sec to prevent tearing
timers/migration: Mark racy updates to tmigr_event::ignore field
timers: Mark racy updates to hlist_node::pprev field
hrtimer: Apply READ_ONCE() to lockless base->running loads
hrtimer: Mark the hrtimer_sleeper structure's ->task field __private
hrtimer: Update hrtimer_resolution only if value changes
futex: Use accessor for hrtimer_sleeper ->task field in requeue
rtmutex: Use accessor for hrtimer_sleeper ->task field
net: pktgen: Use accessor for hrtimer_sleeper ->task field
timers: Use accessor for hrtimer_sleeper ->task field in sleep_timeout.c
futex: Use accessor for hrtimer_sleeper ->task field in waitwake.c
io-uring/rw: Use accessor for hrtimer_sleeper ->task field
wait: Use accessor for hrtimer_sleeper ->task field
aio: Use accessor for hrtimer_sleeper ->task field
hrtimer: Mark data-racy accesses to hrtimer_sleeper ->task field
...
|
||
|
Jens Axboe
|
ecb6817b76 |
io_uring: end the drain when the drain request completes
Since requests are drained based on allocated requests, the drain stays
active after the drain request has posted its CQE, until it is freed and
a new request finds the defer list empty. A drain request run by io-wq
is freed only after the worker drops its reference, so a request
submitted after the drain completed can still be deferred behind it,
and get issued much later than the application expects.
A drain request dispatched from the defer list runs alone. Track it,
and end the drain when its CQE is flushed and nothing else is deferred.
Have io-wq complete drain requests through task_work, so that this
happens under ->uring_lock before the CQE is visible. Linked drain
requests keep waiting until the link is freed.
Reported-by: Wei Gao <wegao@suse.com>
Link: https://lore.kernel.org/io-uring/20261018070154.16944-1-wegao@suse.com/
Fixes:
|
||
|
Austin Schuh
|
02caa59221 |
io_uring/futex: return a wait's data to the cache when it fails to arm
io_futex_wait() takes its io_futex_data from ctx->futex_cache, and a
completed wait gives it back in io_futex_complete(). A wait that fails
to arm kfree()s it instead.
The usual failure is -EAGAIN from futex_wait_setup(): the futex word no
longer holds the expected value. That is routine for a waiter that
re-arms with the value it last read, since the word changes whenever
there is something to wake for. Each such failure takes an object out
of the cache for good, so the cache drains and from then on waits
allocate.
The cache exists so that arming a wait doesn't have to go to the slab
allocator, and that matters most to the waiters that re-arm the most:
event loops, often real-time ones. A kmalloc() or kfree() that misses
the per-CPU slab takes the node's list_lock, shared with every other
user of that kmalloc size on the system, and may go on to the page
allocator.
On PREEMPT_RT list_lock is a sleeping lock. A real-time task that
finds it held waits for the holder, which may be any task on any CPU,
to get to run and release it. Priority inheritance boosts the holder
only up to the waiter's priority, so the waiter still sits behind
whatever outranks that on the holder's CPU, such as threaded interrupt
handlers. Re-arming a futex wait should not depend on any of that.
Put the data back in the cache on that path too, while the submit lock
still protects it.
In an event loop re-arming one IORING_OP_FUTEX_WAIT per message under
load, 0.39 arms per message failed with -EAGAIN, and io_futex_wait() did
0.39 kmalloc()s and kfree()s per message. With this change, 100,000
waits that all fail with -EAGAIN do one kmalloc() in total.
Fixes:
|
||
|
Linus Torvalds
|
a1d4585be1 |
Merge tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull file table updates from Christian Brauner: "This annotates struct fdtable's fd array with __counted_by_ptr so KASAN and the fortified functions can bounds check it against max_fds and adds a KUnit suite for fdtable. A follow-up uses __struct_size() in the test since sparse doesn't know __builtin_dynamic_object_size(). FD_PREPARE() is simplified. It was built as an ACQUIRE-style guard on infrastructure that was never needed. The ACQUIRE machinery and CLASS_INIT() go away, the variable is declared with __cleanup() directly, struct fd_prepare loses its err field since the fd either holds the descriptor or the error, and callers get a const pointer to the guard behind a UNIQUE_ID() name so they can't shadow it. Code generation is identical" * tag 'vfs-7.4-rc1.file' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: file: make struct fd_prepare const and kill its err field cleanup: remove CLASS_INIT() file: declare the FD_PREPARE() variable with __cleanup() directly file: simplify FD_PREPARE() vfs: use __struct_size() in the fdtable KUnit test vfs: Add KUnit tests for fdtable vfs: Annotate struct fdtable's fd field with __counted_by_ptr |
||
|
Linus Torvalds
|
018cf9bf78 |
Merge tag 'kernel-7.4-rc1.signal' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs
Pull signal handling updates from Christian Brauner:
"This stops TIF_NOTIFY_SIGNAL from interrupting work that can't be
restarted and fixes a bunch of coredump and signal bugs.
TIF_NOTIFY_SIGNAL kicks a task out of an uninterruptible sleep so it
returns to userspace, runs its task work and goes back to sleep. That
breaks coredumps: dump_interrupted() only lets fatal signals interrupt
a dump and writes to a regular file ignore TIF_NOTIFY_SIGNAL. But the
pipe and the socket paths check signal_pending() Adeep down in code
that is shared with regular syscalls and bail. So a crashing process
with io_uring requests in flight loses most of its coredump.
In the selftest an unfixed kernel delivers 357324 of at least 5111808
bytes. And io_uring isn't the only source. klp_send_signals(),
bpf_task_work_schedule_signal(), landlock's tsync, kthread_stop() and
the printk kunit test raise the bit as well, some of them from irq
context against any task. So this can't be fixed per subsystem and the
state has to be per-task and ambient.
Add PF_NO_NOTIFY_SIGNAL and no_notify_signal_save()/restore() helpers
modelled on memalloc_nofs_save(). Inside such a section
signal_pending() ignores TIF_NOTIFY_SIGNAL. Coredumps, the smb send
path, which hand-rolled its own version of this, and the pid namespace
reaper, which busy-looped when the bit kept getting raised, are
converted.
The second part is a set of fixes that fell out of looking at the
coredump code with kres and modelling it in TLA+:
- coredump_finish() woke the parked threads without holding rcu. A
parked thread can exit on a spurious wakeup and be freed before
wake_up_process() takes its pi_lock.
- Only SIGKILL and the freezers interrupt a dump now and the three
freezers agree. Power management and cgroup v1 freezing aborted a
dump while a cgroup v2 freeze either waited for the dump or cut it
short depending on where the core went.
- io-wq set its exit bit without a barrier against the task work that
creates workers. On weakly ordered architectures exit could hang on
worker_done.
- retarget_shared_pending() slingshotted a shared signal to the dumper
when a zapped sibling restored its signal mask. That set
TIF_SIGPENDING on the dumper and truncated the core.
- A tracer could clear the signal mask of a user worker with
PTRACE_SETSIGMASK and inject a coredump signal. An io-wq worker of
an SQPOLL ring dumping core deadlocks the whole group.
PTRACE_SETSIGMASK is refused for user workers now and
__set_task_blocked() enforces the fork-time mask.
- exec cancels io_uring requests before de_thread() so a worker
creation queued as task work can't add a thread to a thread group
that de_thread() just took down.
- No io threads are created from a PF_SIGNALED or PF_POSTCOREDUMP
creator anymore. Such a worker was invisible to zap_process() but
still decremented threads_remaining on exit so the dumper could
leave coredump_wait_inactive() early.
Also included are a comment that do_each_pid_task() needs
tasklist_lock, a READ_ONCE() in pid_alive() and a regression test for
fork() into a pid namespace whose ancestor lost its init"
* tag 'kernel-7.4-rc1.signal' of git://git.kernel.org/pub/scm/linux/kernel/git/vfs/vfs: (21 commits)
pid: use READ_ONCE() in pid_alive()
selftests: pid_namespace: test fork with dead ancestor
pid: document that do_each_pid_task() needs tasklist_lock
signal: enforce the user worker signal mask in __set_task_blocked()
fork: use SIG_KERNEL_ONLY_MASK for the user worker signal mask
fork: don't create io threads once PF_POSTCOREDUMP is set
fork: move the coredump and exec checks into create_io_thread()
exec: cancel io_uring requests before de_thread()
selftests/coredump: expect PTRACE_SETSIGMASK to be refused on a user worker
selftests/coredump: test a user worker as the coredumping thread
ptrace: refuse to change the signal mask of a user worker
selftests/coredump: test shared signal retargeting during a dump
signal: only SIGKILL and the freezers interrupt a coredumping task
signal: don't retarget shared signals in a dying thread group
io-wq: order the exit bit against worker creation task work
coredump: hold RCU while releasing parked threads
pid_namespace: prevent TIF_NOTIFY_SIGNAL from interrupting the reaper
smb: prevent TIF_NOTIFY_SIGNAL from interrupting
selftests/coredump: test that TIF_NOTIFY_SIGNAL doesn't truncate a coredump
coredump: prevent TIF_NOTIFY_SIGNAL from interrupting coredumps
...
|
||
|
Austin Schuh
|
5b01d5955a |
io_uring/wait: wake real-time waiters from hard interrupt context
Since commit |
||
|
Nickolai Zeldovich
|
c82621eb2c |
io_uring/waitid: return -EAGAIN for a non-blocking pidfd
IORING_OP_WAITID on a pidfd opened with PIDFD_NONBLOCK completes at once with res 0 and an all-zero siginfo when the child has not changed state, although WNOHANG was not requested. waitid(2) with the same arguments fails with -EAGAIN. An application that submits the same request it would pass to waitid(2) and treats res 0 as "the child changed state" sees si_signo 0 and si_pid 0 instead. kernel_waitid_prepare() turns an O_NONBLOCK pidfd into WNOHANG in wo->wo_flags. kernel_waitid() then maps the resulting 0 return to -EAGAIN when the caller did not ask for WNOHANG itself, as added in commit |
||
|
Tamaki Yanagawa
|
96262cfab5 |
io_uring/net: keep recv buffer select length within MAX_RW_COUNT
Since commit |
||
|
Junyeong Lee
|
98787d3fa4 |
io_uring/waitid: prevent cancel reference wrap
io_waitid packs the cancellation state into bit 31 of ->refs and uses
bits 30:0 as a 31-bit reference counter:
#define IO_WAITID_CANCEL_FLAG BIT(31)
#define IO_WAITID_REF_MASK GENMASK(30, 0)
__io_waitid_cancel() marks a request canceled, then increments the low
bits of ->refs to claim completion ownership. When another path owns the
request, the increment remains even though cancellation returns false.
The request remains discoverable by cancellation, so repeated failed
attempts can wrap the 31-bit reference field to zero. A later
cancellation can claim ownership and complete the request a second time.
Use atomic_fetch_or() when setting the cancel flag and inspect the old
flag value. The first cancel attempt continues to the existing ownership
increment, preserving its wake and issuer handoff semantics. Later
attempts see the cancel flag and return without incrementing the
reference field.
This makes repeated failed cancellation attempts idempotent while
leaving the first-cancel state transitions unchanged.
Fixes:
|
||
|
Jens Axboe
|
67bb1ad9a5 |
io_uring/poll: don't loop on HUP with multiple references
io_poll_check_events() drops down to one reference when a multishot
request sees HUP with more than one, so that the loop goes around once
more. It did so by subtracting one less than it had. With the cmpxchg
that only takes the local count down: the exchange fails, the next pass
reads the unchanged count, and goes through the same motions again. The
loop then only ends once io_poll_issue() stops the request.
Drop the extra references from the counter instead, and loop once more
without trying the exchange. The next pass sees one reference, like it
did before, and ends unless a wake up came in.
Fixes:
|
||
|
Jens Axboe
|
89f193dc3e |
io_uring/timeout: don't lose a cancel as io-wq arms a timeout
An IORING_OP_TIMEOUT run by io-wq, because of IOSQE_ASYNC, has the same
problem as poll, futex and waitid. io_wq_submit_work() looks at
IO_WQ_WORK_CANCEL once, before it issues the request. A cancel in between
flags the work, gets -EALREADY, and searches ->timeout_list in vain,
while the worker then adds the request and starts the timer:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
= -EALREADY
io_timeout_cancel() = -ENOENT
io_timeout()
adds the request to ->timeout_list
hrtimer_start()
= IOU_ISSUE_SKIP_COMPLETE
Nobody looks at the flag again, and the timeout runs its course. At ring
exit, io_uring_try_cancel_requests() cancels io-wq work and then kills
the timeouts the same way, so an exiting task can get stuck for the full
duration of such a timeout.
Look at the flag under ->timeout_lock, which the search takes as well,
and fail the request if it is set. If the search came first, the flag is
visible. If it comes later, it finds the request on the list.
Fixes:
|
||
|
Jens Axboe
|
85f85d573f |
io_uring/timeout: don't flush with the sequence behind the last flush
io_flush_timeouts() measures the events since the last flush as the
distance of the current sequence from ->cq_last_tm_flush. A fired
timeout bumps ->cq_timeouts before its CQE is posted, so until it is
the sequence can be behind the last flush, and the distance wraps:
last flush: seq 100, cq_last_tm_flush = 100
Y: IORING_OP_TIMEOUT, off 50, target_seq 150
pure timeout A fires, cq_timeouts = 1
task_work posts A's CQE, cached_cq_tail = 101
pure timeout B fires before the flush, cq_timeouts = 2
io_flush_timeouts(): seq = 99, events_got = 0xffffffff
Y, and every sequence timeout behind it, is ended with -ETIME although
no event happened. Two timeouts expiring close together is all it takes,
a completion with IOSQE_CQE_SKIP_SUCCESS after a single one does it too.
Nothing but those in flight CQEs separates the two, so count from the
last flush when the sequence is behind it, as io_timeout() does.
Fixes:
|
||
|
Jens Axboe
|
fdfce905a5 |
io_uring/timeout: don't put a sequence target behind the last flush
io_timeout_fn() and io_kill_timeout() bump ->cq_timeouts when a timeout
fires, but its CQE is only posted later from task_work. Until then the
sequence io_timeout() computes is one below the last flush per fired
timeout. Now that ->cq_last_tm_flush is kept while sequence timeouts are
pending, a timeout armed in that window with a small offset gets a target
behind the base. Its distance from the base wraps, and it's never
completed by events, only by its timer. With DEFER_TASKRUN that window
is as wide as the application makes it, as submission runs before the
local task_work.
Nothing but those in flight CQEs separates the two, so use the last
flush as the base for the target when the sequence is behind it.
Fixes:
|
||
|
Juanlu Herrero
|
937d38dcac |
io_uring/poll: avoid retry wakeup loss when dropping ownership
If a wakeup sets IO_POLL_RETRY_FLAG between the owning task consuming
the existing instance of IO_POLL_RETRY_FLAG and the loop check
of io_poll_check_events(), then we stall on processing the request
until an unrelated wakeup comes along.
Using atomic_cmpxchg instead of atomic_sub_return to guarantee both the
count v and poll_refs are 0 and no wake-ups are missed.
Cc: stable@vger.kernel.org
Fixes:
|
||
|
Josef Bacik
|
98bc044d5a |
io_uring: look up the file before queueing to io-wq
io_prep_async_work() decides by the file of a request what kind of work
it is for io-wq. I/O on regular files and block devices is bounded work,
limited to min(SQ entries, 4 * CPUs) workers, for anything else that may
block indefinitely the limit is RLIMIT_NPROC. Writes to a regular file
are hashed by the inode, so that they are run one after the other instead
of having a bunch of workers fight for the inode lock.
Since the file is only assigned when a request is issued, a request that
is queued without an attempt to issue it, REQ_F_FORCE_ASYNC, doesn't have
one at that point. io_prep_async_work() takes that for "not a regular
file or block device":
- Reads and writes end up as unbound work. 32 reads with IOSQE_ASYNC from
a slow block device on a ring with 4 SQ entries get 32 workers, not 4.
- Writes aren't hashed. Four buffered writes to one file get four
workers, three of which do nothing but wait for the lock.
The late assignment is for the benefit of links, where a request may use
a fixed file that one of its predecessors installs. By the time a request
gets to io_queue_iowq() these are done, it is as good as issued. Assign
the file there.
io_assign_file() moves up and is forced inline. With a third caller the
compiler may stop inlining it, and every request that is issued would
pay for a call.
IORING_OP_SPLICE is left as it is, unbound and not hashed. The file of
the request is where it writes to, which says nothing about how long it
takes: a splice from an empty pipe or a socket into a regular file waits
for its input. As bounded work, four of these on a ring with 4 SQ entries
keep all other bounded work of the task from running, and as hashed work
one of them blocks all writes to that file.
REQ_F_FORCE_ASYNC is not only set for IOSQE_ASYNC. What changes, and
brings back what was the case before the file assignment was deferred:
- Requests with IOSQE_ASYNC or IOSQE_IO_DRAIN, and all requests that are
submitted while a drain is active: reads and writes on regular files
and block devices count against the limit for bounded workers, including
the one set with IORING_REGISTER_IOWQ_MAX_WORKERS, so fewer of them run
at the same time.
- Of these, writes to one regular file are run one at a time, unless they
are O_DIRECT and the filesystem has FOP_DIO_PARALLEL_WRITE.
- IORING_OP_FALLOCATE and IORING_OP_FTRUNCATE, which always go to io-wq,
are run one at a time per file, and in turn with these writes.
- The descriptor or the fixed file slot is looked up when the request is
queued, not when a worker gets around to it, so closing or replacing it
in between no longer has an effect on the request.
Fixes:
|
||
|
Josef Bacik
|
c74193430f |
io_uring/timeout: don't end a linked timeout with 1
A linked timeout that fires cancels the request it is linked to, and completes with -ETIME if that worked out: ret = io_try_cancel(req->tctx, &cd, 0); ... io_req_set_res(req, ret ?: -ETIME, 0); io_try_cancel() returns 0 or an error, except for requests that are parked on ->futex_list or ->waitid_list. io_futex_cancel() and io_waitid_cancel() return the number of requests that they have cancelled, and io_try_cancel() hands it on. So the linked timeout of an IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAITV or IORING_OP_WAITID that it has cancelled completes with 1, each time. The waitid test of liburing checks for that 1, and has to be changed to expect -ETIME, which is what the linked timeout of any other request completes with. Fixes: |
||
|
Josef Bacik
|
b088ae34d4 |
io_uring/timeout: no ABS update of multishot timeouts
IORING_TIMEOUT_MULTISHOT can't be combined with IORING_TIMEOUT_ABS, prep
rejects that, as the request is armed again with the same value each time
it fires. IORING_TIMEOUT_UPDATE doesn't check and takes an absolute time
for a multishot timeout. The first expiry is as asked for, but
io_timeout_complete() then arms the timer with ->mode of the original
request, i.e. it uses the absolute time as the interval. With
CLOCK_MONOTONIC the next expiry is an uptime away, with CLOCK_REALTIME
more than fifty years.
Fail such an update with -EINVAL, and do so before the timeout is
touched.
Fixes:
|
||
|
Josef Bacik
|
8444de681a |
io_uring/net: don't retry recv on uncommitted buffers
Commit |
||
|
Josef Bacik
|
e1a64bc648 |
io_uring: don't pass restart error codes to userspace
-ERESTARTSYS and friends are instructions for the system call exit code,
which either restarts the call or turns them into -EINTR. They are not
supposed to be seen by applications. A request isn't a system call,
nobody looks at what it returns before that is put into a CQE.
The read/write and the networking opcodes convert these codes to -EINTR
by hand. The others don't. An io-wq worker that is blocked in the open of
a FIFO, a tty or some other device, or in a splice from an empty pipe,
and that gets interrupted because the request is cancelled, completes it
with -512:
IORING_OP_OPENAT, IOSQE_ASYNC // FIFO without a writer
IORING_OP_ASYNC_CANCEL // -EALREADY, worker is signalled
CQE of the open: res == -512
There are more candidates, like whatever a file system or driver returns
from ->flush(), ->fsync(), ->fallocate() or ->uring_cmd(), or breaking a
lease or delegation.
Take care of it in io_issue_sqe(), which all of them go through. The
helper is moved from rw.c as is. The cost is a compare and a branch that
is not taken per request.
Fixes:
|
||
|
Josef Bacik
|
a3d7ee1128 |
io_uring/openclose: keep the lock in io_pipe_fixed()
io_pipe_fixed() installs both ends of the pipe in the fixed file table,
drops ->uring_lock, and copies the slot numbers to the application. If
that faults, it takes the lock again and removes whatever is in these
slots.
If the request is run by io-wq the lock really is dropped in between, and
the slots are visible to everybody else. By the time the error path gets
to them, they may have been closed and reused, by IORING_OP_CLOSE and a
direct open or accept, or by IORING_REGISTER_FILES_UPDATE. Then a file is
removed from the table that has nothing to do with the failed request.
Do the copy with the lock held. Faulting in user memory under
->uring_lock is fine, the registration paths do it all the time.
Fixes:
|
||
|
Josef Bacik
|
d492dcbb3b |
io_uring/sqpoll: let the thread sleep on disabled rings
__io_sq_thread() doesn't submit anything as long as the ring has
IORING_SETUP_R_DISABLED set. The check that keeps the SQPOLL thread from
going to sleep while the SQ ring holds entries doesn't know about that.
If an application fills in SQEs first and enables the ring later, the
thread finds that there is work each time it is about to sleep, goes
around, doesn't submit it, and so on. It uses up a CPU until the ring
gets enabled.
Entries in the SQ ring of a disabled ring are no reason to stay awake.
io_register_enable_rings() clears the flag and then wakes the thread, and
the flag is looked at after prepare_to_wait(), so the thread can't miss
that.
Fixes:
|
||
|
Josef Bacik
|
6e773b5161 |
io_uring/kbuf: handle index wrap in last buffer check
io_ring_buffer_select() flags the request with REQ_F_BL_EMPTY if it takes
the last buffer of the ring. Head and tail are 16-bit counters, but
"head + 1" is computed as an int. If head is 65535 and tail has wrapped
to 0, that compares 65536 to 0 and the flag isn't set.
A bundle receive relies on the flag to tell that there is no point in
trying to get more data. Without it, it goes for another round, finds no
buffer, and the request, which has received data, is marked as failed on
account of the -ENOBUFS. That severs a link, and ends a multishot.
Fixes:
|
||
|
Josef Bacik
|
ab4536bf2e |
io_uring/zcrx: fix buffer offsets in areas above 4GB
io_zcrx_queue_cqe() computes the offset of the buffer in its area from
net_iov_idx(), which is an unsigned int, so the shift is done in 32 bits
and only the result is widened to the u64 that goes into the CQE. An area
can be up to 1TB in size. For every niov beyond the first 4GB the CQE
points the application at a buffer 4GB, or a multiple of that, below the
one which has the data. It reads the wrong data, and then returns a
buffer it doesn't own via the refill queue while the real one is never
given back.
Fixes:
|
||
|
Josef Bacik
|
21ed976c12 |
io_uring/napi: don't busy poll past an expired timeout
__io_napi_busy_loop() caps the busy poll time to what is left until the
timeout of the wait. The difference is negative if the timeout has passed
already, which is what an application gets if it uses
IORING_ENTER_ABS_TIMER with a time that isn't in the future any more. The
min_t(u64, ...) turns that into a huge value, so the cap has no effect
and the wait, which should return right away, spins for the full busy
poll time first.
Fixes:
|
||
|
Josef Bacik
|
9c15164878 |
io_uring/napi: make stale entries time out after 60s
With dynamic tracking a NAPI id that hasn't been seen for a minute is
meant to be dropped from the list the ring busy polls. The timeout is
added to jiffies, but it is defined as 60 * SEC_CONVERSION, and
SEC_CONVERSION isn't the number of jiffies in a second. It is the
internal factor of __timespec64_to_jiffies(), HZ scaled up by
SEC_JIFFIE_SC bits. For HZ=1000 that is 1000 << 22, which puts the
timeout at about eight years. On 32-bit the multiplication overflows.
So entries are never removed, and every id that a socket of the ring
ever had keeps being polled on each wait.
Fixes:
|
||
|
Josef Bacik
|
83cce67298 |
io_uring/wait: no early -ETIME if min_wait finds work
When the min wait time is over and there is work to run, which for
DEFER_TASKRUN includes task_work that was queued lazily and didn't wake
the waiter, io_cqring_min_timer_wakeup() ends the wait via
io_cqring_timer_wakeup(). That flags it as timed out. io_cqring_wait()
runs the work, leaves the loop, and returns 0 if the CQ ring holds
something.
Running task_work doesn't have to result in a CQE, though. A poll wake up
can turn out to be spurious, for instance if somebody else has read the
data by the time the request is retried. The application then gets -ETIME
once the min wait time has passed, no matter how far away the timeout it
asked for is.
Wake the task without declaring a timeout. From now on a single CQE
satisfies the wait, like in the case where the timer is switched over to
the normal timeout. If running the work doesn't post one, the loop in
io_cqring_wait() goes back to sleep with the normal timeout.
Fixes:
|
||
|
Josef Bacik
|
dad82eb180 |
io_uring/timeout: use the right clock for ABS updates
io_parse_user_time() converts an absolute time from the time namespace of
the caller to the host's view, for the clock given by the flags. The
flags of IORING_TIMEOUT_UPDATE and IORING_LINK_TIMEOUT_UPDATE can't name
a clock, the updated timer keeps running on the clock the timeout was
set up with. So the time is always converted as if it was
CLOCK_MONOTONIC.
In a time namespace with a monotonic offset, that moves the new expiry of
a CLOCK_REALTIME timeout by that offset, although the clock isn't
affected by time namespaces at all, and the one of a CLOCK_BOOTTIME
timeout by the difference between the two offsets.
Do the conversion when the timeout has been looked up and its clock is
known. The update is run by the submitter, its SQPOLL thread or one of
its io-wq workers, all of which share the time namespace.
Fixes:
|
||
|
Josef Bacik
|
3dc2a32cd3 |
io_uring/timeout: keep repeat count on multishot update
IORING_TIMEOUT_UPDATE clears ->off of the timeout it updates, to turn a
timeout that waits for a number of completions into a pure one. For a
multishot timeout ->off isn't a sequence though, it tells whether the
number of repeats is limited: io_timeout_finish() only looks at
->repeats if ->off is set.
A multishot timeout that was set up to fire N times hence fires forever
once its interval has been updated. Leave ->off alone for those, they are
never treated as sequence timeouts anyway.
Fixes:
|
||
|
Josef Bacik
|
c6c0b3eba8 |
io_uring/timeout: don't skip over reached sequences
io_flush_timeouts() copes with the wrap of the CQ sequence by looking at
the distance from ->cq_last_tm_flush. That relies on the target of each
pending timeout being at or ahead of the last flush.
io_timeout() breaks that. It sets ->cq_last_tm_flush to the current
sequence, on the assumption that nothing has been posted since the last
flush. That isn't true. A multishot request posts its CQEs right when it
is issued or retried, but they are committed, and the timeouts flushed,
at the end of the submission or task_work run:
X: IORING_OP_TIMEOUT, off == 1 target_seq == N + 1
io_submit_sqes()
multishot recv, posts two CQEs sequence == N + 2
Y: IORING_OP_TIMEOUT, off == 100 ->cq_last_tm_flush = N + 2
io_flush_timeouts()
X: events_needed = N + 1 - (N + 2) // 0xffffffff
events_got = 0 // not yet
X isn't completed although the completions it was waiting for have
happened, only its timer can end it now. And as the list is sorted and
the walk ends at the first timeout that isn't due, it gets in the way of
the ones behind it. The insertion sort has the same problem, it measures
the distance of the pending targets from the current sequence.
->cq_last_tm_flush only needs to be brought up to date if no timeout is
waiting for a sequence, as io_flush_timeouts() isn't called before the
first one is set up. Otherwise it is current enough. Timeouts with a
sequence come first in the list, so looking at the first entry will do.
Use the last flush as the base for the insertion sort as well.
Fixes:
|
||
|
Josef Bacik
|
e4507a62c2 |
io_uring/fdinfo: fix CQE dump of CQE32 rings
Since the CQ ring dump was changed to deal with IORING_SETUP_CQE_MIXED, a
32b CQE is taken to occupy two entries of the ring. That is how it works
for mixed rings. With IORING_SETUP_CQE32 each entry has twice the size:
head and tail count 32b CQEs, and entry n is found at cqes[n << 1].
For such rings the dump reads cqes[n], and skips every other entry. It
shows half of the pending CQEs at best, and what it shows is taken from
the wrong place unless the head happens to be at the start of the ring.
The extra data of the last entry isn't shown at all, as the check that
keeps the dump of a mixed ring from reading beyond the array is applied to
these rings as well.
Fixes:
|
||
|
Josef Bacik
|
0feb5bcd2d |
io_uring/register: keep 128b SQEs intact on resize
On a ring with IORING_SETUP_SQE_MIXED a 128b SQE takes two entries that
have to be contiguous in memory, io_init_req() fails it with -EINVAL if
it starts on the last entry of the SQ ring. The entry a pending SQE ends
up in after IORING_REGISTER_RESIZE_RINGS depends on the size of the new
ring, and nothing takes care of that. With four entries instead of eight,
a 128b SQE which was fine in entries 3 and 4 is split up between entry 3
and entry 0. The resize succeeds, and the request fails later on.
If the ring has an SQ array, the second half is also taken from the
wrong place unless the array happens to map the following index to the
following entry. The submission side doesn't look at the array for the
second half.
Do what is done for 32b CQEs on IORING_SETUP_CQE_MIXED rings: fail the
resize with -EOVERFLOW if a 128b SQE would wrap. And copy the second half
from where it is.
Fixes:
|
||
|
Josef Bacik
|
e8dee8d549 |
io_uring/register: handle index wrap when resizing
The ring heads and tails are free running 32-bit counters.
io_register_resize_rings() takes that into account when it checks whether
the pending entries fit into the new rings, but then copies them with
for (i = old_head; i < tail; i++)
If the tail has wrapped and the head hasn't, that copies nothing. The new
rings get the old head and tail, so the application finds whatever is in
the freshly allocated memory, zeroes, in place of the CQEs it hasn't
reaped yet, or the kernel in place of the SQEs that have not been
submitted yet. A ring that completes a million requests per second gets
there after little more than an hour.
Loop until the tail is reached, the number of iterations has been bounded
by the size of the new ring at that point.
Fixes:
|
||
|
Josef Bacik
|
0edf1930b8 |
io_uring/register: don't wrap 32b CQEs when resizing
On an IORING_SETUP_CQE_MIXED ring, a 32b CQE uses two contiguous CQ
entries and is never allowed to wrap the end of the ring. When posting,
io_cqe_cache_refill() inserts a dummy IORING_CQE_F_SKIP entry if only
one entry is left before the ring wraps, and the application relies on
that to read a 32b CQE as one contiguous 32 byte slot.
io_register_resize_rings() copies pending CQEs to the new ring one 16b
entry at a time, placing entry i at i & (new_entries - 1). When the CQ
ring is shrunk, a pending 32b CQE that was contiguous in the old ring
can land with its first half on the last entry of the new ring and its
second half on entry 0. For example, with 8 old entries, 4 new entries
and a 32b CQE at index 3, the halves end up at entries 3 and 0. The
application then reads the extra data of that CQE from past the end of
the CQ array, and the CQE content it sees is wrong.
This was found by code inspection and has not been reproduced.
Fix by failing the resize with -EOVERFLOW if a pending 32b CQE would
wrap the new ring, the same way the resize fails when the pending CQEs
don't fit.
Fixes:
|
||
|
Josef Bacik
|
79956bb39b |
io_uring: don't split 128b SQEs at the SQPOLL cap
If an SQPOLL thread serves more than one ring, __io_sq_thread() submits
no more than IORING_SQPOLL_CAP_ENTRIES_VALUE entries per ring and round,
for fairness. On a ring with IORING_SETUP_SQE_MIXED the cap may fall
between the two entries of a 128b SQE. io_init_req() finds that just one
entry is left of what it was asked to submit and fails the request with
-EINVAL, although the second entry is sitting in the SQ ring.
In the next round the second entry is taken for an SQE of its own, so on
top of the failed request the application gets whatever its payload
amounts to when it is read as an SQE.
The application has done nothing wrong, and it can't know where the cap
is going to be applied. For SQPOLL, let the request have its second entry
if the SQ ring holds it, in excess of the number of entries that were to
be submitted.
Fixes:
|
||
|
Josef Bacik
|
78f9c9cffe |
io_uring: drop the CQE cache if a refill fails
->cqe_cached and ->cqe_sentinel delimit the CQ entries which can be
handed out without looking at the ring. io_get_cqe_overflow() only calls
io_cqe_cache_refill() if that range is too small. Without
IORING_SETUP_CQE_MIXED too small means empty. With it, a 32b CQE needs
two entries, and one may be left. If the refill then fails, the entry
stays in the cache, and the next 16b CQE gets it without any checks.
That CQE overtakes the 32b one, which went to the overflow list. The test
for IO_CHECK_CQ_OVERFLOW_BIT, which is there to prevent that, is part of
the refill:
CQ ring with 4 entries, head == 0
16b CQE, 32b CQE tail == 3, one entry cached
32b CQE no room, the CQE overflows
16b CQE written to entry 3, ahead of the 32b CQE
Forget about the rest of the cache when it is refilled.
Fixes:
|
||
|
Josef Bacik
|
3eff68bce2 |
io_uring/futex: allocate private hash before waiting
The futex code relies on there being no private futex waiters by the time
the first private hash of a mm is installed: __futex_pivot_hash() moves
the waiters from the previous private hash to the new one, but it does not
look for them in the global hash which is used as long as there is no
private hash. That is why every clone(CLONE_VM) allocates it up front: a
task which is alone in its mm cannot block in futex_wait() and create a
thread or invoke prctl(PR_FUTEX_HASH) at the same time.
IORING_OP_FUTEX_WAIT and IORING_OP_FUTEX_WAITV do not block the submitter
though:
io_uring_enter()
io_futex_wait()
futex_wait_setup() // no private hash: queued in the global hash
pthread_create()
futex_hash_allocate_default()
__futex_pivot_hash() // first private hash, nothing is moved
futex(FUTEX_WAKE_PRIVATE)
futex_hash() // private hash: no waiter
The wakeup is lost and the request only completes when it is cancelled.
prctl(PR_FUTEX_HASH, PR_FUTEX_HASH_SET_SLOTS, n) instead of
pthread_create() has the same effect.
Allocate the default private hash when a wait request on a private futex
is prepared, which happens in the context of the submitter. This is a RCU
protected pointer check once the hash exists. A failure to allocate it
fails the request with -ENOMEM.
Fixes:
|
||
|
Josef Bacik
|
f2dbe87968 |
io_uring/poll: drop stale double poll entry on rearm
A file that uses two wait queues for poll, e.g. a tty, gets a second io_poll entry, which IORING_OP_POLL_ADD keeps in ->async_data until the request is freed. io_poll_add() can run more than once for a request though: IORING_OP_POLL_REMOVE with one of the update flags disarms and rearms it, and io_poll_task_func() reissues it if a wake up turns out to be spurious. The second poll_wait() of the new round then finds *poll_ptr set, with ->head cleared by the disarm, and __io_queue_proc() fails what it takes for a third wait queue with -EINVAL. For an update that means that the poll request is completed with -ECANCELED, while the update itself reports success. For a spurious wake up, e.g. tcsetattr() on the polled tty, the poll request fails with -EINVAL. Free the old entry and clear the poll flags before arming, which is what io_arm_apoll() does for internal polling. All entries have been removed from their wait queues at that point. Fixes: |
||
|
Josef Bacik
|
356f92f241 |
io_uring/wait: look at all CQ waiters on wake up
io_wake_function() returns -1 if the waiter it is called for doesn't have
enough events yet, which makes __wake_up_common() stop walking the wait
queue. That assumes that the waiters behind it can't be satisfied either.
Each waiter has its own target though, computed from the min_complete it
passed in:
thread A: io_uring_enter(GETEVENTS, min_complete = 10) // queued first
thread B: io_uring_enter(GETEVENTS, min_complete = 1)
one CQE is posted: A isn't woken, and B isn't even considered
B keeps sleeping with the event it asked for sitting in the CQ ring, until
A's demand is met or a timeout or signal ends the wait.
Return 0 to carry on with the next waiter. Only waiters whose condition
is met are woken, like before.
Fixes:
|
||
|
Josef Bacik
|
b88902d874 |
io_uring: wake ring pollers for local task_work
With IORING_SETUP_DEFER_TASKRUN completions are queued as local work, and
only become CQEs when the submitter enters the kernel to wait for events.
For an application that finds out via poll() or epoll on the ring fd when
it is time to do that, io_uring_poll() reports EPOLLIN if local work is
pending.
That only helps if the work is there at the time of the call. When
DEFER_TASKRUN was added, io_req_local_work_add() did io_cqring_wake(),
which took care of tasks sleeping in poll() as well. Then polling got a
wait queue of its own, ->poll_wq, and the wake up in
io_req_local_work_add() was replaced by waking the submitter directly.
Since then ->poll_wq is only woken when CQEs are committed, which for
such a ring is done by the very task that is waiting to be told to:
submitter completion
poll(ring_fd)
io_uring_poll() // nothing yet
schedule()
io_req_local_work_add()
// ->cq_wait_nr: nobody waits
// sleeps on
Wake ->poll_wq when the first entry is put on the work list. Like for the
eventfd, that must not be done right away if the work is queued from a
wake up handler, as the chain of wake ups could get back to the wait
queue whose lock is held. Use an RCU callback in that case, with a
reference to the ring.
Fixes:
|
||
|
Josef Bacik
|
fe6ca13c98 |
io_uring/cancel: let CANCEL_ALL find parked requests
A cancellation with IORING_ASYNC_CANCEL_ALL or _ANY goes on until it finds
nothing more. A request may stay where it was found for a while after it
has been cancelled, so each one that matches is stamped with the sequence
number of the cancellation, and is skipped from then on.
io_cancel_cb() does so for io-wq work as well. There is no need to:
pending work is taken off the list, and the workers are looked at once per
call. And it does harm. A request that is run by io-wq may be parked
while it still is the current work of the worker: poll gets armed for it,
or it is put on ->futex_list or ->waitid_list.
io-wq worker io_try_cancel()
io_wq_submit_work()
io_arm_poll_handler()
io_poll_add_hash()
io_async_cancel_one()
io_cancel_cb()
stamps the request
sets IO_WQ_WORK_CANCEL
= -EALREADY
io_poll_cancel()
skips the request
= -ENOENT
io_assign_current_work(NULL)
The flag comes too late for the worker to see it, and the search that
would have found the request takes it for done. The cancellation returns,
and the request stays.
With a multishot poll, IOSQE_ASYNC, and IORING_OP_ASYNC_CANCEL with
IORING_ASYNC_CANCEL_ALL up to 63us later, that happens to 4 of 1,000
requests (4 CPUs, KVM).
Take the stamp off again if the work matches. If the request is found
on one of the lists it gets a new one, and io_cancel_cb() doesn't match it
a second time.
__io_async_cancel() tries the io-wq of the caller twice. So far the stamp
has kept it from counting work that is running there the second time.
Skip it instead, so that the number that is returned stays the same.
Fixes:
|
||
|
Josef Bacik
|
3774e4d4c8 |
io_uring/cancel: recognize "no timeout" on 32-bit
IORING_REGISTER_SYNC_CANCEL waits without a limit if both fields of
io_uring_sync_cancel_reg::timeout are -1. io_sync_cancel() compares them
with -1UL. They are of type long long. With a long of 32 bits -1UL is
4294967295, which is what it stays when it is converted for the
comparison, and the fields are never equal to it.
So { -1, -1 } is taken for a timeout. ktime_set() takes the nanoseconds as
an unsigned long, so what becomes of it is -1s + 4294967295ns = 3.29s.
If the request is being run by io-wq and can't be interrupted, the call
returns -ETIME after that time, rather than waiting for the request to be
gone. On i386, with a buffered read from a null_blk device that takes 4s
to complete a request, IOSQE_ASYNC, that is what each call does.
Fixes:
|
||
|
Josef Bacik
|
8aa0e5db48 |
io_uring/cancel: don't forget that the work was running
If the request to be cancelled is being run by io-wq, io_async_cancel_one() flags the work, signals the worker and returns -EALREADY. Since commit |
||
|
Josef Bacik
|
5cb049d220 |
io_uring: don't lose a cancel as io-wq parks a request
IORING_OP_FUTEX_WAIT, IORING_OP_FUTEX_WAITV and IORING_OP_WAITID don't
block. They put the request on a list of the ring, where a wake up or a
cancellation finds it, and return IOU_ISSUE_SKIP_COMPLETE. They do so as
well if they are run by io-wq, e.g. because of IOSQE_ASYNC.
io_wq_submit_work() looks at IO_WQ_WORK_CANCEL once, before it issues the
request. Somebody who wants to cancel the request after that, and before
it is on the list, finds it running:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
signals the worker
= -EALREADY
...
io_futex_cancel() = -ENOENT
...
io_futex_wait()
adds the request to ->futex_list
= IOU_ISSUE_SKIP_COMPLETE
Nobody looks at the flag again. IORING_OP_ASYNC_CANCEL returns -EALREADY
and the request stays.
It is the problem that "io_uring/poll: don't lose a cancel as io-wq arms
poll" deals with for poll.
Look at the flag once the request is on the list, and cancel it if it is
set. The lists are searched and the requests are added with ->uring_lock
held, and the flag is set before the search. So either the search finds
the request, or it came first and the flag is seen.
What the canceller is told doesn't change, see the other patch, also for
IORING_ASYNC_CANCEL_ALL and _ANY.
Fixes:
|
||
|
Josef Bacik
|
6d2d4c2530 |
io_uring/poll: don't lose a cancel as io-wq arms poll
A request on a pollable file that was forced to io-wq is issued without
blocking there. If that ends in -EAGAIN, io_wq_submit_work() arms poll
for it and returns. It looks at IO_WQ_WORK_CANCEL once, before it issues
the request.
Somebody who wants to cancel the request in between finds it running:
io-wq worker io_try_cancel()
io_wq_submit_work()
IO_WQ_WORK_CANCEL isn't set
io_issue_sqe() = -EAGAIN
io_async_cancel_one()
sets IO_WQ_WORK_CANCEL
signals the worker
= -EALREADY
io_poll_cancel() = -ENOENT
...
= -ENOENT
io_arm_poll_handler() = IO_APOLL_OK
return
Nobody looks at the flag again, and the signal doesn't matter to a
worker that doesn't sleep. The request is armed as if nothing had
happened:
- a linked timeout completes with -ENOENT, and the request it was
meant to limit waits for as long as it takes.
- IORING_OP_ASYNC_CANCEL returns -EALREADY, and the request stays.
- IORING_REGISTER_SYNC_CANCEL waits for the request to complete if it
gets -EALREADY. Since commit
|
||
|
Josef Bacik
|
e9efeb4fc9 |
io_uring/io-wq: let cancel find picked up hashed work
io_get_next_work() takes all the work items that share a hash off the
work list in one go, and the worker runs them one after the other. Only
the one it is working on is visible as ->cur_work. The others are neither
on the list nor current, so io_wq_cancel_cb() can't find them.
IORING_OP_ASYNC_CANCEL and IORING_REGISTER_SYNC_CANCEL return -ENOENT for
a request that has not even been started, e.g. a buffered write queued
behind another write to the same file that is blocked, and the request is
carried out. The cancellation done for a task or ring that goes away gets
there in the end, as it keeps trying, but it has to wait for each item to
become current.
Keep a pointer to the remainder of the chain next to ->cur_work, and have
io_wq_worker_cancel() look at it. The items on it belong to the worker,
which needs ->lock to move on to the next one. Matching work is flagged
with IO_WQ_WORK_CANCEL and fails with -ECANCELED when its turn comes. The
worker isn't signalled for it, that would interrupt the unrelated work it
is busy with. As the request isn't gone yet, it is reported like running
work, with -EALREADY.
Fixes:
|
||
|
Josef Bacik
|
e231d2e8f8 |
io_uring/io-wq: don't exit with a creation pending
A worker that is the last one to go to sleep while work is pending asks
for another worker. io_wq_dec_running() bumps ->nr_running on behalf of
the new worker and queues create_worker_cb() as task_work for the owner
of the io-wq. Until that has run, everybody else takes it that a worker
is running which is going to look after the work list: io_wq_enqueue()
doesn't create one, and other workers that go to sleep don't find
themselves to be the last.
If the worker that made the request exits, io_worker_exit() cancels the
task_work, and io_worker_cancel_cb() drops ->nr_running without a look at
the work list:
worker A worker B io_wq_enqueue()
blocks, work is pending
nr_running++
queues create_worker_cb()
wakes, handles the work
idle timeout, decides to exit
picks up work 1
queues work 2
B is on the free list
still: wakes B
work 1 blocks
nr_running 2 -> 1
io_worker_exit()
io_worker_cancel_cb()
nr_running 1 -> 0
Work 2 has to wait for work 1, although there is room for more workers.
It takes an owner that doesn't get around to run its task_work for the
time of the idle timeout, less if the worker exits because its affinity
was changed or the io-wq is marked exit-on-idle.
Let the worker stay around until its request has been dealt with. Only
the worker itself sets ->create_state, so it can't be set behind its
back once it has decided to leave.
Fixes:
|
||
|
Josef Bacik
|
ad41f325be |
io_uring/io-wq: don't strand work when a worker exits
A worker exits if its idle timeout fired and it isn't the last one, if
its affinity was changed, or if the wq is marked exit-on-idle. It decides
so after it found the work list empty, and until io_worker_exit() is
through it stays on the free list and in ->nr_running:
worker io_wq_enqueue()
io_acct_run_queue() == false
queues work
io_acct_activate_free_worker()
wake_up_process(worker)
// found one, nothing to create
acct->nr_workers--
break
io_worker_exit()
Nobody is going to look at the work. If other workers exist, it has to
wait until one of them finishes what it is blocked on or hits its idle
timeout. If not, which is possible for the affinity and the exit-on-idle
case, it sits there until something else is queued. For exit-on-idle it
takes a task that closes its last ring and sets up a new one while the
workers are on their way out.
Make the worker take itself off the free list when it gives up its
->nr_workers slot, then leave ->nr_running, and only then take the final
look at the work list. That look and queueing work are both done under
acct->lock, so either the worker sees the work and stays, or the enqueue
side sees that the worker is gone and wakes or creates another one. If it
wants to stay and finds that its slot has been taken, a worker has been
created in its place, which is going to look at the work list, and it can
leave after all.
Fixes:
|
||
|
Josef Bacik
|
f19c38feb9 |
io_uring/io-wq: don't go to sleep on pending work
io_wq_enqueue() wakes a worker off the free list. If there is none it
creates one, unless ->nr_running says that a worker is running. That
relies on whoever takes ->nr_running to zero to check for pending work,
which io_wq_dec_running() no longer does for a worker without
->cur_work, on the grounds that an idle worker is either woken right
away or someone else has created a worker. Neither has to be the case:
worker A (idle) worker B (idle) io_wq_enqueue()
woken for work 1
woken (e.g. timeout) takes work 1
work list is empty
queues work 2
B is first on the free
list: wake_up_process(B)
leaves the free list
work 1 blocks
nr_running 2 -> 1
schedule_timeout()
nr_running 1 -> 0
->cur_work == NULL
Work 2 isn't looked at until work 1 is done or A's idle timeout fires,
five seconds later. ->cur_work is also NULL while a worker blocks on
->uring_lock in io_wq_put_work(), with the same result if it was the
only one running.
Let the idle worker leave ->nr_running by itself before it takes its last
look at the work list. Both that and queueing work are done under
acct->lock, so either the worker sees the work, or the enqueue side sees
that it can't count on this worker. This doesn't bring back the request
for a new worker which the check was meant to avoid, and as idle workers
don't get to io_wq_dec_running() any more, the check can go.
Fixes:
|
||
|
Josef Bacik
|
68a95c3968 |
io_uring/mpscq: disable preemption while pushing
A push makes the node the new tail, and then links it to its predecessor.
In between the queue isn't empty, but mpscq_pop() can't get at the node.
The consumers deal with that by trying again, with a cond_resched() in
the loop to give a preempted producer the chance to finish.
cond_resched() doesn't do that if the consumer has a higher priority than
the producer and both have to use the same CPU:
producer (SCHED_OTHER) consumer (SCHED_FIFO), same CPU
io_req_local_work_add()
mpscq_push()
prev = xchg(&q->tail, node)
<preempted> io_cqring_wait()
__io_run_local_work()
again:
cond_resched() // nothing to do
mpscq_pop() // NULL
// work pending, no events yet
goto again
The producer can be any task that is able to queue task_work for the
ring: an io-wq worker, or somebody who sends a message with
IORING_OP_MSG_RING. The consumer spins, holding ->uring_lock, until the
scheduler takes pity on the fair class, which takes about a second with
the default settings of the fair server, and forever without it. The
loops in tctx_task_work_run() and io_cancel_local_task_work() have the
same problem.
Don't allow preemption between the two steps. That bounds the time the
window is open on the CPU of the producer by what interrupts take, and a
consumer that waits for that is on another CPU. The cost is a
preempt_disable()/preempt_enable() pair per push.
Fixes:
|