Pull workqueue updates from Tejun Heo:
- Flushing a workqueue now visits only the queues used since the last
flush instead of one per possible CPU. Idle flushes get 10 to 100
times faster on large machines, which XFS hits on every log force.
- Per-cpu workqueues are built through the same attribute path as
unbound ones and get a max_active of their own, continuing the work
to turn them into an affinity scope of unbound workqueues.
- New tracepoints for work items turning CPU-intensive, rescuer mayday
and rescue events, and BH workers yielding with work pending.
- A KCSAN annotation and small cleanups
* tag 'wq-for-7.4' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
workqueue: mark pool->nr_running as __data_racy
workqueue: BUG_ON() a pwq release before workqueue_init()
workqueue: Derive the standard attrs from the workqueue flags
workqueue: Give percpu workqueues their own max_active
workqueue: Make flush_workqueue() visit only pwqs active since the last flush
workqueue: Maintain pwq->total_in_flight
workqueue: Add for_each_node_with_fallback()
workqueue: Fix typo "upto" in comment
workqueue: Add workqueue_bh_budget_yield tracepoint
workqueue: Add workqueue_mayday and workqueue_rescued tracepoints
workqueue: Add workqueue_cpu_intensive tracepoint
workqueue: rename the pwq slot helpers shared with percpu
workqueue: build percpu pwqs through the attrs path
workqueue: make the default pwq optional
workqueue: resolve the backing pool in alloc_pwq()
workqueue: move the unbound-only attrs check to apply_workqueue_attrs()
wq_worker_running() increments pool->nr_running with only preemption
disabled, while kick_pool_pick() reads it via need_more_worker() under
pool->lock, so the two can run concurrently on different CPUs. KCSAN
reports:
BUG: KCSAN: data-race in kick_pool_pick / wq_worker_running
write to 0xffff888059c2af24 of 4 bytes by task 3007 on cpu 0:
wq_worker_running+0x114/0x130
schedule_timeout+0x16e/0x180
io_schedule_timeout+0x4c/0x70
submit_bio_wait+0xd7/0x110
blkdev_issue_flush+0x4c/0x70
ext4_sync_file+0x60d/0x6c0
aio_fsync_work+0x7f/0x270
process_one_work+0x4ae/0xa00
read to 0xffff888059c2af24 of 4 bytes by task 3001 on cpu 1:
kick_pool_pick+0x7a/0x3a0
__queue_work+0x72d/0xcf0
queue_work_on+0x9d/0xb0
schedule_on_each_cpu+0x11a/0x2d0
vmstat_refresh+0x23/0x140
proc_sys_call_handler+0x2b9/0x4f0
The race is benign. The increment in wq_worker_running() is the only
lockless write to nr_running; decrements and the reset are done under
pool->lock, so a reader holding the lock sees either the old or the new
count. A stale 0 costs at most a spurious wakeup of an idle worker. A
nonzero count is correct because the incrementing worker is running and
re-checks the worklist under pool->lock before it stops counting. The
increment can also go 1 -> 2, e.g. when a worker sleeping in a work item
wakes up after kick_pool() woke another worker.
The lockless increment is part of how nr_running is designed to work,
as the comment above the field documents. Mark the field __data_racy
so that every reader is covered, not just need_more_worker().
Reported-by: syzbot+8922e6b8343f70be92e8@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=8922e6b8343f70be92e8
Signed-off-by: Nguyen Ngoc Thang <ngocthang2710.1999@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
current_is_workqueue_mem_reclaim() uses current_wq_worker() to decide
whether %current is a workqueue worker and then checks the current
workqueue flags. However, current_wq_worker() only implies that %current
is a kworker; it does not imply that the worker is currently executing a
work item.
worker->current_pwq is set while process_one_work() runs the work
function and cleared afterwards. A kworker outside work-item execution can
therefore have PF_WQ_WORKER set with current_pwq cleared.
Guard the flag check with worker->current_pwq. Without a current_pwq, the
worker is not executing on a WQ_MEM_RECLAIM workqueue, so the helper
should return false.
Fixes: da729ddd4a ("NFS/localio: issue IO inline when not in a memory-reclaim context")
Signed-off-by: Pavankumar Kondeti <pavan.kondeti@oss.qualcomm.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull workqueue fixes from Tejun Heo:
- Fix a NULL dereference in the chained work check when a kworker
queues work on a draining or destroying workqueue outside work item
execution.
* tag 'wq-for-7.3-rc6-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
workqueue: Fix NULL current_pwq deref in chained work check
current_wq_worker() only tells us that %current is a kworker. It does
not guarantee that the worker is currently executing a work item, as
worker->current_pwq is populated only while process_one_work() is running
the work function.
is_chained_work() can be reached while queueing on a draining or
destroying workqueue. If a kworker outside work-item execution reaches
that path, current_wq_worker() returns a worker but worker->current_pwq is
NULL, and the chained-work test can fault before it emits the intended
warning.
Guard the chained-work test with worker->current_pwq. A kworker without
current_pwq is not executing work on the target workqueue, so the helper
should return false and let the draining/destroying warning fire.
Fixes: 8d03ecfe47 ("workqueue: reimplement is_chained_work() using current_wq_worker()")
Cc: stable@vger.kernel.org
Signed-off-by: Pavankumar Kondeti <pavan.kondeti@oss.qualcomm.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull workqueue fix from Tejun Heo:
- Fix a NULL dereference in the flush dependency check when a worker
flushes outside a work item, such as from the OOM path during worker
creation
* tag 'wq-for-7.3-rc4-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
workqueue: Fix NULL current_pwq deref in flush dependency check
check_flush_dependency() uses current_wq_worker() to determine whether
the caller is a workqueue worker and then dereferences worker->current_pwq
to test whether the current workqueue is WQ_MEM_RECLAIM.
current_wq_worker() only means that %current has PF_WQ_WORKER set. A
kworker can reach check_flush_dependency() while it is not executing a
work item. One such path is worker_thread() acting as the pool manager,
where create_worker() does GFP_KERNEL allocation and the allocation path
invokes the OOM notifier. In that state worker->current_pwq is NULL
because current_pwq is set only by process_one_work() and cleared again
after the work function returns.
[ 416.760634][ T375] Call trace:
[ 416.760638][ T375] check_flush_dependency+0x80/0x120 (P)
[ 416.760648][ T375] __flush_work+0x98/0x224
[ 416.760657][ T375] flush_work+0x30/0x44
[ 416.760665][ T375] ...
[ 416.760710][ T375] blocking_notifier_call_chain+0x58/0xa0
[ 416.760719][ T375] out_of_memory+0xb4/0x458
[ 416.760730][ T375] __alloc_pages_may_oom+0x11c/0x1a8
[ 416.760739][ T375] __alloc_pages_slowpath+0x314/0x46c
[ 416.760746][ T375] __alloc_frozen_pages_noprof+0x110/0x1a4
[ 416.760753][ T375] new_slab+0x12c/0x484
[ 416.760759][ T375] ___slab_alloc+0x7a8/0xc7c
[ 416.760765][ T375] __slab_alloc+0x74/0xd8
[ 416.760772][ T375] __kmalloc_cache_node_noprof+0x2ac/0x304
[ 416.760779][ T375] alloc_worker+0x28/0x60
[ 416.760785][ T375] create_worker+0x4c/0x20c
[ 416.760790][ T375] worker_thread+0xe8/0x2b8
[ 416.760796][ T375] kthread+0x1a8/0x200
[ 416.760805][ T375] ret_from_fork+0x10/0x20
Guard the WQ_MEM_RECLAIM-worker warning with worker->current_pwq. If the
kworker is not currently executing a work item, there is no current
workqueue to diagnose with that warning. The PF_MEMALLOC warning is left
unchanged so explicit reclaim context flushing a !WQ_MEM_RECLAIM target
is still reported.
Fixes: fca839c00a ("workqueue: warn if memory reclaim tries to flush !WQ_MEM_RECLAIM workqueue")
Cc: stable@vger.kernel.org
Assisted-by: LLM
Signed-off-by: Pavankumar Kondeti <pavan.kondeti@oss.qualcomm.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
put_pwq() bounces the release to pwq_release_worker to get out from
under pool->lock. workqueue_init() creates that kthread_worker well
after workqueue_init_early() has built the system workqueues, so a pwq
allocation failing partway through apply_wqattrs_prepare() unwinds
through apply_wqattrs_cleanup() and dereferences a NULL worker.
There is no way back from that. Running the release inline is not an
option, that is what the bounce exists for, and workqueue_init_early()
already BUG_ON()s a system workqueue it failed to allocate.
BUG_ON() the missing worker instead of oopsing inside
kthread_queue_work().
Suggested-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/all/144b3dc986135ebe34f671cdd02bd8aa@kernel.org/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
alloc_and_link_pwqs() picks one of three static attrs arrays indexed by
WQ_HIGHPRI, and branches to a separate apply_workqueue_attrs_locked()
call for percpu, ordered and unbound workqueues. The arrays differ only
in the nice level and the ordered bit, which the flags already say.
Build the attrs from the flags instead. alloc_wq_std_attrs() sets nice
for WQ_HIGHPRI and ordered for __WQ_ORDERED, which collapses the three
branches into one call and drops the three arrays along with their
workqueue_init_early() loop.
The ordering WARN now sits behind its own __WQ_ORDERED test rather than
the unbound branch. __WQ_ORDERED is internal and every macro that sets it
pairs it with WQ_UNBOUND, so the WARN covers the same workqueues as
before.
The attrs are allocated per creation now. apply_wqattrs_prepare() already
allocates one on the same path, including in workqueue_init_early(), so
that is not a new constraint, and it only copies what it is given, so the
caller can free them once apply_workqueue_attrs_locked() returns.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
wq->max_active means two different things, depending on how the workqueue
is backed. On a concurrency managed per-cpu pool it caps pwq->nr_active,
so it is a per-cpu limit. On an unbound pool it feeds
wq_node_nr_active() and is shared by the whole workqueue.
Tejun once said [1]:
It kinda sucks that max_active's meaning is different across the
boundary tho. Maybe percpu max_active should be separate into
its own field, idk.
Add wq->percpu_max_active for the first meaning and leave max_active as
the system wide one. wq_adjust_max_active() updates whichever of the two
applies, and min_active stays with max_active as only the unbound side
uses it.
Suggested-by: Tejun Heo <tj@kernel.org>
Link: https://lore.kernel.org/all/amESSqf0TMmzhFGz@slm.duckdns.org/ [1]
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
636b927eba ("workqueue: Make unbound workqueues to use per-cpu
pool_workqueues") fixed unbound workqueue scalability on large machines but
made flush_workqueue() walk one pwq per possible CPU, cycling each pool
lock, up to three times per flush. 85f0d8e39a ("workqueue: Reduce
expensive locks for unbound workqueue") coalesced same-pool locks for
unbound workqueues but the walk remains. Yao Kai reported the XFS CIL
workqueue, flushed on every log force, spending up to 64us per flush in the
walk on a 128-CPU machine.
Idle pwqs can't just be skipped: the lock-and-advance of every pwq's work
color is what keeps a concurrently queued work item from being stamped with
a retired color unseen by the flusher, which would let a later flush return
before it finishes.
Track active pwqs instead. Each workqueue gets a wq_flush_pnode per node
with a lock, a mirror of wq->work_color and a list of pwqs. Queueing to an
off-list pwq syncs pwq->work_color from the mirror and adds it under
fpn->lock, and flush_workqueue_prep_pwqs() advances the mirror and splices
the list in one fpn->lock section per node before visiting the pwqs under
pool->lock as before. That replaces the fence: a racing queueing either gets
its pwq on the list before the splice or stamps the advanced color. A pwq
stays on the list until a visit finds nothing in flight, so a cascade arming
an older color still finds it, barriers need no separate add as the work
item they follow keeps the pwq on the list, and pwq_release_workfn() removes
a released pwq under wq->mutex.
On a 192-CPU 2-node machine, flushing an idle per-cpu workqueue goes from
40k to 4.2M per second and an idle unbound one from 450k to 4.3M. 16 threads
each queueing a work item and flushing, go from 30k to 110k flushes per
second on a per-cpu workqueue and 60k to 180k on an unbound one. Dense
flushes with every pwq active are unchanged on per-cpu workqueues and 15-20%
slower on unbound ones. The queue path is unchanged.
Reported-by: Yao Kai <yaokai34@huawei.com>
Link: https://lore.kernel.org/all/0a030145-c108-4365-ba2d-ac1973a1e352@huawei.com/
Signed-off-by: Tejun Heo <tj@kernel.org>
pwq_busy() scans all of pwq->nr_in_flight[] to tell whether anything is in
flight. Maintain the sum in pwq->total_in_flight, which fits in existing
padding, and test that instead.
Signed-off-by: Tejun Heo <tj@kernel.org>
Per-node arrays in workqueue carry an extra slot at nr_node_ids for
NUMA_NO_NODE, and iterating them open-codes for_each_node() plus separate
handling of that slot. Add for_each_node_with_fallback() and convert
alloc/free_node_nr_active().
Signed-off-by: Tejun Heo <tj@kernel.org>
Correct "upto" to "up to", reported by scripts/checkpatch.pl using the
misspelling list in scripts/spelling.txt. Only touches comments, no code
changes.
Assisted-by: Cursor:claude-opus-5
Signed-off-by: Hemanth Selam <hemanth.selam@gmail.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
Bottom-Half (BH) workqueues execute work items in softirq context.
To prevent softirqs from starving user and kernel threads, bh_worker()
enforces execution limits (i.e., BH_WORKER_JIFFIES and BH_WORKER_RESTARTS).
When keep_working() is still true but either the time slice or restart
count is exhausted, bh_worker() yields execution and re-raises the
softirq via kick_bh_pool().
Currently, there is no observability into when a BH worker hits these
limits and is forced to yield.
Add the workqueue_bh_budget_yield tracepoint, emitted when bh_worker() exits
the processing loop with pending work items remaining. It records, the
worker pool ID, pool CPU, number of loop restarts consumed, a boolean flag
indicating whether the yield was due to a time slice timeout, and a boolean
flag indicating whether this is a high-priority BH pool.
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
When a worker pool fails to create a new worker thread (e.g., typically
under severe memory pressure where memory reclaim paths depend on pending
work items), send_mayday() signals distress to the workqueue's rescuer
thread. The rescuer then takes over processing the pending work items via
assign_rescuer_work().
While pwq->stats[PWQ_STAT_MAYDAY] and pwq->stats[PWQ_STAT_RESCUED] track
these occurrences cumulatively, there is currently no event-driven
mechanism to observe exactly when mayday distress occurs or which work
items require rescue.
Add two new tracepoints namely workqueue_mayday and workqueue_rescued to
make distress and rescuer execution easily observable (e.g., via Ftrace
or eBPF).
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
When a concurrency-managed per-CPU work item runs continuously without
sleeping for longer than wq_cpu_intensive_thresh_us, wq_worker_tick() marks
the worker as WORKER_CPU_INTENSIVE and kicks it out of concurrency
management so that pending work items on the pool are not starved.
While CONFIG_WQ_CPU_INTENSIVE_REPORT logs rate-limited warnings and
pwq->stats[PWQ_STAT_CPU_INTENSIVE] maintains a cumulative counter, there is
currently no tracepoint emitted at the moment of this transition.
Therefore, add the workqueue_cpu_intensive tracepoint, recording the
work_struct pointer and callback function pointer, workqueue name,
executing CPU, and the runtime duration consumed in microseconds.
This enables eBPF profilers, bpftrace, and Ftrace to immediately detect and
attribute CPU-hogging work items in real time.
Signed-off-by: Aaron Tomlin <atomlin@atomlin.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
install_unbound_pwq() fills the pwq slots of a percpu workqueue as well now
that alloc_and_link_pwqs() sends it through the attrs path, and
unbound_pwq_slot() and unbound_pwq() have always addressed and read those
slots for every workqueue, percpu included, from destroy_workqueue().
Their names say otherwise. Rename them to install_pwq(), pwq_slot() and
installed_pwq().
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Drop the WQ_PERCPU allocation path (alloc_and_link_percpu_pwqs()).
Now that alloc_pwq() hands out the static percpu pool for a CPU and the
default pwq is optional, the unbound path produces the same result.
Point a percpu workqueue at it and drop alloc_and_link_percpu_pwqs().
Nothing is installed in wq->cpu_pwq when the attrs path fails, since
apply_wqattrs_commit() does not run, so the enomem cleanup only has the
percpu array left to free.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
apply_wqattrs_prepare() always creates a dfl_pwq, the fallback a CPU
uses when it has no pwq of its own. This is a CPU unbound field, so, we
don't need it for the per cpu affinity version of workqueue.
A percpu workqueue has no use for one. Skip the allocation for
a WQ_PERCPU workqueue and skip installing what was not allocated.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Unify the pool allocation in alloc_pwq(), instead of only getting the unbound
pool, and add a new parameter for the CPU.
Create a helper for unbound allocations (alloc_unbound_pwq()).
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
apply_workqueue_attrs_locked() refuses any workqueue without WQ_UNBOUND.
That is the right rule for changing a workqueue's attributes, but it also
blocks the internal callers, and building a percpu workqueue's pwq table
through this path needs them.
Move the check into apply_workqueue_attrs(), the exported entry point,
which is the one that changes attributes on a live workqueue.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull workqueue fixes from Tejun Heo:
- An unbound worker pool could be freed while still reachable through
the pending-activation list, leading to a use-after-free. Unlink
before dropping the reference
- On PREEMPT_RT, the BH workqueue kick raised softirqs from preemptible
context, tripping a lockdep assertion and possibly losing
concurrently raised softirq bits
- Draining BH work off a dead CPU nests two pools' callback locks,
which lockdep misreported as recursive locking. The nesting cannot
deadlock. Annotate it
- Reject watchdog thresholds that overflow the conversion to jiffies
- Make the drgn workqueue dump script work again on kernels and vmcores
from before the workqueue attrs field rename
* tag 'wq-for-7.3-rc1-fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/wq:
tools/workqueue/wq_dump.py: Support backward compatibility for wq->attrs rename
workqueue: reject watchdog thresholds that overflow jiffies
workqueue: Fix unbound pool lifetime for pending pwqs
workqueue: Use raise_softirq() to trigger softirq in irq_work handler
workqueue: Annotate cb_lock nesting when draining a dead BH pool
The watchdog threshold is supplied in seconds but is multiplied by HZ
before being used as a jiffies interval. Reject values that exceed
MAX_JIFFY_OFFSET / HZ so the multiplication cannot wrap and the
time_after() comparisons remain within their supported range.
The check is performed before changing the threshold or watchdog timer.
Zero remains the value used to disable the watchdog.
Fixes: 82607adcf9 ("workqueue: implement lockup detector")
Signed-off-by: Jiacheng Xu <stitch@zju.edu.cn>
Signed-off-by: Tejun Heo <tj@kernel.org>
Pull NFS client updates from Trond Myklebust:
"Highlights include:
Stable fixes:
- Use-after-free fixes for the sunrpc client code
- Delegation hash table leak
- NULL dereference on lockowner allocation failure
- Fix a handshake completion race in the TLS code
- Fix an error sign checking issue when deciding whether the pNFS
layout is still in use, or can be returned
- Fix a layout segment leak in pnfs_layout_process()
Other bugfixes:
- Fix a missing NULL check in the rpcbind client
- annotate shared socket callbacks with READ_ONCE/WRITE_ONCE
- nfs_inode_set_delegation() error paths should return the delegation
- Use clear_and_wake_up_bit() in nfs_clear_invalid_mapping() and the
pNFS code.
- Fix the nfs4_alloc_client() error paths to free the IDR allocation
- fix folio dereference before NULL check in
nfs_inode_remove_request()
- Fix delayed delegation return
- Fix another state manager race with umount
- Fix device leaks on parse failure
- Avoid cancelling in-flight I/O during a layout recall if the server
doesn't require it
- flexfiles: report cancelled I/O as a layout error
- flexfiles: fix NULL dereference for NFSv4.0 data servers
- Fix incorrect argument passed to nfs4_delete_lease()
- Fix several symlink issues resulting from nfs_atomic_open_v23()
- Fix an uninitialised variable issue in the NFSv4.1 callback code
- fix LAYOUTSTATS send buffer exhaustion
Features and cleanups:
- NFSv4.2: Allow the server to specify that file data may not be cached
- localio: optimise I/O submission when when not doing memory reclaim
- localio: Remove duplicate wait code in nfs_local_commit
- flexfiles: support loosely coupled NFSv4.x data servers
- pNFS: key the data server cache on the NFS version"
* tag 'nfs-for-7.3-1' of git://git.linux-nfs.org/projects/trondmy/linux-nfs: (33 commits)
NFSv4.1: fix layout segment leak on the pnfs_layout_process() forget path
NFSv4/pnfs: key the data server cache on the NFS version
NFSv4.2: fix LAYOUTSTATS send buffer exhaustion
pNFS: Fix EBUSY check in pnfs_layout_need_return
NFSv4.1: zero referring call lists before decoding
nfs: fix ENXIO on O_CREAT open of existing symlink over NFSv3
SUNRPC: wait for in-flight client TLS handshake callback
NFSv4: Fix incorrect argument passed to nfs4_delete_lease() in nfs4_add_lease()
lockd: fix NULL dereference on lockowner allocation failure
NFS: fix delegation_hash_table leak when nfs4_server_common_setup() fails
NFSv4/flexfiles: support loosely coupled data servers
NFSv4/flexfiles: fix NULL dereference for NFSv4.0 data servers
NFSv4: pin the superblock for active state owners
sunrpc: fix use-after-free in __rpc_clnt_handle_event and __rpc_clnt_remove_pipedir
NFS/localio: issue commit inline when not in a memory-reclaim context
NFS/localio: remove dead FLUSH_SYNC handling from nfs_local_commit
NFS/localio: issue IO inline when not in a memory-reclaim context
NFS: Fix delayed delegation return list handling
NFS: Verify symlink inode before caching target
NFS: fix folio dereference before NULL check in nfs_inode_remove_request()
...
KASAN reports a use-after-free of an unbound worker_pool in
node_activate_pending_pwq():
BUG: KASAN: slab-use-after-free in _raw_spin_trylock+0x6d/0x120
Read of size 4 at addr ffff8880089ce000 by task kworker/u22:0/318
CPU: 1 UID: 0 PID: 318 Comm: kworker/u22:0 Not tainted 7.2.0 #1 PREEMPT(lazy)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Workqueue: 0x0 (flush-8:0)
Call Trace:
<TASK>
dump_stack_lvl+0x53/0x70
print_report+0xce/0x610
kasan_report+0xce/0x100
_raw_spin_trylock+0x6d/0x120
pwq_dec_nr_in_flight+0x4b4/0xcb0
process_one_work+0x921/0x11a0
worker_thread+0x4d0/0xd20
kthread+0x2de/0x3c0
ret_from_fork+0x3aa/0x620
ret_from_fork_asm+0x1a/0x30
</TASK>
Allocated by task 311:
alloc_pwq+0x439/0xca0
apply_wqattrs_prepare+0x75e/0xd10
apply_workqueue_attrs_locked+0x44/0xa0
wq_nice_store+0x350/0x450
Freed by task 0:
kfree+0x127/0x3b0
rcu_core+0x523/0x1780
handle_softirqs+0x1b3/0x610
Last potentially related work creation:
put_unbound_pool+0x3f3/0x7d0
pwq_release_workfn+0x494/0x8e0
kthread_worker_fn+0x1ff/0x790
Canceling the last inactive work skips pwq_dec_nr_active(), so an empty
pwq can remain on pending_pwqs when its refcnt reaches zero.
pwq_release_workfn() currently puts the pool before removing that pwq.
If this drops the last pool reference, the pool can be RCU-freed while
the pwq remains reachable, and node_activate_pending_pwq() may trylock
the freed pool->lock.
Remove the pwq from pending_pwqs before putting the pool.
Fixes: 5797b1c189 ("workqueue: Implement system-wide nr_active enforcement for unbound workqueues")
Cc: stable@vger.kernel.org
Signed-off-by: Yao Kai <yaokai34@huawei.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
bh_pool_kick_normal() and bh_pool_kick_highpri() are registered via
init_irq_work() without the IRQ_WORK_HARD_IRQ flag. On PREEMPT_RT, such
irq_work items are processed by the per-CPU irq_workd kthread in preemptible
task context with IRQs enabled. However, raise_softirq_irqoff() requires
IRQs to be disabled. Calling it from irq_workd trips the lockdep assertion
in __raise_softirq_irqoff() and the non-atomic update of the softirq pending
mask can lose bits raised by an interrupt on the same CPU. Replace
raise_softirq_irqoff() with raise_softirq() in the irq_work handlers.
Fixes: 2f34d7337d ("workqueue: Fix queue_work_on() with BH workqueues")
Cc: stable@vger.kernel.org # v6.9+
Signed-off-by: Zqiang <qiang.zhang@linux.dev>
Signed-off-by: Tejun Heo <tj@kernel.org>
On PREEMPT_RT, bh_worker() wraps work item execution in pool->cb_lock to
provide a handshake for canceling BH work items. When a CPU goes down,
drain_dead_softirq_workfn() runs the dead pool's bh_worker() nested inside
the local pool's bh_worker(), acquiring the cb_locks of two different pools
without a nesting annotation. lockdep reports possible recursive locking:
============================================
WARNING: possible recursive locking detected
--------------------------------------------
ktimers/0/16 is trying to acquire lock:
ffff8880b873a990 (&pool->cb_lock){+...}-{3:3}, at: bh_worker+0x7d/0x880
but task is already holding lock:
ffff8880b863a990 (&pool->cb_lock){+...}-{3:3}, at: bh_worker+0x7d/0x880
Call Trace:
bh_worker+0x7d/0x880 kernel/workqueue.c:3688
drain_dead_softirq_workfn+0x95/0x220 kernel/workqueue.c:3763
process_scheduled_works+0xa8e/0x14e0 kernel/workqueue.c:3405
bh_worker+0x46a/0x880 kernel/workqueue.c:3708
tasklet_action+0xc/0x70 kernel/softirq.c:965
The nesting can't deadlock. A pool's bh_worker() runs nested only while the
pool's CPU is dead, entered from a live pool's bh_worker() on the draining
CPU, so the ordering is always live to dead. CPU hotplug operations are
serialized and the drain is synchronous, so the nesting depth never exceeds
two. Annotate the inner acquisition with SINGLE_DEPTH_NESTING.
Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: syzbot+1bd20115328f8254ed62@syzkaller.appspotmail.com
Closes: https://syzkaller.appspot.com/bug?extid=1bd20115328f8254ed62
Fixes: ad7c7f4b9c ("workqueue: Provide a handshake for canceling BH workers")
Cc: stable@vger.kernel.org # v6.18+
Every LOCALIO read and write is currently bounced through the dedicated
!WQ_MEM_RECLAIM nfslocaliod_workqueue. That bounce is only actually
required when the submitting context is a memory-reclaim context: LOCALIO
issues IO directly into a stacked local filesystem (e.g. XFS) which may in
turn flush its own !WQ_MEM_RECLAIM workqueue. Doing that from a
WQ_MEM_RECLAIM worker (most importantly writeback's wb_workfn on bdi_wq) or
an explicit PF_MEMALLOC reclaim task trips check_flush_dependency() and
risks a forward-progress deadlock, which is why commit b9f5dd57f4
("nfs/localio: use dedicated workqueues for filesystem read and write")
introduced the intermediate workqueue.
Outside of reclaim context -- ordinary application/task submission such as
O_DIRECT or fsync-driven writeback -- the workqueue hop buys nothing and
merely adds a context switch and scheduling latency per IO while discarding
the NFS client's inherent application-context parallelism.
Add current_is_workqueue_mem_reclaim(), which reports whether %current is a
WQ_MEM_RECLAIM worker using the same predicate check_flush_dependency()
warns on. Use it, together with the PF_MEMALLOC check, in the new
nfs_local_defer_io() helper to decide per-IO whether nfs_local_do_read()
and nfs_local_do_write() must defer to nfslocaliod_workqueue or may issue
the IO inline. Buffered writeback continues to bounce (wb_workfn is a
WQ_MEM_RECLAIM worker); O_DIRECT and app-context submission now run inline.
Running nfs_local_call_write() inline is safe: it already saves and
restores current->flags around the PF_LOCAL_THROTTLE|PF_MEMALLOC_NOIO it
sets and scopes the file opener's creds. The async O_DIRECT completion
path is likewise unaffected: when the underlying filesystem returns
-EIOCBQUEUED, the kiocb ki_complete callback (nfs_local_read_aio_complete /
nfs_local_write_aio_complete) can run in bottom-half context and so must
still defer the pgio completion (nfs_local_pgio_release -> rpc_call_done) to
nfsiod_workqueue via nfs_local_pgio_aio_complete(). That completion hop is
independent of how the IO was submitted, and this change leaves it as-is;
only the submission side stops unconditionally hopping through
nfslocaliod_workqueue.
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Mike Snitzer <snitzer@kernel.org>
Signed-off-by: Trond Myklebust <trond.myklebust@hammerspace.com>
kick_pool_pick() reads and writes p->wake_cpu while the scheduler can
update it concurrently. KCSAN reports:
BUG: KCSAN: data-race in kick_pool_pick+0xf8/0x2d8
race at unknown origin, with read to 0xffff000663229da4 of 4 bytes by
task 1817002 on cpu 40:
kick_pool_pick+0xf8/0x2d8
process_scheduled_works+0x2bc/0x888
worker_thread+0x394/0x548
kthread+0x1b8/0x1f0
ret_from_fork+0x10/0x20
value changed: 0x0000002b -> 0x0000002f
The race is harmless. wake_cpu is a best-effort placement hint: every
writer stores a valid CPU id and the wakeup path validates it through
select_task_rq(), so a stale value only affects which CPU the worker
wakes up on.
Mark both accesses with READ_ONCE() and WRITE_ONCE() to document that
they are intentionally racy and to stop the compiler from reloading or
tearing them.
Signed-off-by: Breno Leitao <leitao@debian.org>
Reviewed-by: Bradley Morgan <include@grrlz.net>
Signed-off-by: Tejun Heo <tj@kernel.org>
wq_node_nr_active() warns and returns NULL when @wq is not unbound, but
every caller dereferences the result right away, so the WARN_ON_ONCE()
only moves the oops one frame up, as raised by Tejun.
Fix it by BUGing_ON() instead of this silly WARN_ON_ONCE();
Fixes: b72fdc6510 ("workqueue: account nr_active by the backing pool")
Suggested-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
wq->cpu_pwq holds RCU-protected pwq pointers, but the percpu allocation
path fills it in with plain loads and stores, which sparse flags:
kernel/workqueue.c:5682:57: sparse: incorrect type in initializer (different address spaces) @@ expected struct pool_workqueue **pwq_p @@ got struct pool_workqueue [noderef] __rcu ** @@
Allocate the array as __rcu pointers and publish each pwq with
rcu_assign_pointer() once it is initialized and linked, the order
install_unbound_pwq() uses.
The warnings are not new: commit 79f23600bc ("workqueue: factor out
get_percpu_pool()") only turned the flagged assignment into an
initializer.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
workqueue_congested() fetches the pwq out of wq->cpu_pwq with a plain
load, so sparse complains about the dropped __rcu:
kernel/workqueue.c:6304:13: sparse: incorrect type in assignment (different address spaces) @@ expected struct pool_workqueue *pwq @@ got struct pool_workqueue [noderef] __rcu * @@
A pwq is released with kfree_rcu() and the read is protected by the
surrounding preempt_disable(), which is what
commit fd5081f4ef ("workqueue: Remove redundant rcu_read_lock/unlock() in
workqueue_congested()") relied on when it dropped the rcu_read_lock()
here.
Use the rcu_dereference_sched() helper to make that explicit.
Reported-by: kernel test robot <lkp@intel.com>
Closes: https://lore.kernel.org/oe-kbuild-all/202608120931.tvTzq1gD-lkp@intel.com/
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
apply_wqattrs_commit() updates node_nr_active->max unconditionally.
wq->node_nr_active[] is only allocated for unbound workqueues, so guard
the call before per-cpu workqueues start using this path.
No functional change: only unbound workqueues reach apply_wqattrs_*()
today.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
This allocates a pwq and binds it to the pool @attrs asks for.
Which pool that is becomes a property of the attrs (once per-cpu becomes
an affinity scope).
Remove the 'unbound" from the function name, given it will be bigger
than unbound.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
The attrs are where the affinity scope lives, and a per-cpu workqueue
will need one once per-cpu becomes a scope rather than a separate
backend. Allocate them unconditionally.
wq_dump.py used a non-NULL wq->attrs as its test for an unbound
workqueue, which no longer holds; test WQ_UNBOUND there instead.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
The unbound prefix says which workqueues currently have the field rather
than what it holds, and the next patch allocates it for every workqueue.
Rename it first so that change stays a single line.
tools/workqueue/wq_dump.py reads the field by name, so rename it there
too.
wq_sysfs_unbound_attrs[] keeps its name: it is the set of sysfs files
that only unbound workqueues expose.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
workqueue_online_cpu() and workqueue_offline_cpu() decide whether a
workqueue needs a pod affinity update by testing wq->unbound_attrs for
NULL, which is only meaningful because the attrs are allocated for
unbound workqueues alone.
Test the flag instead, so the attrs can later be allocated for every
workqueue.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
pwq_tryinc_nr_active() and pwq_dec_nr_active() choose between the shared
per-node nr_active and the plain per-pwq one by testing
wq_node_nr_active() for NULL.
Test the backing pool with is_percpu_pool() instead, so the accounting
follows the pool that runs the work rather than the workqueue type.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Add is_percpu_pool() and test the pool directly for per cpu. Convert the
other open-coded pool->cpu checks -- in put_unbound_pool(),
pool_allowed_cpus() and the workqueue watchdog -- to the same helper.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Move the per-cpu pwq allocation loop out of alloc_and_link_pwqs() into a
helper. The inner allocation-failure path now returns -ENOMEM and the
caller jumps to the existing enomem cleanup, equivalent to the previous
goto.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Move the static per-cpu worker_pool lookup in alloc_and_link_pwqs() into
a helper, get_percpu_pool(), so the lookup can be shared by other
pool-selection paths.
No functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
The automatic CPU-intensive work item detection reads the worker task's
se.sum_exec_runtime without a lock in wq_worker_running(),
wq_worker_tick() and process_one_work(). The scheduler updates that
field under the rq lock (from the tick via update_curr(), or cross-CPU
via task_sched_runtime()), raising:
BUG: KCSAN: data-race in wq_worker_running+0xa8/0xe8
race at unknown origin, with read to 0xffff0009a11d1df8 of 8 bytes by task 238535 on cpu 68:
wq_worker_running
schedule
schedule_preempt_disabled
__mutex_lock
mutex_lock_nested
cgroup_bpf_release
process_one_work
worker_thread
kthread
ret_from_fork
value changed: 0x0000000088482ba0 -> 0x00000000884893c0
The value only feeds a heuristic, so the race is benign-ish. Unlike
commit ecf5aad9a4 ("workqueue: annotate racy PWQ_STAT_CPU_TIME update
in wq_worker_tick()") that only needs data_race(), these are plain reads
whose result drives a subtraction and comparison, so use READ_ONCE() for
a single, non-torn load, which also silences KCSAN.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
wq_worker_tick() bumps pwq->stats[PWQ_STAT_CPU_TIME] on every scheduler
tick before pool->lock is taken. For unbound workqueues the
pool_workqueue is shared by all workers of the pool across CPUs, so
concurrent ticks on different CPUs perform an unsynchronized 64-bit
read-modify-write on the same counter. KCSAN reports this as a
data-race:
BUG: KCSAN: data-race in wq_worker_tick / wq_worker_tick
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 29:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
update_process_times+0x200/0x260
tick_nohz_handler+0x230/0x2f8
__hrtimer_run_queues+0x1ec/0x6c8
hrtimer_interrupt+0x174/0x4b8
...
read-write to 0xffff0004d6989500 of 8 bytes by interrupt on cpu 24:
wq_worker_tick+0x70/0x418
sched_tick+0x248/0x3a0
...
value changed: 0x000000000010a1d0 -> 0x000000000010a9a0
The counter is purely advisory, so an occasional lost update is
harmless, and every other stats[] update already runs under pool->lock.
Annotate the update with data_race().
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
To identify the task most likely responsible for a stall, add
last_woken_worker (L: pool->lock) to worker_pool and record it in
kick_pool() just before wake_up_process(). This captures the idle
worker that was kicked to take over when the last running worker went to
sleep; if the pool is now stuck with no running worker, that task is the
prime suspect and its backtrace is dumped by show_pool_no_running_worker().
Using struct worker * rather than struct task_struct * avoids any
lifetime concern: workers are only destroyed via set_worker_dying()
which requires pool->lock, and set_worker_dying() clears
last_woken_worker when the dying worker matches.
show_cpu_pool_busy_workers() holds pool->lock while calling
sched_show_task(), so last_woken_worker is either NULL or points to a
live worker with a valid task. More precisely, set_worker_dying() clears
last_woken_worker before setting WORKER_DIE, so a non-NULL
last_woken_worker means the kthread has not yet exited and worker->task
is still alive.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
When a CPU pool is stalled with no running worker, the task occupying the
CPU may not be a workqueue worker at all. Trigger a single-CPU backtrace
for the stalled CPU to capture what it is currently executing.
The CPU is snapshotted under pool->lock and the backtrace is triggered
after releasing the lock to avoid any potential issues with NMI delivery.
Skip the backtrace when the CPU is offline. A pool disassociated by CPU
hotplug keeps its pool->cpu, and an NMI to an offline CPU is never acked,
so nmi_trigger_cpumask_backtrace() would busy-wait for its full timeout
in the watchdog's timer context.
Suggested-by: Petr Mladek <pmladek@suse.com>
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
show_cpu_pool_busy_workers() dumps every in-flight worker in the pool's
busy_hash, including workers that are not currently running on the CPU.
Restore the task_is_running() filter so only running workers are dumped.
When no running worker is found the pool may be stuck, unable to wake an
idle worker to process pending work, and the watchdog would otherwise
give no feedback. Add show_pool_no_running_worker() to report the pool
id, CPU, idle state, and worker counts in that case.
The pool info message is printed inside pool->lock using
printk_deferred_enter/exit, the same pattern used by the existing
busy-worker loop, to avoid deadlocks with console drivers that queue
work while holding locks also taken in their write paths.
This has been running on the Meta fleet for a while and caught some real
issues, for instance EFI stalls stalling the workqueue [1].
Link: https://lore.kernel.org/all/20260616-efi_timeout-v3-0-76dd1d26657b@debian.org/ [1]
Suggested-by: Petr Mladek <pmladek@suse.com>
Fixes: 8823eaef45 ("workqueue: Show all busy workers in stall diagnostics")
Reviewed-by: Petr Mladek <pmladek@suse.com>
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Use kick_pool_pick() to select and claim the worker under pool->lock and
issue the wakeup with wake_up_process() after the lock is dropped.
Unlike __queue_work(), this path has no surrounding RCU section, so take
rcu_read_lock() before dropping pool->lock to keep the picked worker's
task_struct valid across the wakeup.
Signed-off-by: Breno Leitao <leitao@debian.org>
Tested-by: Krishna Magar <kmagar@redhat.com>
Signed-off-by: Tejun Heo <tj@kernel.org>
__queue_work() is the enqueue hot path: it inserts the work item and
calls kick_pool() while holding pool->lock. kick_pool() ends in a
wakeup, which takes the target task's rq->lock, so rq->lock nests under
pool->lock on every enqueue that wakes a worker on a contended unbound
pool.
Use kick_pool_pick() to select and claim the worker under pool->lock and
issue the wakeup with wake_up_process() right after dropping the lock.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Factor the worker selection out of kick_pool() into kick_pool_pick(),
which picks and claims the worker under pool->lock but, instead of waking
it, returns the worker's task via an out-param so the caller can issue
the wakeup after dropping pool->lock. BH kicks and wake_cpu setup still
happen under the lock.
kick_pool() becomes a thin wrapper that wakes the returned task, so all
existing callers keep waking under pool->lock.
Pure refactor, no functional change.
Signed-off-by: Breno Leitao <leitao@debian.org>
Signed-off-by: Tejun Heo <tj@kernel.org>
Linus Torvalds