diff --git a/docs/configuration/optimization.md b/docs/configuration/optimization.md index 18122ea2c8b..4ef9108db7c 100644 --- a/docs/configuration/optimization.md +++ b/docs/configuration/optimization.md @@ -156,10 +156,47 @@ GPU. vLLM can pin each worker with `numactl` before the Python subprocess starts so the interpreter, imports, and early allocator state are created with the desired NUMA policy from the beginning. -Use `--numa-bind` to enable the feature. By default, vLLM auto-detects the -GPU-to-NUMA mapping and uses `--cpunodebind= --membind=` for each -worker. When you need a custom CPU policy, add `--numa-bind-cpus` and vLLM will -switch to `--physcpubind= --membind=`. +By default, vLLM binds automatically when the environment supports it: +CUDA-alike platform, multiple NUMA nodes, `numactl` installed, permission to +set NUMA memory policy, unrestricted CPU affinity, a default NUMA memory policy, +the `spawn` multiprocessing start method (the CLI default), and a local `mp` or +`uni` executor with multiprocessing DP, including supported prefill context +parallelism (PCP) launches. Ray, external launchers and `--device-ids` remapping +are excluded from automatic binding. +Otherwise vLLM starts unbound. Use +`--numa-bind` to make a missing `numactl` or undetectable GPU-to-NUMA topology a +startup error, or `--no-numa-bind` to disable binding. Passing +`--numa-bind-nodes` or `--numa-bind-cpus` explicitly enables binding even without +`--numa-bind`: either list requests strict `--membind`, not automatic soft memory +preference or CPU-only binding. Neither can be combined with `--no-numa-bind`. +Automatic binding preserves an existing non-default memory policy, such as +interleave, preferred-node, or bind. Explicit enablement may override it. + +Automatic binding uses `--cpunodebind= --preferred=` for each worker: +memory allocations prefer the GPU-local node but can fall back to other nodes. +EngineCore prefers its shard's nodes with `--preferred-many=` when the +shard spans multiple NUMA nodes, or `--preferred=` for a single-node shard. +Explicit `--numa-bind` or binding lists retain strict `--membind=`. +When you need a custom CPU policy, add `--numa-bind-cpus` and vLLM will switch +to `--physcpubind=`. + +Automatic eligibility and topology validation happen in each local launcher +before starting its first bound subprocess, using freshly discovered topology. +The mapping must cover all visible GPUs and local workers, and each mapped NUMA +node must have CPUs. The resolved decision and mapping are passed to spawned +children, so an already-bound EngineCore does not repeat detection under its +restricted CPU affinity. +PCP workers use the full TP × PCP × PP shard size for local DP GPU offsets; +EngineCore's CPU affinity covers all workers in its shard. +Multi-node multiprocessing launchers resolve their own host-local mapping; +NUMA node IDs are never interpreted as cluster-wide IDs. + +Before each bound launch, vLLM probes the selected policy with +`numactl true`. If CPU-and-memory binding is rejected, it retries with +CPU-only binding; if that is also rejected, the subprocess starts unbound. +Logs report the selected policy or fallback. If `--preferred-many` is unsupported, +automatic mode falls back to CPU-only binding, never strict memory binding. +These probes test policy acceptance, not future allocation capacity. These `--numa-bind*` options only apply to GPU execution processes. They do not configure the CPU backend's separate thread-affinity controls. Automatic @@ -174,7 +211,7 @@ order as the GPU indices. Each CPU list must use `numactl --physcpubind` syntax such as `0-3`, `0,2,4-7`, or `16-31,48-63`. ```bash -# Auto-detect NUMA nodes for visible GPUs +# Require binding, auto-detecting NUMA nodes for visible GPUs vllm serve meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 4 \ --numa-bind @@ -195,10 +232,12 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \ Notes: -- CLI usage forces multiprocessing to use the `spawn` method automatically. If you enable NUMA binding through the Python API, also set `VLLM_WORKER_MULTIPROC_METHOD=spawn`. -- Automatic detection relies on NVML and NUMA support from the host. If it cannot determine the mapping reliably, pass `--numa-bind-nodes` explicitly. +- The CLI defaults to `spawn` when `VLLM_WORKER_MULTIPROC_METHOD` is unset. Automatic NUMA binding never changes the start method: if it is `fork`, default binding stays off. Python API usage normally defaults to `fork`; set `VLLM_WORKER_MULTIPROC_METHOD=spawn` to allow binding and follow Python's `if __name__ == "__main__":` requirements. +- Automatic memory preference permits allocations on other NUMA nodes when local memory is unavailable. Explicit `--numa-bind` still uses `--membind`, which restricts host allocations to the selected nodes; for host-memory-heavy setups (e.g. large CPU KV offload), avoid explicit strict binding if workers need memory from other nodes. The policy probes do not check allocation capacity or container memory limits. +- Automatic detection relies on NVML (CUDA) or amdsmi (ROCm) and NUMA support from the host. If it cannot determine the mapping reliably, default mode starts unbound; pass `--numa-bind-nodes` to provide a mapping explicitly. - Explicit `--numa-bind-nodes` and `--numa-bind-cpus` values must be valid `numactl` inputs. vLLM does a small amount of validation, but the effective binding semantics are still determined by `numactl`. - The current implementation binds GPU execution processes such as `EngineCore` and multiprocessing workers. It does not apply NUMA binding to frontend API server processes or the DP coordinator. +- Ray actors are not bound by the multiprocessing executable wrapper. Enabling automatic NUMA binding for Ray requires a separate actor-launch implementation. - In containerized environments, NUMA policy syscalls may require extra permissions, such as `--cap-add SYS_NICE` when running via `docker run`. ### CPU Backend Thread Affinity diff --git a/tests/engine/test_arg_utils.py b/tests/engine/test_arg_utils.py index 938cf9b3999..c183b32a8d3 100644 --- a/tests/engine/test_arg_utils.py +++ b/tests/engine/test_arg_utils.py @@ -804,6 +804,11 @@ def test_numa_bind_args(): assert engine_args.numa_bind_nodes == [0, 0, 1, 1] assert engine_args.numa_bind_cpus == ["0-3", "4-7", "8-11", "12-15"] + # Unset means auto; --no-numa-bind disables explicitly. + assert EngineArgs.from_cli_args(args=parser.parse_args([])).numa_bind is None + args = parser.parse_args(["--no-numa-bind"]) + assert EngineArgs.from_cli_args(args=args).numa_bind is False + def test_ir_op_priority(): from vllm.config.kernel import IrOpPriorityConfig, KernelConfig diff --git a/tests/utils_/test_numa_utils.py b/tests/utils_/test_numa_utils.py index 3c09722f82d..014e2f7da7d 100644 --- a/tests/utils_/test_numa_utils.py +++ b/tests/utils_/test_numa_utils.py @@ -2,8 +2,10 @@ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import os +from copy import deepcopy from functools import partial from types import SimpleNamespace +from typing import Any import pytest @@ -39,20 +41,30 @@ def _disable_pct_by_default(monkeypatch): def _make_config(**parallel_kwargs): - parallel_defaults = dict( + parallel_defaults: dict[str, Any] = dict( numa_bind=False, + _numa_bind_auto=False, numa_bind_nodes=None, numa_bind_cpus=None, + assigned_physical_gpu_ids=None, distributed_executor_backend="mp", data_parallel_backend="mp", nnodes_within_dp=1, data_parallel_rank_local=0, data_parallel_index=0, + data_parallel_size_local=1, pipeline_parallel_size=1, tensor_parallel_size=1, prefill_context_parallel_size=1, ) parallel_defaults.update(parallel_kwargs) + parallel_defaults.setdefault( + "local_world_size", + parallel_defaults["pipeline_parallel_size"] + * parallel_defaults["tensor_parallel_size"] + * parallel_defaults["prefill_context_parallel_size"] + // parallel_defaults["nnodes_within_dp"], + ) parallel_config = SimpleNamespace(**parallel_defaults) parallel_config.world_size = ( parallel_config.tensor_parallel_size @@ -214,12 +226,18 @@ def test_pct_binding_returns_none_when_node_cpulist_missing(monkeypatch): assert numa_utils._maybe_get_pct_cpu_binding([0]) is None -def test_get_numactl_args_uses_pct_when_user_did_not_specify_cpus(monkeypatch): +@pytest.mark.parametrize("automatic", [False, True]) +def test_get_numactl_args_uses_pct_when_user_did_not_specify_cpus( + monkeypatch, automatic +): _patch_pct_gates(monkeypatch, model_match=True, highest_perf=46) - vllm_config = _make_config(numa_bind=True, numa_bind_nodes=[0, 1]) + vllm_config = _make_config( + numa_bind=True, _numa_bind_auto=automatic, numa_bind_nodes=[0, 1] + ) + memory_policy = "preferred" if automatic else "membind" assert ( numa_utils._get_numactl_worker_args(vllm_config.parallel_config, local_rank=1) - == "--physcpubind=0,1,16,17,64,65,80,81 --membind=1" + == f"--physcpubind=0,1,16,17,64,65,80,81 --{memory_policy}=1" ) @@ -249,7 +267,10 @@ def test_get_numactl_args_engine_core_baseline_spans_shard_numa_nodes(): ) -def test_get_numactl_args_engine_core_pct_spans_shard_numa_nodes(monkeypatch): +@pytest.mark.parametrize("automatic", [False, True]) +def test_get_numactl_args_engine_core_pct_spans_shard_numa_nodes( + monkeypatch, automatic +): """PCT: EngineCore for a multi-NUMA shard binds to the union of priority cores across all shard nodes, so worker `--physcpubind` is always a subset of EngineCore's `cpus_allowed`.""" @@ -261,15 +282,17 @@ def test_get_numactl_args_engine_core_pct_spans_shard_numa_nodes(monkeypatch): ) vllm_config = _make_config( numa_bind=True, + _numa_bind_auto=automatic, numa_bind_nodes=[0, 0, 1, 1], tensor_parallel_size=4, ) + memory_policy = "preferred-many" if automatic else "membind" assert numa_utils._get_numactl_enginecore_args( vllm_config.parallel_config, local_rank=0 ) == ( "--physcpubind=" "0,1,16,17,64,65,80,81,128,129,144,145,192,193,208,209" - " --membind=0,1" + f" --{memory_policy}=0,1" ) @@ -543,6 +566,18 @@ def test_parallel_config_validates_numa_bind_nodes(): ParallelConfig(numa_bind_nodes=[0, -1]) +@pytest.mark.parametrize( + ("nodes", "cpus"), [([0, 1], None), (None, ["0-3"])], ids=["nodes", "cpus"] +) +def test_parallel_config_numa_lists_imply_numa_bind(nodes, cpus): + assert ParallelConfig().numa_bind is None + config = ParallelConfig(numa_bind_nodes=nodes, numa_bind_cpus=cpus) + assert config.numa_bind is True + assert config._numa_bind_auto is False + with pytest.raises(ValueError, match="cannot be combined with numa_bind=False"): + ParallelConfig(numa_bind=False, numa_bind_nodes=nodes, numa_bind_cpus=cpus) + + @pytest.mark.parametrize("cpuset", ["", "abc", "1-", "4-1", "1,,2", "1:2"]) def test_parallel_config_rejects_invalid_numa_bind_cpus(cpuset): with pytest.raises(ValueError, match="numa_bind_cpus"): @@ -564,7 +599,7 @@ def test_configure_subprocess_numa_fallback(monkeypatch): import multiprocessing monkeypatch.setattr("shutil.which", lambda name: "/usr/bin/numactl") - monkeypatch.setattr(numa_utils.envs, "VLLM_WORKER_MULTIPROC_METHOD", "spawn") + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn") node_config = _make_config(numa_bind=True, numa_bind_nodes=[0]) monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([])) @@ -593,3 +628,409 @@ def test_configure_subprocess_numa_fallback(monkeypatch): with numa_utils.configure_subprocess(node_config, local_rank=0): assert multiprocessing.spawn.get_executable() == before assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + + +def _patch_auto_numa( + monkeypatch, start_method, detect, visible_gpus=1, node_cpus="0-7" +): + monkeypatch.setattr("shutil.which", lambda name: "/usr/bin/numactl") + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", start_method) + monkeypatch.setattr(numa_utils, "_is_auto_numa_available", lambda **kwargs: True) + monkeypatch.setattr(numa_utils, "get_auto_numa_nodes", detect) + monkeypatch.setattr( + "vllm.platforms.current_platform.device_count", lambda: visible_gpus + ) + real_read_text = numa_utils.Path.read_text + + def read_text(path, *args, **kwargs): + if str(path).startswith("/sys/devices/system/node/node"): + return node_cpus + return real_read_text(path, *args, **kwargs) + + monkeypatch.setattr(numa_utils.Path, "read_text", read_text) + + +@pytest.mark.parametrize("nodes", [[0], [0, 1]]) +def test_configure_subprocess_auto_binds_and_records_result(monkeypatch, nodes): + """Default mode resolves once in the launcher and stores the result, so + spawned children bind with it instead of re-checking eligibility.""" + _patch_auto_numa(monkeypatch, "spawn", lambda: nodes, visible_gpus=len(nodes)) + monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([])) + + vllm_config = _make_config(numa_bind=None, tensor_parallel_size=len(nodes)) + node_list = ",".join(map(str, nodes)) + preference = "preferred" if len(nodes) == 1 else "preferred-many" + with numa_utils.configure_subprocess( + vllm_config, local_rank=0, process_kind="EngineCore" + ): + assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == ( + f"--cpunodebind={node_list} --{preference}={node_list}" + ) + child_config = deepcopy(vllm_config) + assert vllm_config.parallel_config.numa_bind is True + assert child_config.parallel_config._numa_bind_auto is True + assert child_config.parallel_config.numa_bind_nodes == nodes + + monkeypatch.setattr(numa_utils, "get_auto_numa_nodes", _raise_probe_error) + monkeypatch.setattr(numa_utils, "_is_auto_numa_available", lambda **kwargs: False) + with numa_utils.configure_subprocess(child_config, local_rank=len(nodes) - 1): + assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == ( + f"--cpunodebind={nodes[-1]} --preferred={nodes[-1]}" + ) + + +def test_parallel_config_auto_policy_survives_copy_without_changing_hash(): + """Copying the launch decision must not change the DP configuration hash.""" + from vllm.config.utils import replace + + parallel_config = ParallelConfig() + original_hash = parallel_config.compute_hash() + parallel_config.numa_bind = True + parallel_config.numa_bind_nodes = [0] + parallel_config._numa_bind_auto = True + for copied in ( + replace(parallel_config), + deepcopy(parallel_config), + ): + assert copied._numa_bind_auto is True + assert copied.compute_hash() == original_hash + + +@pytest.mark.parametrize("dp_local_rank", [0, 1]) +def test_auto_numa_accepts_pcp_dp_topology(monkeypatch, dp_local_rank): + """Automatic eligibility must cover every PCP worker in both DP shards.""" + _patch_auto_numa(monkeypatch, "spawn", lambda: [0, 0, 1, 1], visible_gpus=4) + config = ParallelConfig( + distributed_executor_backend="mp", + prefill_context_parallel_size=2, + data_parallel_size=2, + data_parallel_size_local=2, + ) + assert numa_utils._can_auto_bind(config, local_rank=0, dp_local_rank=dp_local_rank) + assert config.numa_bind_nodes == [0, 0, 1, 1] + + +def test_auto_numa_redetects_when_visible_devices_change(monkeypatch): + """Independent launchers must not reuse another visible GPU set's mapping.""" + _patch_auto_numa( + monkeypatch, "spawn", numa_utils.get_auto_numa_nodes, visible_gpus=2 + ) + monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([])) + detections = [] + + def detect(): + devices = os.environ["CUDA_VISIBLE_DEVICES"] + detections.append(devices) + return {"0,1": [0, 0], "2,3": [1, 1]}[devices] + + monkeypatch.setattr( + "vllm.platforms.current_platform.get_all_device_numa_nodes", + detect, + raising=False, + ) + for devices, node in [("0,1", 0), ("2,3", 1)]: + monkeypatch.setenv("CUDA_VISIBLE_DEVICES", devices) + vllm_config = _make_config(numa_bind=None, tensor_parallel_size=2) + with numa_utils.configure_subprocess( + vllm_config, local_rank=0, process_kind="EngineCore" + ): + numactl_args = os.environ[numa_utils._NUMACTL_ARGS_ENV] + assert numactl_args == f"--cpunodebind={node} --preferred={node}" + assert vllm_config.parallel_config.numa_bind_nodes == [node, node] + assert detections == ["0,1", "2,3"] + + +@pytest.mark.parametrize("explicit", [False, True], ids=["automatic", "explicit"]) +@pytest.mark.parametrize( + "policy", + [0, 1, 2, 3, 4, 5, None], + ids=[ + "default", + "preferred", + "bind", + "interleave", + "local", + "preferred-many", + "unreadable", + ], +) +def test_configure_subprocess_respects_existing_mempolicy( + monkeypatch, policy, explicit +): + """Only explicit enablement may replace an existing non-default policy.""" + is_auto_numa_available = numa_utils._is_auto_numa_available + _patch_auto_numa(monkeypatch, "spawn", numa_utils.get_auto_numa_nodes) + monkeypatch.setattr(numa_utils, "_is_auto_numa_available", is_auto_numa_available) + monkeypatch.setattr("vllm.platforms.current_platform.is_cuda_alike", lambda: True) + monkeypatch.setattr(numa_utils.os.path, "isdir", lambda path: True) + monkeypatch.setattr(numa_utils.psutil, "cpu_count", lambda: 4) + monkeypatch.setattr( + numa_utils.psutil, + "Process", + lambda pid: SimpleNamespace(cpu_affinity=lambda: [0, 1, 2, 3]), + ) + monkeypatch.setattr( + "vllm.platforms.current_platform.get_all_device_numa_nodes", + lambda: [0], + raising=False, + ) + + def get_mempolicy(mode, *_args): + if policy is None: + return -1 + mode._obj.value = policy + return 0 + + monkeypatch.setattr( + numa_utils, + "get_libnuma", + lambda: SimpleNamespace(numa_available=lambda: 0, get_mempolicy=get_mempolicy), + ) + monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([])) + vllm_config = _make_config(numa_bind=True if explicit else None) + if explicit and policy is None: + with ( + pytest.raises(RuntimeError, match="could not detect"), + numa_utils.configure_subprocess(vllm_config, local_rank=0), + ): + pass + return + + expected_bind = policy is not None and (policy == 0 or explicit) + memory_arg = "--membind=0" if explicit else "--preferred=0" + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + numactl_args = os.environ.get(numa_utils._NUMACTL_ARGS_ENV) + assert numactl_args == ( + f"--cpunodebind=0 {memory_arg}" if expected_bind else None + ) + assert vllm_config.parallel_config.numa_bind is expected_bind + + +def _raise_probe_error(): + raise RuntimeError("amdsmi init failed") + + +@pytest.mark.parametrize( + ("start_method", "detect", "assigned_gpus"), + [ + ("fork", lambda: [0], None), + ("spawn", lambda: None, None), + ("spawn", lambda: [], None), + ("spawn", _raise_probe_error, None), + ("spawn", lambda: [0, 1], [1]), + ], + ids=["fork", "no-topology", "no-visible-gpus", "probe-raises", "device-ids"], +) +def test_configure_subprocess_auto_falls_back_to_unbound( + monkeypatch, start_method, detect, assigned_gpus +): + """Default mode never fails startup or forces spawn; when binding cannot + engage (or would pick the wrong node) it starts unbound.""" + import multiprocessing + + _patch_auto_numa(monkeypatch, start_method, detect) + + vllm_config = _make_config(numa_bind=None, assigned_physical_gpu_ids=assigned_gpus) + before = multiprocessing.spawn.get_executable() + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + assert multiprocessing.spawn.get_executable() == before + assert vllm_config.parallel_config.numa_bind is False + + +@pytest.mark.parametrize( + "parallel_kwargs", + [ + {"distributed_executor_backend": "ray"}, + {"data_parallel_backend": "ray"}, + {"distributed_executor_backend": "external_launcher"}, + ], + ids=["ray-executor", "ray-dp", "external-launcher"], +) +def test_auto_numa_skips_unsupported_launchers(monkeypatch, parallel_kwargs): + detections = [] + + def detect(): + detections.append(True) + return [0, 1] + + _patch_auto_numa(monkeypatch, "spawn", detect, visible_gpus=2) + vllm_config = _make_config(numa_bind=None, **parallel_kwargs) + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + assert vllm_config.parallel_config.numa_bind is False + assert detections == [] + + +@pytest.mark.parametrize("nodes", [[0], [0, 1, 2], [0, -1], [0, None]]) +def test_auto_numa_rejects_incomplete_or_invalid_topology(monkeypatch, nodes): + _patch_auto_numa(monkeypatch, "spawn", lambda: nodes, visible_gpus=2) + vllm_config = _make_config(numa_bind=None, tensor_parallel_size=2) + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + assert vllm_config.parallel_config.numa_bind is False + assert vllm_config.parallel_config.numa_bind_nodes is None + + +def test_auto_numa_checks_all_local_dp_workers_before_first_engine(monkeypatch): + _patch_auto_numa(monkeypatch, "spawn", lambda: [0, 1], visible_gpus=2) + vllm_config = _make_config( + numa_bind=None, tensor_parallel_size=2, data_parallel_size_local=2 + ) + with numa_utils.configure_subprocess( + vllm_config, local_rank=0, dp_local_rank=0, process_kind="EngineCore" + ): + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + assert vllm_config.parallel_config.numa_bind is False + + +def test_auto_numa_multi_node_topology_is_local(monkeypatch): + _patch_auto_numa(monkeypatch, "spawn", lambda: [2, 3], visible_gpus=2) + monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([])) + vllm_config = _make_config( + numa_bind=None, tensor_parallel_size=4, nnodes_within_dp=2 + ) + with numa_utils.configure_subprocess( + vllm_config, local_rank=0, process_kind="EngineCore" + ): + assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == ( + "--cpunodebind=2,3 --preferred-many=2,3" + ) + child_config = deepcopy(vllm_config) + with numa_utils.configure_subprocess(child_config, local_rank=1): + assert ( + os.environ[numa_utils._NUMACTL_ARGS_ENV] == "--cpunodebind=3 --preferred=3" + ) + + +@pytest.mark.parametrize("node_cpus", [None, "", "invalid"]) +def test_auto_numa_skips_nodes_without_usable_cpus(monkeypatch, node_cpus): + _patch_auto_numa(monkeypatch, "spawn", lambda: [0], node_cpus=node_cpus) + vllm_config = _make_config(numa_bind=None) + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + assert vllm_config.parallel_config.numa_bind is False + + +@pytest.mark.parametrize( + "nodes, rejected_args, expected_args", + [ + ([0], "--preferred=", "--cpunodebind=0"), + ([0], "--cpunodebind=", None), + ([0, 1], "--preferred-many=", "--cpunodebind=0,1"), + ([0, 1], "--cpunodebind=", None), + ], +) +def test_auto_numa_records_effective_policy( + monkeypatch, caplog, nodes, rejected_args, expected_args +): + caplog.set_level("INFO", logger=numa_utils.logger.name) + _patch_auto_numa(monkeypatch, "spawn", lambda: nodes, visible_gpus=len(nodes)) + probes = [] + fake_run = _fake_numactl_run([rejected_args]) + + def run(cmd, *args, **kwargs): + probes.extend(cmd) + return fake_run(cmd, *args, **kwargs) + + monkeypatch.setattr(numa_utils.subprocess, "run", run) + vllm_config = _make_config(numa_bind=None, tensor_parallel_size=len(nodes)) + with numa_utils.configure_subprocess( + vllm_config, local_rank=0, process_kind="EngineCore" + ): + assert os.environ.get(numa_utils._NUMACTL_ARGS_ENV) == expected_args + assert vllm_config.parallel_config.numa_bind is (expected_args is not None) + launch_logs = [r.getMessage() for r in caplog.records if "Launching" in r.message] + if expected_args is None: + assert launch_logs == [] + assert vllm_config.parallel_config.numa_bind_nodes is None + assert vllm_config.parallel_config._numa_bind_auto is False + else: + assert len(launch_logs) == 1 + assert launch_logs[0].endswith(f"numactl {expected_args}") + assert not any(arg.startswith("--membind=") for arg in probes) + + +def test_auto_numa_does_not_swallow_launch_errors(monkeypatch): + import multiprocessing + + _patch_auto_numa(monkeypatch, "spawn", lambda: [0]) + monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([])) + monkeypatch.delenv(numa_utils._NUMACTL_ARGS_ENV, raising=False) + monkeypatch.delenv(numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV, raising=False) + original_executable = multiprocessing.spawn.get_executable() + with ( + pytest.raises(RuntimeError, match="worker startup failed"), + numa_utils.configure_subprocess(_make_config(numa_bind=None), local_rank=0), + ): + raise RuntimeError("worker startup failed") + assert multiprocessing.spawn.get_executable() == original_executable + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + assert numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV not in os.environ + + +@pytest.mark.parametrize("missing_numactl", [False, True]) +def test_auto_numa_skips_unavailable_environment(monkeypatch, missing_numactl): + _patch_auto_numa(monkeypatch, "spawn", lambda: [0]) + if missing_numactl: + monkeypatch.setattr("shutil.which", lambda name: None) + else: + monkeypatch.setattr( + numa_utils, "_is_auto_numa_available", lambda **kwargs: False + ) + vllm_config = _make_config(numa_bind=None) + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + assert vllm_config.parallel_config.numa_bind is False + + +def test_explicit_numa_nodes_bypass_auto_eligibility(monkeypatch): + _patch_auto_numa(monkeypatch, "spawn", _raise_probe_error) + monkeypatch.setattr(numa_utils, "_is_auto_numa_available", lambda **kwargs: False) + monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([])) + vllm_config = _make_config(numa_bind=True, numa_bind_nodes=[7]) + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == "--cpunodebind=7 --membind=7" + + +def test_explicit_numa_requires_detectable_topology(monkeypatch): + _patch_auto_numa(monkeypatch, "spawn", lambda: None) + with ( + pytest.raises(RuntimeError, match="could not detect"), + numa_utils.configure_subprocess(_make_config(numa_bind=True), local_rank=0), + ): + pass + + +@pytest.mark.parametrize("numa_bind", [False, True]) +def test_explicit_numa_handles_missing_numactl(monkeypatch, numa_bind): + monkeypatch.setattr("shutil.which", lambda name: None) + vllm_config = _make_config(numa_bind=numa_bind, numa_bind_nodes=[0]) + if numa_bind: + with ( + pytest.raises(RuntimeError, match="numactl is required"), + numa_utils.configure_subprocess(vllm_config, local_rank=0), + ): + pass + else: + with numa_utils.configure_subprocess(vllm_config, local_rank=0): + assert numa_utils._NUMACTL_ARGS_ENV not in os.environ + + +@pytest.mark.parametrize("affinity", [[0, 1], None, [3, 2, 1, 0]]) +def test_auto_numa_checks_cpu_affinity(monkeypatch, affinity): + monkeypatch.setattr("vllm.platforms.current_platform.is_cuda_alike", lambda: True) + monkeypatch.setattr(numa_utils.os.path, "isdir", lambda path: True) + monkeypatch.setattr(numa_utils.psutil, "cpu_count", lambda: 4) + process = SimpleNamespace() + if affinity is not None: + process.cpu_affinity = lambda: affinity + monkeypatch.setattr(numa_utils.psutil, "Process", lambda pid: process) + monkeypatch.setattr(numa_utils, "_get_mempolicy", lambda: 0) + monkeypatch.setattr( + "vllm.platforms.current_platform.get_all_device_numa_nodes", + lambda: [0, 1], + raising=False, + ) + assert numa_utils._is_auto_numa_available( + verbose=False, require_unbound_process=True + ) is (affinity == [3, 2, 1, 0]) diff --git a/tests/utils_/test_system_utils.py b/tests/utils_/test_system_utils.py index 592af714a5b..25d4ef4b5d1 100644 --- a/tests/utils_/test_system_utils.py +++ b/tests/utils_/test_system_utils.py @@ -11,6 +11,7 @@ import tempfile from pathlib import Path import psutil +import pytest from vllm.utils import system_utils from vllm.utils.system_utils import ( @@ -32,11 +33,39 @@ def test_unique_filepath(): assert len(list(Path(temp_dir).glob("*.txt"))) == 10 -def test_numa_bind_forces_spawn(monkeypatch): - monkeypatch.delenv("VLLM_WORKER_MULTIPROC_METHOD", raising=False) - monkeypatch.setattr("sys.argv", ["vllm", "serve", "--numa-bind"]) +@pytest.mark.parametrize("start_method", [None, "fork"]) +@pytest.mark.parametrize( + "numa_args, force_spawn", + [ + (["--numa-bind"], True), + (["--numa-bind-nodes", "0"], True), + (["--numa-bind-nodes=0"], True), + (["--numa-bind-cpus", "0-3"], True), + (["--numa-bind-cpus=0-3"], True), + (["--numa_bind"], True), + (["--numa_bind_nodes", "0"], True), + (["--numa_bind_cpus=0-3"], True), + ([], False), + (["--no-numa-bind"], False), + ], +) +def test_numa_bind_cli_start_method(monkeypatch, start_method, numa_args, force_spawn): + if start_method is None: + monkeypatch.delenv("VLLM_WORKER_MULTIPROC_METHOD", raising=False) + else: + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", start_method) + monkeypatch.setattr(sys, "argv", ["vllm", "serve", *numa_args]) + for check in ( + "is_in_ray_actor", + "cuda_is_initialized", + "xpu_is_initialized", + "in_wsl", + ): + monkeypatch.setattr(system_utils, check, lambda: False) + _maybe_force_spawn() - assert os.environ["VLLM_WORKER_MULTIPROC_METHOD"] == "spawn" + expected_method = "spawn" if force_spawn else start_method + assert os.environ.get("VLLM_WORKER_MULTIPROC_METHOD") == expected_method def test_suppress_stdout_when_sys_stdout_has_no_fd(capfd): diff --git a/tests/v1/distributed/test_dense_dp_world_size.py b/tests/v1/distributed/test_dense_dp_world_size.py index 1fc37936463..dd17b0234a2 100644 --- a/tests/v1/distributed/test_dense_dp_world_size.py +++ b/tests/v1/distributed/test_dense_dp_world_size.py @@ -1,17 +1,80 @@ # SPDX-License-Identifier: Apache-2.0 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project import os +from typing import TYPE_CHECKING +import pytest import requests -from tests.utils import RemoteOpenAIServer +from tests.utils import VLLM_PATH, RemoteOpenAIServer from vllm.platforms import current_platform +if TYPE_CHECKING: + import torch + + from vllm.config import ParallelConfig + MODEL_NAME = "Qwen/Qwen3-0.6B" DP_SIZE = int(os.getenv("DP_SIZE", "2")) TP_SIZE = int(os.getenv("TP_SIZE", "1")) +class NumaBindingWorkerExtension: + parallel_config: "ParallelConfig" + local_rank: int + device: "torch.device" + + def check_numa_binding(self): + import subprocess + from pathlib import Path + + import psutil + import torch.distributed as dist + + from vllm.distributed import get_pcp_group, get_world_group + from vllm.utils.cpu_resource_utils import parse_id_list + + parallel = self.parallel_config + assert parallel.numa_bind and parallel._numa_bind_auto + assert parallel.numa_bind_nodes is not None + gpu_index = self.local_rank + node = current_platform.get_device_numa_node(self.device.index) + assert parallel.numa_bind_nodes[gpu_index] == node + node_cpus = parse_id_list( + Path(f"/sys/devices/system/node/node{node}/cpulist").read_text().strip() + ) + cpus = set(psutil.Process().cpu_affinity()) + assert cpus and cpus <= set(node_cpus) + result = subprocess.run( + ["numactl", "--show"], + check=True, + capture_output=True, + text=True, + timeout=10, + ) + policy = dict(line.split(":", 1) for line in result.stdout.splitlines()) + assert policy["policy"].strip() == "preferred" + assert policy["preferred node"].strip() == str(node) + parent = psutil.Process(os.getppid()) + parent_cpus = set(parent.cpu_affinity()) + assert cpus <= parent_cpus + state = { + "pid": os.getpid(), + "gpu_index": gpu_index, + "gpu_uuid": current_platform.get_device_uuid(self.device.index), + "dp_rank": parallel.data_parallel_rank, + "pcp_rank": get_pcp_group().rank_in_group, + "node": node, + "cpus": sorted(cpus), + "engine_core_pid": parent.pid, + "engine_core_cpus": sorted(parent_cpus), + } + world = get_world_group() + states = [None] * world.world_size + dist.all_gather_object(states, state, group=world.cpu_group) + return states + + def test_dense_dp_world_size(): server_args = [ "--dtype", @@ -50,3 +113,79 @@ def test_dense_dp_world_size(): ) response.raise_for_status() assert response.json() == {"world_size": TP_SIZE} + + +@pytest.mark.slow_test +def test_pcp_dp_with_auto_numa(): + """Automatic binding must cover both PCP workers in both local DP shards.""" + from shutil import which + + from vllm.utils import numa_utils + + if not current_platform.is_cuda() or current_platform.device_count() < 4: + pytest.skip("PCP+DP NUMA smoke test requires four NVIDIA GPUs") + if which("numactl") is None or not numa_utils._is_auto_numa_available( + verbose=False, require_unbound_process=True + ): + pytest.skip("Host cannot engage automatic NUMA binding") + + server_args = [ + "--dtype", + "bfloat16", + "--max-model-len", + "2048", + "--max-num-seqs", + "8", + "--max-num-batched-tokens", + "2048", + "--kv-cache-memory-bytes", + str(256 << 20), + "--enforce-eager", + "--distributed-executor-backend", + "mp", + "--prefill-context-parallel-size", + "2", + "--data-parallel-size", + "2", + "--data-parallel-size-local", + "2", + "--worker-extension-cls", + "tests.v1.distributed.test_dense_dp_world_size.NumaBindingWorkerExtension", + ] + with RemoteOpenAIServer( + "deepseek-ai/DeepSeek-V2-Lite-Chat", + server_args, + env_dict={ + "VLLM_SERVER_DEV_MODE": "1", + "VLLM_WORKER_MULTIPROC_METHOD": "spawn", + "PYTHONPATH": os.pathsep.join( + filter(None, [str(VLLM_PATH), os.environ.get("PYTHONPATH")]) + ), + }, + max_wait_seconds=1200, + ) as server: + completion = server.get_client().completions.create( + model="deepseek-ai/DeepSeek-V2-Lite-Chat", + prompt="The first five positive integers are", + temperature=0, + max_tokens=16, + ) + assert completion.choices[0].text.strip() + assert completion.usage.completion_tokens > 0 + print(f"PCP+DP completion: {completion.choices[0].text!r}") + response = requests.post( + server.url_for("collective_rpc"), + json={"method": "check_numa_binding", "timeout": 30}, + timeout=60, + ) + response.raise_for_status() + states = response.json()["results"][0] + print(f"PCP+DP NUMA worker states: {states}") + assert sorted(s["gpu_index"] for s in states) == [0, 1, 2, 3] + assert len({s["gpu_uuid"] for s in states}) == 4 + assert sorted((s["dp_rank"], s["pcp_rank"]) for s in states) == [ + (0, 0), + (0, 1), + (1, 0), + (1, 1), + ] diff --git a/tests/v1/simple_kv_offload/test_integration.py b/tests/v1/simple_kv_offload/test_integration.py index c9119b7bff2..d0916a6e440 100644 --- a/tests/v1/simple_kv_offload/test_integration.py +++ b/tests/v1/simple_kv_offload/test_integration.py @@ -10,8 +10,11 @@ from transformers import Llama4ForCausalLM, Llama4TextConfig from tests.models.utils import check_logprobs_close from vllm import LLM, SamplingParams, TokensPrompt, envs -from vllm.config import KVTransferConfig +from vllm.config import KVTransferConfig, ParallelConfig from vllm.platforms import current_platform +from vllm.utils import numa_utils +from vllm.v1.metrics.reader import Counter +from vllm.v1.simple_kv_offload.metrics import MetricName if not current_platform.is_cuda_alike(): pytest.skip("Requires CUDA or ROCm", allow_module_level=True) @@ -113,7 +116,7 @@ def test_chunked_local_partial_gpu_hit(vllm_runner, tmp_path, monkeypatch): assert mixed_hit.num_cached_tokens == 16 -def _make_llm(model: str, lazy: bool, cpu_bytes_to_use: int) -> LLM: +def _make_llm(model: str, lazy: bool, cpu_bytes_to_use: int, **llm_kwargs) -> LLM: kv_transfer_config = KVTransferConfig( kv_connector="SimpleCPUOffloadConnector", kv_role="kv_both", @@ -122,12 +125,18 @@ def _make_llm(model: str, lazy: bool, cpu_bytes_to_use: int) -> LLM: "lazy_offload": lazy, }, ) + llm_kwargs = ( + dict( + kv_cache_memory_bytes=40 << 30, # 40 GiB + disable_hybrid_kv_cache_manager=False, + enable_prefix_caching=True, + ) + | llm_kwargs + ) return LLM( model=model, - kv_cache_memory_bytes=40 << 30, # 40 GiB - disable_hybrid_kv_cache_manager=False, - enable_prefix_caching=True, kv_transfer_config=kv_transfer_config, + **llm_kwargs, ) @@ -231,6 +240,90 @@ def _latency_test(llm: LLM, lazy: bool = False): ) +def _assert_worker_auto_numa(worker): + import subprocess + from pathlib import Path + + import psutil + + from vllm.utils.cpu_resource_utils import parse_id_list + + parallel_config = worker.parallel_config + assert parallel_config.numa_bind and parallel_config._numa_bind_auto + gpu_index = parallel_config.get_local_gpu_index(worker.local_rank) + node = parallel_config.numa_bind_nodes[gpu_index] + node_cpus = parse_id_list( + Path(f"/sys/devices/system/node/node{node}/cpulist").read_text().strip() + ) + cpus = set(psutil.Process().cpu_affinity()) + assert cpus and cpus <= set(node_cpus) + result = subprocess.run( + ["numactl", "--show"], check=True, capture_output=True, text=True, timeout=10 + ) + policy = dict(line.split(":", 1) for line in result.stdout.splitlines()) + assert policy["policy"].strip() == "preferred" + assert policy["preferred node"].strip() == str(node) + return {"node": node, "cpus": sorted(cpus), "policy": "preferred"} + + +@pytest.mark.slow_test +def test_simple_cpu_offload_with_auto_numa(monkeypatch, enable_pickle): + """Soft-bound NVIDIA workers must store and restore KV, not silently recompute.""" + if not current_platform.is_cuda(): + pytest.skip("Automatic NUMA KV-offload smoke test requires NVIDIA") + monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn") + monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "1") + preflight = ParallelConfig(distributed_executor_backend="mp") + preflight._numa_bind_auto = True + if not numa_utils._can_auto_bind(preflight, local_rank=0) or not ( + numa_utils._probe_numactl_args( + numa_utils._get_numactl_worker_args(preflight, local_rank=0) + ) + ): + pytest.skip("Host cannot engage automatic NUMA memory preference") + + model = "Qwen/Qwen3-0.6B" + llm = _make_llm( + model, + False, + 256 << 20, + kv_cache_memory_bytes=128 << 20, + max_model_len=1024, + distributed_executor_backend="mp", + enforce_eager=True, + disable_log_stats=False, + ) + + def loaded_blocks(llm: LLM): + return sum( + metric.value + for metric in llm.get_metrics() + if isinstance(metric, Counter) + and metric.name == MetricName.LOAD_BLOCKS.removesuffix("_total") + and metric.labels.get("model_name") == model + ) + + try: + states = llm.collective_rpc(_assert_worker_auto_numa, timeout=30) + print(f"NUMA worker state: {states}") + sampling_params = SamplingParams(max_tokens=8, temperature=0) + prompt = "hi " * 256 + "Let's count to ten. One, two, three, " + cold = llm.generate(prompt, sampling_params, use_tqdm=False)[0] + llm.generate(prompt, sampling_params, use_tqdm=False) + deadline = time.monotonic() + 30 + while not llm.reset_prefix_cache(reset_connector=False): + assert time.monotonic() < deadline, "CPU offload did not drain" + llm.generate("0", sampling_params, use_tqdm=False) + loaded_before = loaded_blocks(llm) + restored = llm.generate(prompt, sampling_params, use_tqdm=False)[0] + loaded = loaded_blocks(llm) - loaded_before + assert loaded > 0 + assert restored.outputs[0].token_ids == cold.outputs[0].token_ids + print(f"KV offload blocks restored={loaded}; tokens match") + finally: + del llm + + @pytest.mark.optional @pytest.mark.slow_test @pytest.mark.parametrize("model", SMALL_MODELS) diff --git a/vllm/config/parallel.py b/vllm/config/parallel.py index f51a35c5c4a..ca3daa16313 100644 --- a/vllm/config/parallel.py +++ b/vllm/config/parallel.py @@ -326,16 +326,32 @@ class ParallelConfig: nnodes: int = Field(default=1, ge=1) """num of nodes for multi-node distributed inference when distributed_executor_backend is mp.""" - numa_bind: bool = False + numa_bind: bool | None = None """Enable NUMA binding for GPU worker subprocesses. - By default, workers are pinned to their GPU's NUMA-local CPUs and - memory; on PCT-capable Xeons they also auto-bind to the SKU's - PCT priority cores. + Workers are pinned to their GPU's NUMA-local CPUs; on PCT-capable + Xeons they also auto-bind to the SKU's PCT priority cores. + + If None (default), vLLM binds only when the environment supports it + (multiple NUMA nodes, `numactl` installed, unrestricted CPU affinity, + a default NUMA memory policy, and the `spawn` start method with an + mp/uni executor and mp DP, including PCP). + Ray, external launchers and device-ID remapping are excluded + from automatic binding. Automatic mode prefers GPU-local memory but + permits allocations on other nodes. Unsupported environments start + unbound. With `True`, strict memory binding may override an existing + policy, and a missing `numactl` or undetectable GPU-to-NUMA topology is + a startup error; `False` disables binding. """ + _numa_bind_auto: bool = Field(default=False, repr=False) + """Internal launch decision: use soft memory preference in spawned children.""" numa_bind_nodes: list[int] | None = None """NUMA node to bind each GPU worker to. + Setting this list implies explicit `numa_bind=True` and requests strict + `--membind`, not automatic soft memory preference. It cannot be combined + with `numa_bind=False`. + Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]` for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the @@ -345,6 +361,10 @@ class ParallelConfig: numa_bind_cpus: list[str] | None = None """Optional CPU lists to bind each GPU worker to. + Setting this list implies explicit `numa_bind=True` and requests strict + `--membind`, not CPU-only binding or automatic soft memory preference. + It cannot be combined with `numa_bind=False`. + Specify one CPU list per visible GPU, for example `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses `numactl --physcpubind` instead of `--cpunodebind`. This is useful @@ -553,12 +573,12 @@ class ParallelConfig: "data_parallel_external_lb can only be set when data_parallel_size > 1" ) - if not self.numa_bind and ( - self.numa_bind_nodes is not None or self.numa_bind_cpus is not None - ): - raise ValueError( - "numa_bind_nodes and numa_bind_cpus require numa_bind=True." - ) + if self.numa_bind_nodes is not None or self.numa_bind_cpus is not None: + if self.numa_bind is False: + raise ValueError( + "NUMA binding lists cannot be combined with numa_bind=False." + ) + self.numa_bind = True if self.enable_eplb: if not current_platform.is_cuda_alike() and not current_platform.is_xpu(): @@ -949,6 +969,7 @@ class ParallelConfig: # NUMA node in numa_bind_nodes (see vllm/utils/numa_utils.py # `_get_numa_node`), which would otherwise diverge the DP hash. "numa_bind", + "_numa_bind_auto", "numa_bind_nodes", "numa_bind_cpus", "assigned_physical_gpu_ids", diff --git a/vllm/engine/arg_utils.py b/vllm/engine/arg_utils.py index 423544a2a13..316881e5d61 100644 --- a/vllm/engine/arg_utils.py +++ b/vllm/engine/arg_utils.py @@ -541,7 +541,7 @@ class EngineArgs: cpu_distributed_timeout_seconds: int | None = ( ParallelConfig.cpu_distributed_timeout_seconds ) - numa_bind: bool = ParallelConfig.numa_bind + numa_bind: bool | None = ParallelConfig.numa_bind numa_bind_nodes: list[int] | None = ParallelConfig.numa_bind_nodes numa_bind_cpus: list[str] | None = ParallelConfig.numa_bind_cpus device_ids: list[int | str] | None = None diff --git a/vllm/utils/numa_utils.py b/vllm/utils/numa_utils.py index 755d46e711f..fefa30ea9ca 100644 --- a/vllm/utils/numa_utils.py +++ b/vllm/utils/numa_utils.py @@ -26,6 +26,7 @@ if TYPE_CHECKING: logger = logging.getLogger(__name__) _NUMACTL_ARGS_ENV = "_VLLM_INTERNAL_NUMACTL_ARGS" _NUMACTL_PYTHON_EXECUTABLE_ENV = "_VLLM_INTERNAL_NUMACTL_PYTHON_EXECUTABLE" +_MPOL_DEFAULT = 0 @cache @@ -41,55 +42,84 @@ def get_libnuma(): return libnuma -def _can_set_mempolicy() -> bool: - """Check whether the current process can use NUMA memory policy syscalls.""" +def _get_mempolicy() -> int | None: + """Return the current memory-policy mode, or None if it cannot be read.""" try: libnuma = get_libnuma() if libnuma is None or libnuma.numa_available() < 0: - return False + return None mode = ctypes.c_int() ret = libnuma.get_mempolicy( ctypes.byref(mode), None, ctypes.c_ulong(0), None, ctypes.c_ulong(0) ) - return ret == 0 + return mode.value if ret == 0 else None except Exception: - return False + return None -def _is_auto_numa_available() -> bool: - """Check whether automatic GPU-to-NUMA detection should be attempted.""" +def _is_auto_numa_available( + verbose: bool = True, *, require_unbound_process: bool = False +) -> bool: + """Check whether automatic GPU-to-NUMA detection should be attempted. + + Args: + verbose: Log why detection is skipped at warning level; otherwise at + debug level, for the default mode where skipping is expected. + require_unbound_process: Require known unrestricted CPU affinity and + the default memory policy for default automatic binding. Explicit + topology discovery keeps its existing best-effort checks. + + """ from vllm.platforms import current_platform + log = logger.warning if verbose else logger.debug + if not current_platform.is_cuda_alike(): + log("Automatic NUMA binding requires a CUDA or ROCm platform.") return False if not os.path.isdir("/sys/devices/system/node/node1"): + log("Automatic NUMA binding requires multiple NUMA nodes.") return False try: process = psutil.Process(os.getpid()) cpu_affinity = process.cpu_affinity() cpu_count = psutil.cpu_count() - if cpu_count is not None and cpu_affinity != list(range(cpu_count)): - logger.warning( + if cpu_count is not None and set(cpu_affinity) != set(range(cpu_count)): + log( "CPU affinity is already constrained for this process. " "Skipping automatic NUMA binding; pass --numa-bind-nodes " "explicitly to override." ) return False + if cpu_count is None and require_unbound_process: + log("Could not determine CPU availability for automatic NUMA binding.") + return False except (AttributeError, NotImplementedError, psutil.Error): - pass + if require_unbound_process: + log("Could not check CPU affinity for automatic NUMA binding.") + return False - if not _can_set_mempolicy(): - logger.warning( - "User lacks permission to set NUMA memory policy. " + mempolicy = _get_mempolicy() + if mempolicy is None: + log( + "Could not read NUMA memory policy. " "Automatic NUMA detection may not work; if you are using Docker, " "try adding --cap-add SYS_NICE." ) return False + if require_unbound_process and mempolicy != _MPOL_DEFAULT: + log( + "NUMA memory policy is already set for this process (mode=%d). " + "Skipping automatic NUMA binding; pass --numa-bind explicitly " + "to override.", + mempolicy, + ) + return False if not hasattr(current_platform, "get_all_device_numa_nodes"): - logger.warning( + log( "Platform %s does not support automatic NUMA detection", type(current_platform).__name__, ) @@ -98,7 +128,6 @@ def _is_auto_numa_available() -> bool: return True -@cache def get_auto_numa_nodes() -> list[int] | None: """Auto-detect NUMA nodes for all visible GPUs.""" from vllm.platforms import current_platform @@ -112,6 +141,76 @@ def get_auto_numa_nodes() -> list[int] | None: return numa_nodes +def _can_auto_bind( + parallel_config, local_rank: int, dp_local_rank: int | None = None +) -> bool: + """Validate and store this launcher's automatic GPU-to-NUMA topology. + + Unlike ``--numa-bind``, this never fails startup or forces the ``spawn`` + start method: any unmet requirement means starting unbound. + """ + from shutil import which + + if envs.VLLM_WORKER_MULTIPROC_METHOD != "spawn": + logger.debug("Automatic NUMA binding requires the spawn start method.") + return False + if ( + parallel_config.distributed_executor_backend not in ("mp", "uni") + or parallel_config.data_parallel_backend != "mp" + ): + logger.debug("Automatic NUMA binding requires local mp/uni launchers.") + return False + # Topology is indexed in visible-device order, which --device-ids remaps. + if parallel_config.assigned_physical_gpu_ids is not None: + logger.debug("Automatic NUMA binding does not support --device-ids remapping.") + return False + if which("numactl") is None: + logger.debug("Automatic NUMA binding requires numactl on PATH.") + return False + try: + from vllm.platforms import current_platform + from vllm.utils.cpu_resource_utils import parse_id_list + + if not _is_auto_numa_available(verbose=False, require_unbound_process=True): + return False + numa_nodes = get_auto_numa_nodes() + if not numa_nodes: + logger.debug("No GPU-to-NUMA topology available for automatic binding.") + return False + if len(numa_nodes) != current_platform.device_count() or any( + not isinstance(node, int) or node < 0 for node in numa_nodes + ): + logger.debug("Incomplete or invalid GPU-to-NUMA topology: %s", numa_nodes) + return False + + last_local_rank = max(local_rank, parallel_config.local_world_size - 1) + last_dp_local_rank = (parallel_config.data_parallel_rank_local or 0) + max( + parallel_config.data_parallel_size_local - 1, 0 + ) + last_gpu_index = max( + parallel_config.get_local_gpu_index(last_local_rank, dp_local_rank), + parallel_config.get_local_gpu_index(last_local_rank, last_dp_local_rank), + ) + if last_gpu_index >= len(numa_nodes): + logger.debug( + "GPU-to-NUMA topology does not cover all local workers: " + "GPU index %d, mapping size %d.", + last_gpu_index, + len(numa_nodes), + ) + return False + for node in set(numa_nodes): + cpulist_path = Path(f"/sys/devices/system/node/node{node}/cpulist") + if not parse_id_list(cpulist_path.read_text().strip()): + logger.debug("NUMA node %d has no CPUs for automatic binding.", node) + return False + parallel_config.numa_bind_nodes = list(numa_nodes) + return True + except Exception: + logger.debug("Automatic GPU-to-NUMA topology validation failed", exc_info=True) + return False + + # PCT (Priority Core Turbo) auto-detection workaround for Granite Rapids # Xeon SKUs. # @@ -326,24 +425,13 @@ def _get_numactl_worker_args( gpu_index = parallel_config.get_local_gpu_index(local_rank, dp_local_rank) numa_node = _get_numa_node(parallel_config, gpu_index) cpu_binding = _get_cpu_binding(parallel_config, gpu_index, [numa_node]) + memory_policy = "preferred" if parallel_config._numa_bind_auto else "membind" + memory_arg = f"--{memory_policy}={numa_node}" if cpu_binding is not None: - logger.info( - "Binding worker subprocess (local_rank=%s, gpu_index=%s) to CPUs %s and NUMA node %s", # noqa: E501 - local_rank, - gpu_index, - cpu_binding, - numa_node, - ) - return f"--physcpubind={cpu_binding} --membind={numa_node}" + return f"--physcpubind={cpu_binding} {memory_arg}" - logger.info( - "Binding worker subprocess (local_rank=%s, gpu_index=%s) to NUMA node %s", - local_rank, - gpu_index, - numa_node, - ) - return f"--cpunodebind={numa_node} --membind={numa_node}" + return f"--cpunodebind={numa_node} {memory_arg}" def _get_enginecore_numa_nodes( @@ -387,7 +475,11 @@ def _get_numactl_enginecore_args( is also a safe superset by construction). """ shard_nodes = _get_enginecore_numa_nodes(parallel_config, dp_local_rank) - membind_arg = ",".join(str(n) for n in shard_nodes) + node_binding = ",".join(str(n) for n in shard_nodes) + memory_policy = "membind" + if parallel_config._numa_bind_auto: + memory_policy = "preferred" if len(shard_nodes) == 1 else "preferred-many" + memory_arg = f"--{memory_policy}={node_binding}" pct_cpus = ( None @@ -397,21 +489,9 @@ def _get_numactl_enginecore_args( if pct_cpus is not None: cpu_binding = ",".join(str(c) for c in pct_cpus) - logger.info( - "Binding EngineCore subprocess (local_rank=%s) to CPUs %s " - "and NUMA nodes %s", - local_rank, - cpu_binding, - membind_arg, - ) - return f"--physcpubind={cpu_binding} --membind={membind_arg}" + return f"--physcpubind={cpu_binding} {memory_arg}" - logger.info( - "Binding EngineCore subprocess (local_rank=%s) to NUMA nodes %s", - local_rank, - membind_arg, - ) - return f"--cpunodebind={membind_arg} --membind={membind_arg}" + return f"--cpunodebind={node_binding} {memory_arg}" def _log_numactl_show(label: str) -> bool: @@ -455,20 +535,22 @@ def _probe_numactl_args(numactl_args: str) -> bool: def _resolve_numactl_args(numactl_args: str) -> str: - """Drop ``--membind`` if the container rejects it, keeping CPU binding.""" + """Drop a rejected memory policy, keeping CPU binding when possible.""" cpu_only = " ".join( - t for t in numactl_args.split() if not t.startswith("--membind=") + t + for t in numactl_args.split() + if not t.startswith(("--membind=", "--preferred=", "--preferred-many=")) ) for candidate in (numactl_args, cpu_only, ""): if _probe_numactl_args(candidate): if candidate != numactl_args: logger.warning( - "numactl args %r rejected; falling back to %r. Add " - "--cap-add SYS_NICE for full NUMA binding.", + "numactl args %r rejected; falling back to %r.", numactl_args, candidate or "no binding", ) return candidate + logger.warning("numactl probes failed; starting without NUMA binding.") return "" @@ -481,6 +563,19 @@ def configure_subprocess( ): """Temporarily replace the multiprocessing executable with a numactl wrapper.""" parallel_config = vllm_config.parallel_config + if parallel_config.numa_bind is None: + # Resolve once in the launching process; spawned children inherit the + # result through the pickled config instead of re-checking (a bound + # EngineCore would fail the unconstrained-affinity check). + parallel_config.numa_bind = _can_auto_bind( + parallel_config, local_rank, dp_local_rank + ) + parallel_config._numa_bind_auto = parallel_config.numa_bind + if not parallel_config.numa_bind: + logger.info( + "Automatic NUMA binding unavailable on this host; starting " + "unbound. Enable debug logging for the skip reason." + ) if not parallel_config.numa_bind: yield return @@ -502,8 +597,20 @@ def configure_subprocess( numactl_args = _resolve_numactl_args(numactl_args) if not numactl_args: # No NUMA binding possible here; launch without the wrapper. + if parallel_config._numa_bind_auto: + parallel_config.numa_bind = False + parallel_config.numa_bind_nodes = None + parallel_config._numa_bind_auto = False yield return + if envs.VLLM_WORKER_MULTIPROC_METHOD == "spawn": + logger.info( + "Launching %s subprocess (local_rank=%s, dp_local_rank=%s) with numactl %s", + process_kind, + local_rank, + dp_local_rank, + numactl_args, + ) python_executable = os.fsdecode(multiprocessing.spawn.get_executable()) with ( _set_numa_wrapper_env(numactl_args, python_executable), diff --git a/vllm/utils/system_utils.py b/vllm/utils/system_utils.py index 5ae13232b06..cbea3c862d2 100644 --- a/vllm/utils/system_utils.py +++ b/vllm/utils/system_utils.py @@ -137,9 +137,12 @@ def _maybe_force_spawn(): os.environ["RAY_ADDRESS"] = ray.get_runtime_context().gcs_address reasons.append("In a Ray actor and can only be spawned") - # Force spawn if NUMA binding is enabled via --numa-bind. - # NUMA binding uses executable hijacking which requires spawn - if "--numa-bind" in sys.argv: + # NUMA binding uses executable hijacking, which requires spawn. + if any( + arg.split("=", 1)[0].replace("_", "-") + in ("--numa-bind", "--numa-bind-nodes", "--numa-bind-cpus") + for arg in sys.argv + ): reasons.append("NUMA binding requires spawn method") if cuda_is_initialized():