[Feature][Core] Enable best-effort NUMA binding by default (#61258)

docs/configuration/optimization.md

@@ -156,10 +156,47 @@ GPU. vLLM can pin each worker with `numactl` before the Python subprocess starts
 so the interpreter, imports, and early allocator state are created with the
 desired NUMA policy from the beginning.
 
-Use `--numa-bind` to enable the feature. By default, vLLM auto-detects the
-GPU-to-NUMA mapping and uses `--cpunodebind=<node> --membind=<node>` for each
-worker. When you need a custom CPU policy, add `--numa-bind-cpus` and vLLM will
-switch to `--physcpubind=<cpu-list> --membind=<node>`.
+By default, vLLM binds automatically when the environment supports it:
+CUDA-alike platform, multiple NUMA nodes, `numactl` installed, permission to
+set NUMA memory policy, unrestricted CPU affinity, a default NUMA memory policy,
+the `spawn` multiprocessing start method (the CLI default), and a local `mp` or
+`uni` executor with multiprocessing DP, including supported prefill context
+parallelism (PCP) launches. Ray, external launchers and `--device-ids` remapping
+are excluded from automatic binding.
+Otherwise vLLM starts unbound. Use
+`--numa-bind` to make a missing `numactl` or undetectable GPU-to-NUMA topology a
+startup error, or `--no-numa-bind` to disable binding. Passing
+`--numa-bind-nodes` or `--numa-bind-cpus` explicitly enables binding even without
+`--numa-bind`: either list requests strict `--membind`, not automatic soft memory
+preference or CPU-only binding. Neither can be combined with `--no-numa-bind`.
+Automatic binding preserves an existing non-default memory policy, such as
+interleave, preferred-node, or bind. Explicit enablement may override it.
+
+Automatic binding uses `--cpunodebind=<node> --preferred=<node>` for each worker:
+memory allocations prefer the GPU-local node but can fall back to other nodes.
+EngineCore prefers its shard's nodes with `--preferred-many=<nodes>` when the
+shard spans multiple NUMA nodes, or `--preferred=<node>` for a single-node shard.
+Explicit `--numa-bind` or binding lists retain strict `--membind=<nodes>`.
+When you need a custom CPU policy, add `--numa-bind-cpus` and vLLM will switch
+to `--physcpubind=<cpu-list>`.
+
+Automatic eligibility and topology validation happen in each local launcher
+before starting its first bound subprocess, using freshly discovered topology.
+The mapping must cover all visible GPUs and local workers, and each mapped NUMA
+node must have CPUs. The resolved decision and mapping are passed to spawned
+children, so an already-bound EngineCore does not repeat detection under its
+restricted CPU affinity.
+PCP workers use the full TP × PCP × PP shard size for local DP GPU offsets;
+EngineCore's CPU affinity covers all workers in its shard.
+Multi-node multiprocessing launchers resolve their own host-local mapping;
+NUMA node IDs are never interpreted as cluster-wide IDs.
+
+Before each bound launch, vLLM probes the selected policy with
+`numactl <args> true`. If CPU-and-memory binding is rejected, it retries with
+CPU-only binding; if that is also rejected, the subprocess starts unbound.
+Logs report the selected policy or fallback. If `--preferred-many` is unsupported,
+automatic mode falls back to CPU-only binding, never strict memory binding.
+These probes test policy acceptance, not future allocation capacity.
 
 These `--numa-bind*` options only apply to GPU execution processes. They do not
 configure the CPU backend's separate thread-affinity controls. Automatic
@@ -174,7 +211,7 @@ order as the GPU indices. Each CPU list must use
 `numactl --physcpubind` syntax such as `0-3`, `0,2,4-7`, or `16-31,48-63`.
 
 ```bash
-# Auto-detect NUMA nodes for visible GPUs
+# Require binding, auto-detecting NUMA nodes for visible GPUs
 vllm serve meta-llama/Llama-3.1-8B-Instruct \
   --tensor-parallel-size 4 \
   --numa-bind
@@ -195,10 +232,12 @@ vllm serve meta-llama/Llama-3.1-8B-Instruct \
 
 Notes:
 
-- CLI usage forces multiprocessing to use the `spawn` method automatically. If you enable NUMA binding through the Python API, also set `VLLM_WORKER_MULTIPROC_METHOD=spawn`.
-- Automatic detection relies on NVML and NUMA support from the host. If it cannot determine the mapping reliably, pass `--numa-bind-nodes` explicitly.
+- The CLI defaults to `spawn` when `VLLM_WORKER_MULTIPROC_METHOD` is unset. Automatic NUMA binding never changes the start method: if it is `fork`, default binding stays off. Python API usage normally defaults to `fork`; set `VLLM_WORKER_MULTIPROC_METHOD=spawn` to allow binding and follow Python's `if __name__ == "__main__":` requirements.
+- Automatic memory preference permits allocations on other NUMA nodes when local memory is unavailable. Explicit `--numa-bind` still uses `--membind`, which restricts host allocations to the selected nodes; for host-memory-heavy setups (e.g. large CPU KV offload), avoid explicit strict binding if workers need memory from other nodes. The policy probes do not check allocation capacity or container memory limits.
+- Automatic detection relies on NVML (CUDA) or amdsmi (ROCm) and NUMA support from the host. If it cannot determine the mapping reliably, default mode starts unbound; pass `--numa-bind-nodes` to provide a mapping explicitly.
 - Explicit `--numa-bind-nodes` and `--numa-bind-cpus` values must be valid `numactl` inputs. vLLM does a small amount of validation, but the effective binding semantics are still determined by `numactl`.
 - The current implementation binds GPU execution processes such as `EngineCore` and multiprocessing workers. It does not apply NUMA binding to frontend API server processes or the DP coordinator.
+- Ray actors are not bound by the multiprocessing executable wrapper. Enabling automatic NUMA binding for Ray requires a separate actor-launch implementation.
 - In containerized environments, NUMA policy syscalls may require extra permissions, such as `--cap-add SYS_NICE` when running via `docker run`.
 
 ### CPU Backend Thread Affinity

tests/engine/test_arg_utils.py

@@ -804,6 +804,11 @@ def test_numa_bind_args():
     assert engine_args.numa_bind_nodes == [0, 0, 1, 1]
     assert engine_args.numa_bind_cpus == ["0-3", "4-7", "8-11", "12-15"]
 
+    # Unset means auto; --no-numa-bind disables explicitly.
+    assert EngineArgs.from_cli_args(args=parser.parse_args([])).numa_bind is None
+    args = parser.parse_args(["--no-numa-bind"])
+    assert EngineArgs.from_cli_args(args=args).numa_bind is False
+
 
 def test_ir_op_priority():
     from vllm.config.kernel import IrOpPriorityConfig, KernelConfig

tests/utils_/test_numa_utils.py

@@ -2,8 +2,10 @@
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
 
 import os
+from copy import deepcopy
 from functools import partial
 from types import SimpleNamespace
+from typing import Any
 
 import pytest
 
@@ -39,20 +41,30 @@ def _disable_pct_by_default(monkeypatch):
 
 
 def _make_config(**parallel_kwargs):
-    parallel_defaults = dict(
+    parallel_defaults: dict[str, Any] = dict(
         numa_bind=False,
+        _numa_bind_auto=False,
         numa_bind_nodes=None,
         numa_bind_cpus=None,
+        assigned_physical_gpu_ids=None,
         distributed_executor_backend="mp",
         data_parallel_backend="mp",
         nnodes_within_dp=1,
         data_parallel_rank_local=0,
         data_parallel_index=0,
+        data_parallel_size_local=1,
         pipeline_parallel_size=1,
         tensor_parallel_size=1,
         prefill_context_parallel_size=1,
     )
     parallel_defaults.update(parallel_kwargs)
+    parallel_defaults.setdefault(
+        "local_world_size",
+        parallel_defaults["pipeline_parallel_size"]
+        * parallel_defaults["tensor_parallel_size"]
+        * parallel_defaults["prefill_context_parallel_size"]
+        // parallel_defaults["nnodes_within_dp"],
+    )
     parallel_config = SimpleNamespace(**parallel_defaults)
     parallel_config.world_size = (
         parallel_config.tensor_parallel_size
@@ -214,12 +226,18 @@ def test_pct_binding_returns_none_when_node_cpulist_missing(monkeypatch):
     assert numa_utils._maybe_get_pct_cpu_binding([0]) is None
 
 
-def test_get_numactl_args_uses_pct_when_user_did_not_specify_cpus(monkeypatch):
+@pytest.mark.parametrize("automatic", [False, True])
+def test_get_numactl_args_uses_pct_when_user_did_not_specify_cpus(
+    monkeypatch, automatic
+):
     _patch_pct_gates(monkeypatch, model_match=True, highest_perf=46)
-    vllm_config = _make_config(numa_bind=True, numa_bind_nodes=[0, 1])
+    vllm_config = _make_config(
+        numa_bind=True, _numa_bind_auto=automatic, numa_bind_nodes=[0, 1]
+    )
+    memory_policy = "preferred" if automatic else "membind"
     assert (
         numa_utils._get_numactl_worker_args(vllm_config.parallel_config, local_rank=1)
-        == "--physcpubind=0,1,16,17,64,65,80,81 --membind=1"
+        == f"--physcpubind=0,1,16,17,64,65,80,81 --{memory_policy}=1"
     )
 
 
@@ -249,7 +267,10 @@ def test_get_numactl_args_engine_core_baseline_spans_shard_numa_nodes():
     )
 
 
-def test_get_numactl_args_engine_core_pct_spans_shard_numa_nodes(monkeypatch):
+@pytest.mark.parametrize("automatic", [False, True])
+def test_get_numactl_args_engine_core_pct_spans_shard_numa_nodes(
+    monkeypatch, automatic
+):
     """PCT: EngineCore for a multi-NUMA shard binds to the union of priority
     cores across all shard nodes, so worker `--physcpubind` is always a
     subset of EngineCore's `cpus_allowed`."""
@@ -261,15 +282,17 @@ def test_get_numactl_args_engine_core_pct_spans_shard_numa_nodes(monkeypatch):
     )
     vllm_config = _make_config(
         numa_bind=True,
+        _numa_bind_auto=automatic,
         numa_bind_nodes=[0, 0, 1, 1],
         tensor_parallel_size=4,
     )
+    memory_policy = "preferred-many" if automatic else "membind"
     assert numa_utils._get_numactl_enginecore_args(
         vllm_config.parallel_config, local_rank=0
     ) == (
         "--physcpubind="
         "0,1,16,17,64,65,80,81,128,129,144,145,192,193,208,209"
-        " --membind=0,1"
+        f" --{memory_policy}=0,1"
     )
 
 
@@ -543,6 +566,18 @@ def test_parallel_config_validates_numa_bind_nodes():
         ParallelConfig(numa_bind_nodes=[0, -1])
 
 
+@pytest.mark.parametrize(
+    ("nodes", "cpus"), [([0, 1], None), (None, ["0-3"])], ids=["nodes", "cpus"]
+)
+def test_parallel_config_numa_lists_imply_numa_bind(nodes, cpus):
+    assert ParallelConfig().numa_bind is None
+    config = ParallelConfig(numa_bind_nodes=nodes, numa_bind_cpus=cpus)
+    assert config.numa_bind is True
+    assert config._numa_bind_auto is False
+    with pytest.raises(ValueError, match="cannot be combined with numa_bind=False"):
+        ParallelConfig(numa_bind=False, numa_bind_nodes=nodes, numa_bind_cpus=cpus)
+
+
 @pytest.mark.parametrize("cpuset", ["", "abc", "1-", "4-1", "1,,2", "1:2"])
 def test_parallel_config_rejects_invalid_numa_bind_cpus(cpuset):
     with pytest.raises(ValueError, match="numa_bind_cpus"):
@@ -564,7 +599,7 @@ def test_configure_subprocess_numa_fallback(monkeypatch):
     import multiprocessing
 
     monkeypatch.setattr("shutil.which", lambda name: "/usr/bin/numactl")
-    monkeypatch.setattr(numa_utils.envs, "VLLM_WORKER_MULTIPROC_METHOD", "spawn")
+    monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn")
     node_config = _make_config(numa_bind=True, numa_bind_nodes=[0])
 
     monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([]))
@@ -593,3 +628,409 @@ def test_configure_subprocess_numa_fallback(monkeypatch):
     with numa_utils.configure_subprocess(node_config, local_rank=0):
         assert multiprocessing.spawn.get_executable() == before
         assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+
+
+def _patch_auto_numa(
+    monkeypatch, start_method, detect, visible_gpus=1, node_cpus="0-7"
+):
+    monkeypatch.setattr("shutil.which", lambda name: "/usr/bin/numactl")
+    monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", start_method)
+    monkeypatch.setattr(numa_utils, "_is_auto_numa_available", lambda **kwargs: True)
+    monkeypatch.setattr(numa_utils, "get_auto_numa_nodes", detect)
+    monkeypatch.setattr(
+        "vllm.platforms.current_platform.device_count", lambda: visible_gpus
+    )
+    real_read_text = numa_utils.Path.read_text
+
+    def read_text(path, *args, **kwargs):
+        if str(path).startswith("/sys/devices/system/node/node"):
+            return node_cpus
+        return real_read_text(path, *args, **kwargs)
+
+    monkeypatch.setattr(numa_utils.Path, "read_text", read_text)
+
+
+@pytest.mark.parametrize("nodes", [[0], [0, 1]])
+def test_configure_subprocess_auto_binds_and_records_result(monkeypatch, nodes):
+    """Default mode resolves once in the launcher and stores the result, so
+    spawned children bind with it instead of re-checking eligibility."""
+    _patch_auto_numa(monkeypatch, "spawn", lambda: nodes, visible_gpus=len(nodes))
+    monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([]))
+
+    vllm_config = _make_config(numa_bind=None, tensor_parallel_size=len(nodes))
+    node_list = ",".join(map(str, nodes))
+    preference = "preferred" if len(nodes) == 1 else "preferred-many"
+    with numa_utils.configure_subprocess(
+        vllm_config, local_rank=0, process_kind="EngineCore"
+    ):
+        assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == (
+            f"--cpunodebind={node_list} --{preference}={node_list}"
+        )
+        child_config = deepcopy(vllm_config)
+    assert vllm_config.parallel_config.numa_bind is True
+    assert child_config.parallel_config._numa_bind_auto is True
+    assert child_config.parallel_config.numa_bind_nodes == nodes
+
+    monkeypatch.setattr(numa_utils, "get_auto_numa_nodes", _raise_probe_error)
+    monkeypatch.setattr(numa_utils, "_is_auto_numa_available", lambda **kwargs: False)
+    with numa_utils.configure_subprocess(child_config, local_rank=len(nodes) - 1):
+        assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == (
+            f"--cpunodebind={nodes[-1]} --preferred={nodes[-1]}"
+        )
+
+
+def test_parallel_config_auto_policy_survives_copy_without_changing_hash():
+    """Copying the launch decision must not change the DP configuration hash."""
+    from vllm.config.utils import replace
+
+    parallel_config = ParallelConfig()
+    original_hash = parallel_config.compute_hash()
+    parallel_config.numa_bind = True
+    parallel_config.numa_bind_nodes = [0]
+    parallel_config._numa_bind_auto = True
+    for copied in (
+        replace(parallel_config),
+        deepcopy(parallel_config),
+    ):
+        assert copied._numa_bind_auto is True
+        assert copied.compute_hash() == original_hash
+
+
+@pytest.mark.parametrize("dp_local_rank", [0, 1])
+def test_auto_numa_accepts_pcp_dp_topology(monkeypatch, dp_local_rank):
+    """Automatic eligibility must cover every PCP worker in both DP shards."""
+    _patch_auto_numa(monkeypatch, "spawn", lambda: [0, 0, 1, 1], visible_gpus=4)
+    config = ParallelConfig(
+        distributed_executor_backend="mp",
+        prefill_context_parallel_size=2,
+        data_parallel_size=2,
+        data_parallel_size_local=2,
+    )
+    assert numa_utils._can_auto_bind(config, local_rank=0, dp_local_rank=dp_local_rank)
+    assert config.numa_bind_nodes == [0, 0, 1, 1]
+
+
+def test_auto_numa_redetects_when_visible_devices_change(monkeypatch):
+    """Independent launchers must not reuse another visible GPU set's mapping."""
+    _patch_auto_numa(
+        monkeypatch, "spawn", numa_utils.get_auto_numa_nodes, visible_gpus=2
+    )
+    monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([]))
+    detections = []
+
+    def detect():
+        devices = os.environ["CUDA_VISIBLE_DEVICES"]
+        detections.append(devices)
+        return {"0,1": [0, 0], "2,3": [1, 1]}[devices]
+
+    monkeypatch.setattr(
+        "vllm.platforms.current_platform.get_all_device_numa_nodes",
+        detect,
+        raising=False,
+    )
+    for devices, node in [("0,1", 0), ("2,3", 1)]:
+        monkeypatch.setenv("CUDA_VISIBLE_DEVICES", devices)
+        vllm_config = _make_config(numa_bind=None, tensor_parallel_size=2)
+        with numa_utils.configure_subprocess(
+            vllm_config, local_rank=0, process_kind="EngineCore"
+        ):
+            numactl_args = os.environ[numa_utils._NUMACTL_ARGS_ENV]
+            assert numactl_args == f"--cpunodebind={node} --preferred={node}"
+        assert vllm_config.parallel_config.numa_bind_nodes == [node, node]
+    assert detections == ["0,1", "2,3"]
+
+
+@pytest.mark.parametrize("explicit", [False, True], ids=["automatic", "explicit"])
+@pytest.mark.parametrize(
+    "policy",
+    [0, 1, 2, 3, 4, 5, None],
+    ids=[
+        "default",
+        "preferred",
+        "bind",
+        "interleave",
+        "local",
+        "preferred-many",
+        "unreadable",
+    ],
+)
+def test_configure_subprocess_respects_existing_mempolicy(
+    monkeypatch, policy, explicit
+):
+    """Only explicit enablement may replace an existing non-default policy."""
+    is_auto_numa_available = numa_utils._is_auto_numa_available
+    _patch_auto_numa(monkeypatch, "spawn", numa_utils.get_auto_numa_nodes)
+    monkeypatch.setattr(numa_utils, "_is_auto_numa_available", is_auto_numa_available)
+    monkeypatch.setattr("vllm.platforms.current_platform.is_cuda_alike", lambda: True)
+    monkeypatch.setattr(numa_utils.os.path, "isdir", lambda path: True)
+    monkeypatch.setattr(numa_utils.psutil, "cpu_count", lambda: 4)
+    monkeypatch.setattr(
+        numa_utils.psutil,
+        "Process",
+        lambda pid: SimpleNamespace(cpu_affinity=lambda: [0, 1, 2, 3]),
+    )
+    monkeypatch.setattr(
+        "vllm.platforms.current_platform.get_all_device_numa_nodes",
+        lambda: [0],
+        raising=False,
+    )
+
+    def get_mempolicy(mode, *_args):
+        if policy is None:
+            return -1
+        mode._obj.value = policy
+        return 0
+
+    monkeypatch.setattr(
+        numa_utils,
+        "get_libnuma",
+        lambda: SimpleNamespace(numa_available=lambda: 0, get_mempolicy=get_mempolicy),
+    )
+    monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([]))
+    vllm_config = _make_config(numa_bind=True if explicit else None)
+    if explicit and policy is None:
+        with (
+            pytest.raises(RuntimeError, match="could not detect"),
+            numa_utils.configure_subprocess(vllm_config, local_rank=0),
+        ):
+            pass
+        return
+
+    expected_bind = policy is not None and (policy == 0 or explicit)
+    memory_arg = "--membind=0" if explicit else "--preferred=0"
+    with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+        numactl_args = os.environ.get(numa_utils._NUMACTL_ARGS_ENV)
+        assert numactl_args == (
+            f"--cpunodebind=0 {memory_arg}" if expected_bind else None
+        )
+    assert vllm_config.parallel_config.numa_bind is expected_bind
+
+
+def _raise_probe_error():
+    raise RuntimeError("amdsmi init failed")
+
+
+@pytest.mark.parametrize(
+    ("start_method", "detect", "assigned_gpus"),
+    [
+        ("fork", lambda: [0], None),
+        ("spawn", lambda: None, None),
+        ("spawn", lambda: [], None),
+        ("spawn", _raise_probe_error, None),
+        ("spawn", lambda: [0, 1], [1]),
+    ],
+    ids=["fork", "no-topology", "no-visible-gpus", "probe-raises", "device-ids"],
+)
+def test_configure_subprocess_auto_falls_back_to_unbound(
+    monkeypatch, start_method, detect, assigned_gpus
+):
+    """Default mode never fails startup or forces spawn; when binding cannot
+    engage (or would pick the wrong node) it starts unbound."""
+    import multiprocessing
+
+    _patch_auto_numa(monkeypatch, start_method, detect)
+
+    vllm_config = _make_config(numa_bind=None, assigned_physical_gpu_ids=assigned_gpus)
+    before = multiprocessing.spawn.get_executable()
+    with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+        assert multiprocessing.spawn.get_executable() == before
+    assert vllm_config.parallel_config.numa_bind is False
+
+
+@pytest.mark.parametrize(
+    "parallel_kwargs",
+    [
+        {"distributed_executor_backend": "ray"},
+        {"data_parallel_backend": "ray"},
+        {"distributed_executor_backend": "external_launcher"},
+    ],
+    ids=["ray-executor", "ray-dp", "external-launcher"],
+)
+def test_auto_numa_skips_unsupported_launchers(monkeypatch, parallel_kwargs):
+    detections = []
+
+    def detect():
+        detections.append(True)
+        return [0, 1]
+
+    _patch_auto_numa(monkeypatch, "spawn", detect, visible_gpus=2)
+    vllm_config = _make_config(numa_bind=None, **parallel_kwargs)
+    with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+        assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+    assert vllm_config.parallel_config.numa_bind is False
+    assert detections == []
+
+
+@pytest.mark.parametrize("nodes", [[0], [0, 1, 2], [0, -1], [0, None]])
+def test_auto_numa_rejects_incomplete_or_invalid_topology(monkeypatch, nodes):
+    _patch_auto_numa(monkeypatch, "spawn", lambda: nodes, visible_gpus=2)
+    vllm_config = _make_config(numa_bind=None, tensor_parallel_size=2)
+    with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+        assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+    assert vllm_config.parallel_config.numa_bind is False
+    assert vllm_config.parallel_config.numa_bind_nodes is None
+
+
+def test_auto_numa_checks_all_local_dp_workers_before_first_engine(monkeypatch):
+    _patch_auto_numa(monkeypatch, "spawn", lambda: [0, 1], visible_gpus=2)
+    vllm_config = _make_config(
+        numa_bind=None, tensor_parallel_size=2, data_parallel_size_local=2
+    )
+    with numa_utils.configure_subprocess(
+        vllm_config, local_rank=0, dp_local_rank=0, process_kind="EngineCore"
+    ):
+        assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+    assert vllm_config.parallel_config.numa_bind is False
+
+
+def test_auto_numa_multi_node_topology_is_local(monkeypatch):
+    _patch_auto_numa(monkeypatch, "spawn", lambda: [2, 3], visible_gpus=2)
+    monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([]))
+    vllm_config = _make_config(
+        numa_bind=None, tensor_parallel_size=4, nnodes_within_dp=2
+    )
+    with numa_utils.configure_subprocess(
+        vllm_config, local_rank=0, process_kind="EngineCore"
+    ):
+        assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == (
+            "--cpunodebind=2,3 --preferred-many=2,3"
+        )
+        child_config = deepcopy(vllm_config)
+    with numa_utils.configure_subprocess(child_config, local_rank=1):
+        assert (
+            os.environ[numa_utils._NUMACTL_ARGS_ENV] == "--cpunodebind=3 --preferred=3"
+        )
+
+
+@pytest.mark.parametrize("node_cpus", [None, "", "invalid"])
+def test_auto_numa_skips_nodes_without_usable_cpus(monkeypatch, node_cpus):
+    _patch_auto_numa(monkeypatch, "spawn", lambda: [0], node_cpus=node_cpus)
+    vllm_config = _make_config(numa_bind=None)
+    with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+        assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+    assert vllm_config.parallel_config.numa_bind is False
+
+
+@pytest.mark.parametrize(
+    "nodes, rejected_args, expected_args",
+    [
+        ([0], "--preferred=", "--cpunodebind=0"),
+        ([0], "--cpunodebind=", None),
+        ([0, 1], "--preferred-many=", "--cpunodebind=0,1"),
+        ([0, 1], "--cpunodebind=", None),
+    ],
+)
+def test_auto_numa_records_effective_policy(
+    monkeypatch, caplog, nodes, rejected_args, expected_args
+):
+    caplog.set_level("INFO", logger=numa_utils.logger.name)
+    _patch_auto_numa(monkeypatch, "spawn", lambda: nodes, visible_gpus=len(nodes))
+    probes = []
+    fake_run = _fake_numactl_run([rejected_args])
+
+    def run(cmd, *args, **kwargs):
+        probes.extend(cmd)
+        return fake_run(cmd, *args, **kwargs)
+
+    monkeypatch.setattr(numa_utils.subprocess, "run", run)
+    vllm_config = _make_config(numa_bind=None, tensor_parallel_size=len(nodes))
+    with numa_utils.configure_subprocess(
+        vllm_config, local_rank=0, process_kind="EngineCore"
+    ):
+        assert os.environ.get(numa_utils._NUMACTL_ARGS_ENV) == expected_args
+        assert vllm_config.parallel_config.numa_bind is (expected_args is not None)
+    launch_logs = [r.getMessage() for r in caplog.records if "Launching" in r.message]
+    if expected_args is None:
+        assert launch_logs == []
+        assert vllm_config.parallel_config.numa_bind_nodes is None
+        assert vllm_config.parallel_config._numa_bind_auto is False
+    else:
+        assert len(launch_logs) == 1
+        assert launch_logs[0].endswith(f"numactl {expected_args}")
+    assert not any(arg.startswith("--membind=") for arg in probes)
+
+
+def test_auto_numa_does_not_swallow_launch_errors(monkeypatch):
+    import multiprocessing
+
+    _patch_auto_numa(monkeypatch, "spawn", lambda: [0])
+    monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([]))
+    monkeypatch.delenv(numa_utils._NUMACTL_ARGS_ENV, raising=False)
+    monkeypatch.delenv(numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV, raising=False)
+    original_executable = multiprocessing.spawn.get_executable()
+    with (
+        pytest.raises(RuntimeError, match="worker startup failed"),
+        numa_utils.configure_subprocess(_make_config(numa_bind=None), local_rank=0),
+    ):
+        raise RuntimeError("worker startup failed")
+    assert multiprocessing.spawn.get_executable() == original_executable
+    assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+    assert numa_utils._NUMACTL_PYTHON_EXECUTABLE_ENV not in os.environ
+
+
+@pytest.mark.parametrize("missing_numactl", [False, True])
+def test_auto_numa_skips_unavailable_environment(monkeypatch, missing_numactl):
+    _patch_auto_numa(monkeypatch, "spawn", lambda: [0])
+    if missing_numactl:
+        monkeypatch.setattr("shutil.which", lambda name: None)
+    else:
+        monkeypatch.setattr(
+            numa_utils, "_is_auto_numa_available", lambda **kwargs: False
+        )
+    vllm_config = _make_config(numa_bind=None)
+    with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+        assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+    assert vllm_config.parallel_config.numa_bind is False
+
+
+def test_explicit_numa_nodes_bypass_auto_eligibility(monkeypatch):
+    _patch_auto_numa(monkeypatch, "spawn", _raise_probe_error)
+    monkeypatch.setattr(numa_utils, "_is_auto_numa_available", lambda **kwargs: False)
+    monkeypatch.setattr(numa_utils.subprocess, "run", _fake_numactl_run([]))
+    vllm_config = _make_config(numa_bind=True, numa_bind_nodes=[7])
+    with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+        assert os.environ[numa_utils._NUMACTL_ARGS_ENV] == "--cpunodebind=7 --membind=7"
+
+
+def test_explicit_numa_requires_detectable_topology(monkeypatch):
+    _patch_auto_numa(monkeypatch, "spawn", lambda: None)
+    with (
+        pytest.raises(RuntimeError, match="could not detect"),
+        numa_utils.configure_subprocess(_make_config(numa_bind=True), local_rank=0),
+    ):
+        pass
+
+
+@pytest.mark.parametrize("numa_bind", [False, True])
+def test_explicit_numa_handles_missing_numactl(monkeypatch, numa_bind):
+    monkeypatch.setattr("shutil.which", lambda name: None)
+    vllm_config = _make_config(numa_bind=numa_bind, numa_bind_nodes=[0])
+    if numa_bind:
+        with (
+            pytest.raises(RuntimeError, match="numactl is required"),
+            numa_utils.configure_subprocess(vllm_config, local_rank=0),
+        ):
+            pass
+    else:
+        with numa_utils.configure_subprocess(vllm_config, local_rank=0):
+            assert numa_utils._NUMACTL_ARGS_ENV not in os.environ
+
+
+@pytest.mark.parametrize("affinity", [[0, 1], None, [3, 2, 1, 0]])
+def test_auto_numa_checks_cpu_affinity(monkeypatch, affinity):
+    monkeypatch.setattr("vllm.platforms.current_platform.is_cuda_alike", lambda: True)
+    monkeypatch.setattr(numa_utils.os.path, "isdir", lambda path: True)
+    monkeypatch.setattr(numa_utils.psutil, "cpu_count", lambda: 4)
+    process = SimpleNamespace()
+    if affinity is not None:
+        process.cpu_affinity = lambda: affinity
+    monkeypatch.setattr(numa_utils.psutil, "Process", lambda pid: process)
+    monkeypatch.setattr(numa_utils, "_get_mempolicy", lambda: 0)
+    monkeypatch.setattr(
+        "vllm.platforms.current_platform.get_all_device_numa_nodes",
+        lambda: [0, 1],
+        raising=False,
+    )
+    assert numa_utils._is_auto_numa_available(
+        verbose=False, require_unbound_process=True
+    ) is (affinity == [3, 2, 1, 0])

tests/utils_/test_system_utils.py

@@ -11,6 +11,7 @@ import tempfile
 from pathlib import Path
 
 import psutil
+import pytest
 
 from vllm.utils import system_utils
 from vllm.utils.system_utils import (
@@ -32,11 +33,39 @@ def test_unique_filepath():
     assert len(list(Path(temp_dir).glob("*.txt"))) == 10
 
 
-def test_numa_bind_forces_spawn(monkeypatch):
-    monkeypatch.delenv("VLLM_WORKER_MULTIPROC_METHOD", raising=False)
-    monkeypatch.setattr("sys.argv", ["vllm", "serve", "--numa-bind"])
+@pytest.mark.parametrize("start_method", [None, "fork"])
+@pytest.mark.parametrize(
+    "numa_args, force_spawn",
+    [
+        (["--numa-bind"], True),
+        (["--numa-bind-nodes", "0"], True),
+        (["--numa-bind-nodes=0"], True),
+        (["--numa-bind-cpus", "0-3"], True),
+        (["--numa-bind-cpus=0-3"], True),
+        (["--numa_bind"], True),
+        (["--numa_bind_nodes", "0"], True),
+        (["--numa_bind_cpus=0-3"], True),
+        ([], False),
+        (["--no-numa-bind"], False),
+    ],
+)
+def test_numa_bind_cli_start_method(monkeypatch, start_method, numa_args, force_spawn):
+    if start_method is None:
+        monkeypatch.delenv("VLLM_WORKER_MULTIPROC_METHOD", raising=False)
+    else:
+        monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", start_method)
+    monkeypatch.setattr(sys, "argv", ["vllm", "serve", *numa_args])
+    for check in (
+        "is_in_ray_actor",
+        "cuda_is_initialized",
+        "xpu_is_initialized",
+        "in_wsl",
+    ):
+        monkeypatch.setattr(system_utils, check, lambda: False)
+
     _maybe_force_spawn()
-    assert os.environ["VLLM_WORKER_MULTIPROC_METHOD"] == "spawn"
+    expected_method = "spawn" if force_spawn else start_method
+    assert os.environ.get("VLLM_WORKER_MULTIPROC_METHOD") == expected_method
 
 
 def test_suppress_stdout_when_sys_stdout_has_no_fd(capfd):

tests/v1/distributed/test_dense_dp_world_size.py

@@ -1,17 +1,80 @@
 # SPDX-License-Identifier: Apache-2.0
 # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
 import os
+from typing import TYPE_CHECKING
 
+import pytest
 import requests
 
-from tests.utils import RemoteOpenAIServer
+from tests.utils import VLLM_PATH, RemoteOpenAIServer
 from vllm.platforms import current_platform
 
+if TYPE_CHECKING:
+    import torch
+
+    from vllm.config import ParallelConfig
+
 MODEL_NAME = "Qwen/Qwen3-0.6B"
 DP_SIZE = int(os.getenv("DP_SIZE", "2"))
 TP_SIZE = int(os.getenv("TP_SIZE", "1"))
 
 
+class NumaBindingWorkerExtension:
+    parallel_config: "ParallelConfig"
+    local_rank: int
+    device: "torch.device"
+
+    def check_numa_binding(self):
+        import subprocess
+        from pathlib import Path
+
+        import psutil
+        import torch.distributed as dist
+
+        from vllm.distributed import get_pcp_group, get_world_group
+        from vllm.utils.cpu_resource_utils import parse_id_list
+
+        parallel = self.parallel_config
+        assert parallel.numa_bind and parallel._numa_bind_auto
+        assert parallel.numa_bind_nodes is not None
+        gpu_index = self.local_rank
+        node = current_platform.get_device_numa_node(self.device.index)
+        assert parallel.numa_bind_nodes[gpu_index] == node
+        node_cpus = parse_id_list(
+            Path(f"/sys/devices/system/node/node{node}/cpulist").read_text().strip()
+        )
+        cpus = set(psutil.Process().cpu_affinity())
+        assert cpus and cpus <= set(node_cpus)
+        result = subprocess.run(
+            ["numactl", "--show"],
+            check=True,
+            capture_output=True,
+            text=True,
+            timeout=10,
+        )
+        policy = dict(line.split(":", 1) for line in result.stdout.splitlines())
+        assert policy["policy"].strip() == "preferred"
+        assert policy["preferred node"].strip() == str(node)
+        parent = psutil.Process(os.getppid())
+        parent_cpus = set(parent.cpu_affinity())
+        assert cpus <= parent_cpus
+        state = {
+            "pid": os.getpid(),
+            "gpu_index": gpu_index,
+            "gpu_uuid": current_platform.get_device_uuid(self.device.index),
+            "dp_rank": parallel.data_parallel_rank,
+            "pcp_rank": get_pcp_group().rank_in_group,
+            "node": node,
+            "cpus": sorted(cpus),
+            "engine_core_pid": parent.pid,
+            "engine_core_cpus": sorted(parent_cpus),
+        }
+        world = get_world_group()
+        states = [None] * world.world_size
+        dist.all_gather_object(states, state, group=world.cpu_group)
+        return states
+
+
 def test_dense_dp_world_size():
     server_args = [
         "--dtype",
@@ -50,3 +113,79 @@ def test_dense_dp_world_size():
         )
         response.raise_for_status()
         assert response.json() == {"world_size": TP_SIZE}
+
+
+@pytest.mark.slow_test
+def test_pcp_dp_with_auto_numa():
+    """Automatic binding must cover both PCP workers in both local DP shards."""
+    from shutil import which
+
+    from vllm.utils import numa_utils
+
+    if not current_platform.is_cuda() or current_platform.device_count() < 4:
+        pytest.skip("PCP+DP NUMA smoke test requires four NVIDIA GPUs")
+    if which("numactl") is None or not numa_utils._is_auto_numa_available(
+        verbose=False, require_unbound_process=True
+    ):
+        pytest.skip("Host cannot engage automatic NUMA binding")
+
+    server_args = [
+        "--dtype",
+        "bfloat16",
+        "--max-model-len",
+        "2048",
+        "--max-num-seqs",
+        "8",
+        "--max-num-batched-tokens",
+        "2048",
+        "--kv-cache-memory-bytes",
+        str(256 << 20),
+        "--enforce-eager",
+        "--distributed-executor-backend",
+        "mp",
+        "--prefill-context-parallel-size",
+        "2",
+        "--data-parallel-size",
+        "2",
+        "--data-parallel-size-local",
+        "2",
+        "--worker-extension-cls",
+        "tests.v1.distributed.test_dense_dp_world_size.NumaBindingWorkerExtension",
+    ]
+    with RemoteOpenAIServer(
+        "deepseek-ai/DeepSeek-V2-Lite-Chat",
+        server_args,
+        env_dict={
+            "VLLM_SERVER_DEV_MODE": "1",
+            "VLLM_WORKER_MULTIPROC_METHOD": "spawn",
+            "PYTHONPATH": os.pathsep.join(
+                filter(None, [str(VLLM_PATH), os.environ.get("PYTHONPATH")])
+            ),
+        },
+        max_wait_seconds=1200,
+    ) as server:
+        completion = server.get_client().completions.create(
+            model="deepseek-ai/DeepSeek-V2-Lite-Chat",
+            prompt="The first five positive integers are",
+            temperature=0,
+            max_tokens=16,
+        )
+        assert completion.choices[0].text.strip()
+        assert completion.usage.completion_tokens > 0
+        print(f"PCP+DP completion: {completion.choices[0].text!r}")
+        response = requests.post(
+            server.url_for("collective_rpc"),
+            json={"method": "check_numa_binding", "timeout": 30},
+            timeout=60,
+        )
+        response.raise_for_status()
+        states = response.json()["results"][0]
+        print(f"PCP+DP NUMA worker states: {states}")
+        assert sorted(s["gpu_index"] for s in states) == [0, 1, 2, 3]
+        assert len({s["gpu_uuid"] for s in states}) == 4
+        assert sorted((s["dp_rank"], s["pcp_rank"]) for s in states) == [
+            (0, 0),
+            (0, 1),
+            (1, 0),
+            (1, 1),
+        ]

tests/v1/simple_kv_offload/test_integration.py

@@ -10,8 +10,11 @@ from transformers import Llama4ForCausalLM, Llama4TextConfig
 
 from tests.models.utils import check_logprobs_close
 from vllm import LLM, SamplingParams, TokensPrompt, envs
-from vllm.config import KVTransferConfig
+from vllm.config import KVTransferConfig, ParallelConfig
 from vllm.platforms import current_platform
+from vllm.utils import numa_utils
+from vllm.v1.metrics.reader import Counter
+from vllm.v1.simple_kv_offload.metrics import MetricName
 
 if not current_platform.is_cuda_alike():
     pytest.skip("Requires CUDA or ROCm", allow_module_level=True)
@@ -113,7 +116,7 @@ def test_chunked_local_partial_gpu_hit(vllm_runner, tmp_path, monkeypatch):
         assert mixed_hit.num_cached_tokens == 16
 
 
-def _make_llm(model: str, lazy: bool, cpu_bytes_to_use: int) -> LLM:
+def _make_llm(model: str, lazy: bool, cpu_bytes_to_use: int, **llm_kwargs) -> LLM:
     kv_transfer_config = KVTransferConfig(
         kv_connector="SimpleCPUOffloadConnector",
         kv_role="kv_both",
@@ -122,12 +125,18 @@ def _make_llm(model: str, lazy: bool, cpu_bytes_to_use: int) -> LLM:
             "lazy_offload": lazy,
         },
     )
+    llm_kwargs = (
+        dict(
+            kv_cache_memory_bytes=40 << 30,  # 40 GiB
+            disable_hybrid_kv_cache_manager=False,
+            enable_prefix_caching=True,
+        )
+        | llm_kwargs
+    )
     return LLM(
         model=model,
-        kv_cache_memory_bytes=40 << 30,  # 40 GiB
-        disable_hybrid_kv_cache_manager=False,
-        enable_prefix_caching=True,
         kv_transfer_config=kv_transfer_config,
+        **llm_kwargs,
     )
 
 
@@ -231,6 +240,90 @@ def _latency_test(llm: LLM, lazy: bool = False):
     )
 
 
+def _assert_worker_auto_numa(worker):
+    import subprocess
+    from pathlib import Path
+
+    import psutil
+
+    from vllm.utils.cpu_resource_utils import parse_id_list
+
+    parallel_config = worker.parallel_config
+    assert parallel_config.numa_bind and parallel_config._numa_bind_auto
+    gpu_index = parallel_config.get_local_gpu_index(worker.local_rank)
+    node = parallel_config.numa_bind_nodes[gpu_index]
+    node_cpus = parse_id_list(
+        Path(f"/sys/devices/system/node/node{node}/cpulist").read_text().strip()
+    )
+    cpus = set(psutil.Process().cpu_affinity())
+    assert cpus and cpus <= set(node_cpus)
+    result = subprocess.run(
+        ["numactl", "--show"], check=True, capture_output=True, text=True, timeout=10
+    )
+    policy = dict(line.split(":", 1) for line in result.stdout.splitlines())
+    assert policy["policy"].strip() == "preferred"
+    assert policy["preferred node"].strip() == str(node)
+    return {"node": node, "cpus": sorted(cpus), "policy": "preferred"}
+
+
+@pytest.mark.slow_test
+def test_simple_cpu_offload_with_auto_numa(monkeypatch, enable_pickle):
+    """Soft-bound NVIDIA workers must store and restore KV, not silently recompute."""
+    if not current_platform.is_cuda():
+        pytest.skip("Automatic NUMA KV-offload smoke test requires NVIDIA")
+    monkeypatch.setenv("VLLM_WORKER_MULTIPROC_METHOD", "spawn")
+    monkeypatch.setenv("VLLM_ENABLE_V1_MULTIPROCESSING", "1")
+    preflight = ParallelConfig(distributed_executor_backend="mp")
+    preflight._numa_bind_auto = True
+    if not numa_utils._can_auto_bind(preflight, local_rank=0) or not (
+        numa_utils._probe_numactl_args(
+            numa_utils._get_numactl_worker_args(preflight, local_rank=0)
+        )
+    ):
+        pytest.skip("Host cannot engage automatic NUMA memory preference")
+
+    model = "Qwen/Qwen3-0.6B"
+    llm = _make_llm(
+        model,
+        False,
+        256 << 20,
+        kv_cache_memory_bytes=128 << 20,
+        max_model_len=1024,
+        distributed_executor_backend="mp",
+        enforce_eager=True,
+        disable_log_stats=False,
+    )
+
+    def loaded_blocks(llm: LLM):
+        return sum(
+            metric.value
+            for metric in llm.get_metrics()
+            if isinstance(metric, Counter)
+            and metric.name == MetricName.LOAD_BLOCKS.removesuffix("_total")
+            and metric.labels.get("model_name") == model
+        )
+
+    try:
+        states = llm.collective_rpc(_assert_worker_auto_numa, timeout=30)
+        print(f"NUMA worker state: {states}")
+        sampling_params = SamplingParams(max_tokens=8, temperature=0)
+        prompt = "hi " * 256 + "Let's count to ten. One, two, three, "
+        cold = llm.generate(prompt, sampling_params, use_tqdm=False)[0]
+        llm.generate(prompt, sampling_params, use_tqdm=False)
+        deadline = time.monotonic() + 30
+        while not llm.reset_prefix_cache(reset_connector=False):
+            assert time.monotonic() < deadline, "CPU offload did not drain"
+            llm.generate("0", sampling_params, use_tqdm=False)
+        loaded_before = loaded_blocks(llm)
+        restored = llm.generate(prompt, sampling_params, use_tqdm=False)[0]
+        loaded = loaded_blocks(llm) - loaded_before
+        assert loaded > 0
+        assert restored.outputs[0].token_ids == cold.outputs[0].token_ids
+        print(f"KV offload blocks restored={loaded}; tokens match")
+    finally:
+        del llm
+
+
 @pytest.mark.optional
 @pytest.mark.slow_test
 @pytest.mark.parametrize("model", SMALL_MODELS)

vllm/config/parallel.py

@@ -326,16 +326,32 @@ class ParallelConfig:
     nnodes: int = Field(default=1, ge=1)
     """num of nodes for multi-node distributed
     inference when distributed_executor_backend is mp."""
-    numa_bind: bool = False
+    numa_bind: bool | None = None
     """Enable NUMA binding for GPU worker subprocesses.
 
-    By default, workers are pinned to their GPU's NUMA-local CPUs and
-    memory; on PCT-capable Xeons they also auto-bind to the SKU's
-    PCT priority cores.
+    Workers are pinned to their GPU's NUMA-local CPUs; on PCT-capable
+    Xeons they also auto-bind to the SKU's PCT priority cores.
+
+    If None (default), vLLM binds only when the environment supports it
+    (multiple NUMA nodes, `numactl` installed, unrestricted CPU affinity,
+    a default NUMA memory policy, and the `spawn` start method with an
+    mp/uni executor and mp DP, including PCP).
+    Ray, external launchers and device-ID remapping are excluded
+    from automatic binding. Automatic mode prefers GPU-local memory but
+    permits allocations on other nodes. Unsupported environments start
+    unbound. With `True`, strict memory binding may override an existing
+    policy, and a missing `numactl` or undetectable GPU-to-NUMA topology is
+    a startup error; `False` disables binding.
     """
+    _numa_bind_auto: bool = Field(default=False, repr=False)
+    """Internal launch decision: use soft memory preference in spawned children."""
     numa_bind_nodes: list[int] | None = None
     """NUMA node to bind each GPU worker to.
 
+    Setting this list implies explicit `numa_bind=True` and requests strict
+    `--membind`, not automatic soft memory preference. It cannot be combined
+    with `numa_bind=False`.
+
     Specify one NUMA node per visible GPU, for example `[0, 0, 1, 1]`
     for a 4-GPU system with GPUs 0-1 on NUMA node 0 and GPUs 2-3 on
     NUMA node 1. If unset and `numa_bind=True`, vLLM auto-detects the
@@ -345,6 +361,10 @@ class ParallelConfig:
     numa_bind_cpus: list[str] | None = None
     """Optional CPU lists to bind each GPU worker to.
 
+    Setting this list implies explicit `numa_bind=True` and requests strict
+    `--membind`, not CPU-only binding or automatic soft memory preference.
+    It cannot be combined with `numa_bind=False`.
+
     Specify one CPU list per visible GPU, for example
     `["0-3", "4-7", "8-11", "12-15"]`. When set, vLLM uses
     `numactl --physcpubind` instead of `--cpunodebind`. This is useful
@@ -553,12 +573,12 @@ class ParallelConfig:
                 "data_parallel_external_lb can only be set when data_parallel_size > 1"
             )
 
-        if not self.numa_bind and (
-            self.numa_bind_nodes is not None or self.numa_bind_cpus is not None
-        ):
-            raise ValueError(
-                "numa_bind_nodes and numa_bind_cpus require numa_bind=True."
-            )
+        if self.numa_bind_nodes is not None or self.numa_bind_cpus is not None:
+            if self.numa_bind is False:
+                raise ValueError(
+                    "NUMA binding lists cannot be combined with numa_bind=False."
+                )
+            self.numa_bind = True
 
         if self.enable_eplb:
             if not current_platform.is_cuda_alike() and not current_platform.is_xpu():
@@ -949,6 +969,7 @@ class ParallelConfig:
             # NUMA node in numa_bind_nodes (see vllm/utils/numa_utils.py
             # `_get_numa_node`), which would otherwise diverge the DP hash.
             "numa_bind",
+            "_numa_bind_auto",
             "numa_bind_nodes",
             "numa_bind_cpus",
             "assigned_physical_gpu_ids",

vllm/engine/arg_utils.py

@@ -541,7 +541,7 @@ class EngineArgs:
     cpu_distributed_timeout_seconds: int | None = (
         ParallelConfig.cpu_distributed_timeout_seconds
     )
-    numa_bind: bool = ParallelConfig.numa_bind
+    numa_bind: bool | None = ParallelConfig.numa_bind
     numa_bind_nodes: list[int] | None = ParallelConfig.numa_bind_nodes
     numa_bind_cpus: list[str] | None = ParallelConfig.numa_bind_cpus
     device_ids: list[int | str] | None = None

vllm/utils/numa_utils.py

@@ -26,6 +26,7 @@ if TYPE_CHECKING:
 logger = logging.getLogger(__name__)
 _NUMACTL_ARGS_ENV = "_VLLM_INTERNAL_NUMACTL_ARGS"
 _NUMACTL_PYTHON_EXECUTABLE_ENV = "_VLLM_INTERNAL_NUMACTL_PYTHON_EXECUTABLE"
+_MPOL_DEFAULT = 0
 
 
 @cache
@@ -41,55 +42,84 @@ def get_libnuma():
     return libnuma
 
 
-def _can_set_mempolicy() -> bool:
-    """Check whether the current process can use NUMA memory policy syscalls."""
+def _get_mempolicy() -> int | None:
+    """Return the current memory-policy mode, or None if it cannot be read."""
     try:
         libnuma = get_libnuma()
         if libnuma is None or libnuma.numa_available() < 0:
-            return False
+            return None
         mode = ctypes.c_int()
         ret = libnuma.get_mempolicy(
             ctypes.byref(mode), None, ctypes.c_ulong(0), None, ctypes.c_ulong(0)
         )
-        return ret == 0
+        return mode.value if ret == 0 else None
     except Exception:
-        return False
+        return None
 
 
-def _is_auto_numa_available() -> bool:
-    """Check whether automatic GPU-to-NUMA detection should be attempted."""
+def _is_auto_numa_available(
+    verbose: bool = True, *, require_unbound_process: bool = False
+) -> bool:
+    """Check whether automatic GPU-to-NUMA detection should be attempted.
+
+    Args:
+        verbose: Log why detection is skipped at warning level; otherwise at
+            debug level, for the default mode where skipping is expected.
+        require_unbound_process: Require known unrestricted CPU affinity and
+            the default memory policy for default automatic binding. Explicit
+            topology discovery keeps its existing best-effort checks.
+
+    """
     from vllm.platforms import current_platform
 
+    log = logger.warning if verbose else logger.debug
+
     if not current_platform.is_cuda_alike():
+        log("Automatic NUMA binding requires a CUDA or ROCm platform.")
         return False
 
     if not os.path.isdir("/sys/devices/system/node/node1"):
+        log("Automatic NUMA binding requires multiple NUMA nodes.")
         return False
 
     try:
         process = psutil.Process(os.getpid())
         cpu_affinity = process.cpu_affinity()
         cpu_count = psutil.cpu_count()
-        if cpu_count is not None and cpu_affinity != list(range(cpu_count)):
-            logger.warning(
+        if cpu_count is not None and set(cpu_affinity) != set(range(cpu_count)):
+            log(
                 "CPU affinity is already constrained for this process. "
                 "Skipping automatic NUMA binding; pass --numa-bind-nodes "
                 "explicitly to override."
             )
             return False
+        if cpu_count is None and require_unbound_process:
+            log("Could not determine CPU availability for automatic NUMA binding.")
+            return False
     except (AttributeError, NotImplementedError, psutil.Error):
-        pass
+        if require_unbound_process:
+            log("Could not check CPU affinity for automatic NUMA binding.")
+            return False
 
-    if not _can_set_mempolicy():
-        logger.warning(
-            "User lacks permission to set NUMA memory policy. "
+    mempolicy = _get_mempolicy()
+    if mempolicy is None:
+        log(
+            "Could not read NUMA memory policy. "
             "Automatic NUMA detection may not work; if you are using Docker, "
             "try adding --cap-add SYS_NICE."
         )
         return False
+    if require_unbound_process and mempolicy != _MPOL_DEFAULT:
+        log(
+            "NUMA memory policy is already set for this process (mode=%d). "
+            "Skipping automatic NUMA binding; pass --numa-bind explicitly "
+            "to override.",
+            mempolicy,
+        )
+        return False
 
     if not hasattr(current_platform, "get_all_device_numa_nodes"):
-        logger.warning(
+        log(
             "Platform %s does not support automatic NUMA detection",
             type(current_platform).__name__,
         )
@@ -98,7 +128,6 @@ def _is_auto_numa_available() -> bool:
     return True
 
 
-@cache
 def get_auto_numa_nodes() -> list[int] | None:
     """Auto-detect NUMA nodes for all visible GPUs."""
     from vllm.platforms import current_platform
@@ -112,6 +141,76 @@ def get_auto_numa_nodes() -> list[int] | None:
     return numa_nodes
 
 
+def _can_auto_bind(
+    parallel_config, local_rank: int, dp_local_rank: int | None = None
+) -> bool:
+    """Validate and store this launcher's automatic GPU-to-NUMA topology.
+
+    Unlike ``--numa-bind``, this never fails startup or forces the ``spawn``
+    start method: any unmet requirement means starting unbound.
+    """
+    from shutil import which
+
+    if envs.VLLM_WORKER_MULTIPROC_METHOD != "spawn":
+        logger.debug("Automatic NUMA binding requires the spawn start method.")
+        return False
+    if (
+        parallel_config.distributed_executor_backend not in ("mp", "uni")
+        or parallel_config.data_parallel_backend != "mp"
+    ):
+        logger.debug("Automatic NUMA binding requires local mp/uni launchers.")
+        return False
+    # Topology is indexed in visible-device order, which --device-ids remaps.
+    if parallel_config.assigned_physical_gpu_ids is not None:
+        logger.debug("Automatic NUMA binding does not support --device-ids remapping.")
+        return False
+    if which("numactl") is None:
+        logger.debug("Automatic NUMA binding requires numactl on PATH.")
+        return False
+    try:
+        from vllm.platforms import current_platform
+        from vllm.utils.cpu_resource_utils import parse_id_list
+
+        if not _is_auto_numa_available(verbose=False, require_unbound_process=True):
+            return False
+        numa_nodes = get_auto_numa_nodes()
+        if not numa_nodes:
+            logger.debug("No GPU-to-NUMA topology available for automatic binding.")
+            return False
+        if len(numa_nodes) != current_platform.device_count() or any(
+            not isinstance(node, int) or node < 0 for node in numa_nodes
+        ):
+            logger.debug("Incomplete or invalid GPU-to-NUMA topology: %s", numa_nodes)
+            return False
+
+        last_local_rank = max(local_rank, parallel_config.local_world_size - 1)
+        last_dp_local_rank = (parallel_config.data_parallel_rank_local or 0) + max(
+            parallel_config.data_parallel_size_local - 1, 0
+        )
+        last_gpu_index = max(
+            parallel_config.get_local_gpu_index(last_local_rank, dp_local_rank),
+            parallel_config.get_local_gpu_index(last_local_rank, last_dp_local_rank),
+        )
+        if last_gpu_index >= len(numa_nodes):
+            logger.debug(
+                "GPU-to-NUMA topology does not cover all local workers: "
+                "GPU index %d, mapping size %d.",
+                last_gpu_index,
+                len(numa_nodes),
+            )
+            return False
+        for node in set(numa_nodes):
+            cpulist_path = Path(f"/sys/devices/system/node/node{node}/cpulist")
+            if not parse_id_list(cpulist_path.read_text().strip()):
+                logger.debug("NUMA node %d has no CPUs for automatic binding.", node)
+                return False
+        parallel_config.numa_bind_nodes = list(numa_nodes)
+        return True
+    except Exception:
+        logger.debug("Automatic GPU-to-NUMA topology validation failed", exc_info=True)
+        return False
+
+
 # PCT (Priority Core Turbo) auto-detection workaround for Granite Rapids
 # Xeon SKUs.
 #
@@ -326,24 +425,13 @@ def _get_numactl_worker_args(
     gpu_index = parallel_config.get_local_gpu_index(local_rank, dp_local_rank)
     numa_node = _get_numa_node(parallel_config, gpu_index)
     cpu_binding = _get_cpu_binding(parallel_config, gpu_index, [numa_node])
+    memory_policy = "preferred" if parallel_config._numa_bind_auto else "membind"
+    memory_arg = f"--{memory_policy}={numa_node}"
 
     if cpu_binding is not None:
-        logger.info(
-            "Binding worker subprocess (local_rank=%s, gpu_index=%s) to CPUs %s and NUMA node %s",  # noqa: E501
-            local_rank,
-            gpu_index,
-            cpu_binding,
-            numa_node,
-        )
-        return f"--physcpubind={cpu_binding} --membind={numa_node}"
+        return f"--physcpubind={cpu_binding} {memory_arg}"
 
-    logger.info(
-        "Binding worker subprocess (local_rank=%s, gpu_index=%s) to NUMA node %s",
-        local_rank,
-        gpu_index,
-        numa_node,
-    )
-    return f"--cpunodebind={numa_node} --membind={numa_node}"
+    return f"--cpunodebind={numa_node} {memory_arg}"
 
 
 def _get_enginecore_numa_nodes(
@@ -387,7 +475,11 @@ def _get_numactl_enginecore_args(
     is also a safe superset by construction).
     """
     shard_nodes = _get_enginecore_numa_nodes(parallel_config, dp_local_rank)
-    membind_arg = ",".join(str(n) for n in shard_nodes)
+    node_binding = ",".join(str(n) for n in shard_nodes)
+    memory_policy = "membind"
+    if parallel_config._numa_bind_auto:
+        memory_policy = "preferred" if len(shard_nodes) == 1 else "preferred-many"
+    memory_arg = f"--{memory_policy}={node_binding}"
 
     pct_cpus = (
         None
@@ -397,21 +489,9 @@ def _get_numactl_enginecore_args(
 
     if pct_cpus is not None:
         cpu_binding = ",".join(str(c) for c in pct_cpus)
-        logger.info(
-            "Binding EngineCore subprocess (local_rank=%s) to CPUs %s "
-            "and NUMA nodes %s",
-            local_rank,
-            cpu_binding,
-            membind_arg,
-        )
-        return f"--physcpubind={cpu_binding} --membind={membind_arg}"
+        return f"--physcpubind={cpu_binding} {memory_arg}"
 
-    logger.info(
-        "Binding EngineCore subprocess (local_rank=%s) to NUMA nodes %s",
-        local_rank,
-        membind_arg,
-    )
-    return f"--cpunodebind={membind_arg} --membind={membind_arg}"
+    return f"--cpunodebind={node_binding} {memory_arg}"
 
 
 def _log_numactl_show(label: str) -> bool:
@@ -455,20 +535,22 @@ def _probe_numactl_args(numactl_args: str) -> bool:
 
 
 def _resolve_numactl_args(numactl_args: str) -> str:
-    """Drop ``--membind`` if the container rejects it, keeping CPU binding."""
+    """Drop a rejected memory policy, keeping CPU binding when possible."""
     cpu_only = " ".join(
-        t for t in numactl_args.split() if not t.startswith("--membind=")
+        t
+        for t in numactl_args.split()
+        if not t.startswith(("--membind=", "--preferred=", "--preferred-many="))
     )
     for candidate in (numactl_args, cpu_only, ""):
         if _probe_numactl_args(candidate):
             if candidate != numactl_args:
                 logger.warning(
-                    "numactl args %r rejected; falling back to %r. Add "
-                    "--cap-add SYS_NICE for full NUMA binding.",
+                    "numactl args %r rejected; falling back to %r.",
                     numactl_args,
                     candidate or "no binding",
                 )
             return candidate
+    logger.warning("numactl probes failed; starting without NUMA binding.")
     return ""
 
 
@@ -481,6 +563,19 @@ def configure_subprocess(
 ):
     """Temporarily replace the multiprocessing executable with a numactl wrapper."""
     parallel_config = vllm_config.parallel_config
+    if parallel_config.numa_bind is None:
+        # Resolve once in the launching process; spawned children inherit the
+        # result through the pickled config instead of re-checking (a bound
+        # EngineCore would fail the unconstrained-affinity check).
+        parallel_config.numa_bind = _can_auto_bind(
+            parallel_config, local_rank, dp_local_rank
+        )
+        parallel_config._numa_bind_auto = parallel_config.numa_bind
+        if not parallel_config.numa_bind:
+            logger.info(
+                "Automatic NUMA binding unavailable on this host; starting "
+                "unbound. Enable debug logging for the skip reason."
+            )
     if not parallel_config.numa_bind:
         yield
         return
@@ -502,8 +597,20 @@ def configure_subprocess(
     numactl_args = _resolve_numactl_args(numactl_args)
     if not numactl_args:
         # No NUMA binding possible here; launch without the wrapper.
+        if parallel_config._numa_bind_auto:
+            parallel_config.numa_bind = False
+            parallel_config.numa_bind_nodes = None
+            parallel_config._numa_bind_auto = False
         yield
         return
+    if envs.VLLM_WORKER_MULTIPROC_METHOD == "spawn":
+        logger.info(
+            "Launching %s subprocess (local_rank=%s, dp_local_rank=%s) with numactl %s",
+            process_kind,
+            local_rank,
+            dp_local_rank,
+            numactl_args,
+        )
     python_executable = os.fsdecode(multiprocessing.spawn.get_executable())
     with (
         _set_numa_wrapper_env(numactl_args, python_executable),

vllm/utils/system_utils.py

@@ -137,9 +137,12 @@ def _maybe_force_spawn():
         os.environ["RAY_ADDRESS"] = ray.get_runtime_context().gcs_address
         reasons.append("In a Ray actor and can only be spawned")
 
-    # Force spawn if NUMA binding is enabled via --numa-bind.
-    # NUMA binding uses executable hijacking which requires spawn
-    if "--numa-bind" in sys.argv:
+    # NUMA binding uses executable hijacking, which requires spawn.
+    if any(
+        arg.split("=", 1)[0].replace("_", "-")
+        in ("--numa-bind", "--numa-bind-nodes", "--numa-bind-cpus")
+        for arg in sys.argv
+    ):
         reasons.append("NUMA binding requires spawn method")
 
     if cuda_is_initialized():