mirrors/pytorch
0
0 Fork 0
mirror of https://github.com/pytorch/pytorch.git synced 2026-10-28 18:56:51 +08:00
Code Activity
Files
main
pytorch/cmake/FileMirroring.cmake
T
Klaus Zimmermann 5d0ef99d2c Address the Python wheel tree by name, not through the install prefix (#191494) 
Since v2.13.0 a standalone `cmake --install` scatters Python package
directories (_dynamo/, _export/, _inductor/, csrc/, testing/, utils/) across
CMAKE_INSTALL_PREFIX alongside bin/, include/, lib/ and share/, and writes
into the source checkout besides. Reported as #191102. The prefix-root
scattering comes from #177643, which added cmake/PackageData.cmake and its
unconditional include; the source-checkout writes from #177642, which did the
same for cmake/FileMirroring.cmake.

scikit-build-core models a wheel as several trees -- platlib, purelib, data,
headers, scripts, metadata -- and exports each to CMake as SKBUILD_<TREE>_DIR.
`wheel.install-dir` selects which tree, plus optional subpath, becomes
CMAKE_INSTALL_PREFIX; it defaults to empty, which makes the prefix the platlib
root itself. We set it to "torch" because the C++ tree has to sit at torch/lib
for the $ORIGIN/lib install rpath and torch/__init__.py's __file__ lookups to
resolve, and that pushes the prefix one level below platlib.

Most of our Python-side install rules were nonetheless written relative to the
prefix, using it as a proxy for the platlib tree. That only holds while
scikit-build-core is driving and install-dir is exactly "torch". A standalone
cmake install has a conventional prefix and no platlib at all, so the same
rules deposit torch/ package data at the prefix root. cmake/FileMirroring.cmake
did address the tree properly via SKBUILD_PLATLIB_DIR, but fell back to
PROJECT_SOURCE_DIR when it was undefined, which is how a standalone install
came to write into the checkout.

This is not an edge case: docs/libtorch.rst documents building libtorch with
`cmake --build . --target install` into a CMAKE_INSTALL_PREFIX of your choice,
which is the flow the report used. The layout that install should produce is
nowhere documented, which is a gap worth closing separately.

The rules now address SKBUILD_PLATLIB_DIR directly -- cmake/PackageData.cmake,
cmake/FileMirroring.cmake, and the _C, _nccl_ep, version.py and _rocm_init.py
rules in torch/CMakeLists.txt -- guarded on SKBUILD, the sentinel
scikit-build-core sets for exactly this question. A build scikit-build-core does
not drive (libtorch, tools/build_libtorch.py, downstream cmake --install)
installs the C++
distribution alone, which restores the pre-2.13 contract: setup.py owned all
the Python packaging back when a standalone install was C++ only.

Two alternatives are worth naming. The reporter suggested a named install
COMPONENT held out of the default install; that is orthogonal, and already in
flight as #184873 for wheel scoping, but it would not fix this on its own,
because CMake runs every install() rule on a component-less `cmake --install`
whatever its tag. Dropping `wheel.install-dir` instead would make the prefix
the platlib root and coincide the two, but it needs a torch/ prefix on ~90 C++
destinations, ~40 of them raw literals not routed through TORCH_INSTALL_LIB_DIR
and friends, and it inverts the conditional: standalone C++ installs would then
need torch/ suppressed everywhere.

The prefix-must-be-<platlib>/torch coupling is now enforced rather than
assumed. install-dir is what makes the two agree, so changing it silently
produces an install that only fails at import; a mismatch is now a
configure-time FATAL_ERROR naming both paths and pointing at pyproject.toml.

Note that tools/build_libtorch.py loses the torchgen/packaged mirroring, as it
builds with BUILD_PYTHON=OFF and no longer produces a Python payload. The only
consumer of those files is torchgen/gen_backend_stubs.py at runtime in an
installed torchgen, which is not in the libtorch path.

Three things surfaced while reviewing the above and are folded in. #191625 added
an install rule for the gitignored torch/_rocm_init.py while this was open, in
the same prefix-relative form; it was the last remaining DESTINATION "." in the
tree and is now guarded and addressed like its neighbours. The _C rule's ARCHIVE
clause is dropped: _C is a MODULE library, so no Windows import library is
produced and none ships -- the 2026-08-18 cp312 win_amd64 nightlies carry 13 to
16 .lib files across the cpu, cu126 and xpu variants, every one a genuine C++
import library and no _C.lib among them. It was also the one prefix-relative
destination left in a rule whose other two were absolute.

A lint rule is added so this class of regression is rejected when it is written
rather than after a release. #191625 is the evidence that it recurs: it landed
three weeks after this fix and nothing caught it, because no CI job installs
standalone and inspects the result -- tools/build_libtorch.py is the only
standalone-install path exercised in CI and it configures with BUILD_PYTHON=OFF,
so it never evaluates these rules. CMAKE_INSTALL_PREFIX_ROOT forbids a
destination of "." across the build system; CMAKE_PLATLIB_DESTINATION
additionally requires every destination in the two package-data files to name
the wheel tree, which catches the prefix-relative variant. Both reuse the
existing grep_linter adapter and are registered in VERY_FAST_LINTERS in
.spin/cmds.py, which spin requires of every linter. This is a text check, not a
behavioural one: a destination that reaches the prefix root through a variable,
or a new rule that uses the wheel tree but omits its SKBUILD guard, still gets
through.

Test Plan:

All on Linux/CPU unless noted, against the reporter's exact configure from the
issue:

```
cmake -S . -B build-standalone -GNinja \
  -DCMAKE_INSTALL_PREFIX=/tmp/prefix \
  -DATEN_NO_TEST=ON -DBUILD_BINARY=ON -DBUILD_CUSTOM_PROTOBUF=OFF \
  -DBUILD_PYTHON=ON -DBUILD_TEST=OFF -DUSE_OPENMP=ON \
  -DCMAKE_INSTALL_LIBDIR=lib -DUSE_DISTRIBUTED=OFF -DUSE_CUDA=OFF

grep -rhoE '\$\{CMAKE_INSTALL_PREFIX\}/[A-Za-z_./]*' \
  --include=cmake_install.cmake build-standalone/ \
  | sed 's|${CMAKE_INSTALL_PREFIX}/||' | cut -d/ -f1 | sort -u
grep -rh --include=cmake_install.cmake 'CMAKE_INSTALL_PREFIX}/\."' build-standalone/
grep -rh --include=cmake_install.cmake "DESTINATION \"$PWD" build-standalone/
grep -rhE --include=cmake_install.cmake 'DESTINATION "(/torch|/torchgen)' build-standalone/
```

Before: `. bin csrc _dynamo _export _inductor include lib share testing utils`,
plus seven rules with destinations inside the source checkout. After:
`bin include lib share`, and the other three greps empty. The last one checks
that an unset SKBUILD_PLATLIB_DIR has not turned a destination into an absolute
path at the filesystem root. The greps must recurse over every
cmake_install.cmake, not just the top-level one, since several of the offending
rules live in subdirectory scopes.

Same configure with -DUSE_CUDA=ON -DUSE_DISTRIBUTED=ON -DUSE_NCCL=ON
-DUSE_NCCL_EP=ON -DUSE_SYSTEM_NCCL=ON: the _nccl_ep target is still built (it
appears in build.ninja, producing torch/_nccl_ep.cpython-310-x86_64-linux-gnu.so)
but contributes no install rule, and the prefix root stays clean.

The guard, exercised the way it can actually regress -- by changing install-dir
in pyproject.toml to something other than "torch" and building a wheel -- fails
configure with the FATAL_ERROR, reporting platlib, the expected prefix and the
actual one.

Wheel contents, via scikit-build-core:

```
pip install -e . -v --no-build-isolation   # twice, consecutively
python -c "import torch, torchgen.packaged, functorch; print(torch.__version__)"
```

Both builds succeed, each configure picking up its own staging directory. The
installed file list under site-packages is identical to a pre-change build of
the same base: 9679 files, none gained or lost, including the mirrored copies
that exist only in the install tree (valgrind.h/callgrind.h, torchgen/packaged
with native_functions.yaml, tags.yaml, 42 templates and 16 autograd files) and
the generated version.py, _C, 43 .pyi stubs and 20 jinja templates. import and
a matmul both work.

The new lint rules, checked in both directions -- clean on this tree, and
flagging the real regressions on the pre-fix tree:

```
spin lint -- --take CMAKE_INSTALL_PREFIX_ROOT,CMAKE_PLATLIB_DESTINATION
```

CMAKE_INSTALL_PREFIX_ROOT flags 6 lines on the pre-fix tree -- torch/CMakeLists.txt
474, 475, 517, 548 and 555, plus cmake/PackageData.cmake:21 -- where 555 is the
#191625 _rocm_init.py rule, so the check demonstrably rejects the regression that
actually happened. CMAKE_PLATLIB_DESTINATION flags 12.

Lint:

```
lintrunner -a --take CMAKE,NEWLINE,SPACES,TABS,CODESPELL,COPYRIGHT,EXEC \
  CMakeLists.txt cmake/FileMirroring.cmake cmake/PackageData.cmake \
  cmake/Summary.cmake torch/CMakeLists.txt .lintrunner.toml .spin/cmds.py
```

Not covered: Windows and macOS were not exercised locally; ciflow/binaries is
what covers them.

This change was authored with the assistance of an AI coding assistant.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Fixes #191102

Pull Request resolved: https://github.com/pytorch/pytorch/pull/191494
Approved by: https://github.com/atalman
2026-08-28 18:27:22 +00:00

69 lines
2.9 KiB
CMake
Raw Permalink Blame History

# File mirroring previously handled by setup.py.
# Copies source files into torchgen/packaged/ and torch/_inductor/kernel/
# so they are included in the installed packages.
#
# These install into SKBUILD_PLATLIB_DIR (the wheel's platlib tree) rather than
# the install prefix, since they target packages other than torch/ (e.g.
# torchgen/) or specific torch/ subdirectories. Only included when SKBUILD is
# set.
# --- mirror_files_into_torchgen ---
# Copy ATen native function definitions and templates into torchgen/packaged/
# so that torchgen can be used standalone without the full source tree.
install(FILES
"${PROJECT_SOURCE_DIR}/aten/src/ATen/native/native_functions.yaml"
"${PROJECT_SOURCE_DIR}/aten/src/ATen/native/tags.yaml"
DESTINATION "${SKBUILD_PLATLIB_DIR}/torchgen/packaged/ATen/native"
)
install(DIRECTORY
"${PROJECT_SOURCE_DIR}/aten/src/ATen/templates/"
DESTINATION "${SKBUILD_PLATLIB_DIR}/torchgen/packaged/ATen/templates"
)
install(DIRECTORY
"${PROJECT_SOURCE_DIR}/tools/autograd/"
DESTINATION "${SKBUILD_PLATLIB_DIR}/torchgen/packaged/autograd"
PATTERN "BUILD.bazel" EXCLUDE
PATTERN "*.bzl" EXCLUDE
)
# --- mirror_inductor_external_kernels ---
# Copy vendored external kernel sources into torch/_inductor.
# The original Python code raises RuntimeError when the source is missing
# on CUDA-enabled builds (cutlass submodule should be present).
set(_cutedsl_rel_path "examples/python/CuTeDSL/cute/blackwell/kernel/grouped_gemm/grouped_gemm.py")
set(_cutedsl_src "${PROJECT_SOURCE_DIR}/third_party/cutlass/${_cutedsl_rel_path}")
if(EXISTS "${_cutedsl_src}")
set(_cutedsl_dest "${SKBUILD_PLATLIB_DIR}/torch/_inductor/kernel/vendored_templates/cutedsl/kernels")
install(FILES "${_cutedsl_src}"
DESTINATION "${_cutedsl_dest}"
RENAME "cutedsl_grouped_gemm.py"
)
# Only create __init__.py for cutedsl/kernels/ (the new directory).
# vendored_templates/__init__.py and cutedsl/__init__.py are tracked
# files with real content — do not overwrite them.
file(WRITE "${CMAKE_CURRENT_BINARY_DIR}/_empty_init.py" "")
install(FILES "${CMAKE_CURRENT_BINARY_DIR}/_empty_init.py"
DESTINATION "${_cutedsl_dest}"
RENAME "__init__.py"
)
elseif(USE_CUDA)
message(FATAL_ERROR
"CuTeDSL source not found at ${_cutedsl_src}.\n"
"The cutlass submodule may be missing. Please run:\n"
" git submodule update --init --recursive"
)
endif()
# --- Symlink-replacement copies ---
# torch/_utils_internal.py and tools/shared/_utils_internal.py exist as two
# regular-file copies in the source tree; only the torch/ one ships in the
# wheel.
install(FILES
"${PROJECT_SOURCE_DIR}/third_party/valgrind-headers/callgrind.h"
DESTINATION "${SKBUILD_PLATLIB_DIR}/torch/utils/benchmark/utils/valgrind_wrapper"
)
install(FILES
"${PROJECT_SOURCE_DIR}/third_party/valgrind-headers/valgrind.h"
DESTINATION "${SKBUILD_PLATLIB_DIR}/torch/utils/benchmark/utils/valgrind_wrapper"
)
View Git Blame Copy Permalink