## Summary
Today, we call uv metadata before we discover the ty project. This is
fine for `uv check` where we always want to call uv metadata, but it's
problematic for `ty check` because it prevents us from adding a
configuration option to skip the metadata discovery (an opt out).
This PR refactors the discovery to:
1. Find the ty project
2. Use the project to decide whether to call uv (for now, this is
skipped because this PR doesn't add the setting)
3. Call uv metadata and if the uv discovered project is different from
ty's, make that project the root instead
## Test Plan
Testing: Project, CLI uv workspace, and language server tests pass;
Clippy and hooks pass.
## Summary
Refresh the `uv workspace metadata` when a user manually changes the
project structure, a `pyproject.toml`, `uv.toml`, or after changing the
active environment using e.g. `uv add`, `uv sync`, or `uv pip`.
ty now watches refetches the uv metadata when:
* a `uv.toml`, `pyproject.toml`, `uv.lock` or `.python-version` file
changes
* a `.dist-info` directory is created or deleted, same for
`site-packages`
* A new included directory is created or deleted (could be a new
workspace member)
I intentionally limited this PR to workspace layouts where all members
are within the workspace root. Members outside aren't watched, to keep
the implementation simpler (we also couldn't watch those paths when `uv`
initially fails because ty then doesn't even know the member's path).
It was also my goal to balance the watcher complexity with avoiding
unnecessary `uv workspace metadata` calls. In most cases, I preferred a
simpler watcher implementation over avoiding unnecessary calls because a
`uv workspace metadata` call for an unchanged workspace should be very
cheap.
## Review
I recommend reviewing this PR commit by commit.
## Summary
This PR improves the uv-metadata diagnostics. Specifically, it fixes
that uv-metadata diagnostics spawn multiple lines when using
`output-format=concise`.
* Concise: `warning[uv-metadata] Failed to load uv metadata: No solution
found when resolving dependencies`: `Failed to load uv metadata:
{first_line_in_uv_output}`.
* Full: `uv workspace metadata` failed with `{status}` + subdiagnostic
with the details
```
error[uv-metadata]: `uv workspace metadata` failed with `exit status: 1`
--> script.py:2:1
info: No solution found when resolving dependencies
cause: Because missing-script-dependency was not found in the cache and
you require missing-script-dependency==99.0.0, we can conclude that your
requirements are unsatisfiable.
hint: Packages were unavailable because the network was disabled. When
the network is disabled, registry packages may only be read from the
cache.
```
Now, this is all best-effort:
* Concise: Only taking the first line could theoretically result in a
mid-sentence truncation if uv emits a multiline summary message.
* Full: Rendering `info: {first_line}` risks that second line is
misaligned if it line e.g. annotates the first line.
I don't think there's much we can do about this without uv emitting
structured diagnostics.
The first two commits restructure the uv file watching tests and change
them to show the diagnostic output. While not strictly necessary for
this change, a separate PR felt a bit overkill.
I recommend reviewing this commit by commit.
## Test Plan
Focused CLI, LSP, and uv file-watching tests pass.
## Summary
This PR is a first step towards updating the project's uv metadata when
the virtual environment changes. The scope of this PR is to add the
non-uv specific virtual environment file watching. That is, it adds the
necessary file watching handling to update the detected virtual
environment and its Python version when:
* a `pyenv.cfg` changes
* the default `.venv` or the explicitly specified `environment.python`
virtual environment gets created or deleted.
Unfortunately, this does require storing the path to the resolved
virtual environment in `ProgramSettings`. Otherwise, we'd have to
resolve it every time we see a related file watcher event. It's
unfortunate because it doesn't feel like a setting. But there isn't
really any better to carry forward this information.
I recommend reviewing this PR commit by commit.
## Test Plan
Native file-watching regression tests pass.
## Summary
This PR implements watching for script search paths.
Before, changes to files in a script's search path were not picked up by
ty, because the virtual environment is typically outside the
`project_root` (at least when created by uv) and ty didn't explicitly
watch the search paths for changes.
This PR extends the CLI's watcher to also register script search paths
as watched folders (while still deduplicating watched paths as before).
The one caveat of registering script search paths is that we need to
update the search paths after every edit because editing a script
metadata block could change the script's search paths (e.g. when a user
adds a new `tool.ty.environment` block). To make this as cheap as
possible, computing the watch paths is a salsa query and we compute a
`cache_key` that allows us to bail early (and extremely cheaply) if the
search paths remain unchanged.
I plan to make the same change for the LSP, but it's a bit more work
because the LSP watcher simply watches the project folder. We never
implemented the necessary dynamic registration to watch other search
paths (e.g. extra paths outside the project etc). I plan to tackle this
next.
## Test Plan
Before, changes to a script's search path were not detected by ty:
https://github.com/user-attachments/assets/06cfe9b8-fa74-4437-aeee-6f54b80d6a5a
Now, changes are recognized and trigger a recheck of the script
https://github.com/user-attachments/assets/2685261b-306d-4129-b238-a3eca2c88713
## Summary
Removes the "Who's using ruff?" list.
The section was useful when Ruff was young and not well known. Ruff is
now widely used in the ecosystem, so that this list provides diminishing
returns and we also lack a clear guidance on what it takes for a project
to be added.
## Summary
Now that we queue all scripts on startup, I thought it's important to
have a way to cancel already queued sync requests if they're superseded
by a new sync.
The implementation here is relatively simple. We use our existing
`CancellationToken` and check it before a worker invokes `uv metadata`.
This is similar to what we do in the LSP.
The downside of this is that the new request for the cancelled script
will, again, be queued at the back. I think this is fine for now. We can
explore with more elaborate schemas later.
## Test Plan
The focused CLI, LSP, and file-watching tests pass.
## Summary
This PR moves from a lazy PEP 723 script discovery do eagerly discover
all scripts as part of the project's file discovery.
There are two motivations for doing this:
1. Initializing the script's environment before checking the script is
not sufficient. Other files can import the script, and type inference
can then resolve `file_settings` and `python_version_with_source` which
are both environment dependent (the former less so, but uv can tell us
the python version). Initializing all scripts before we start checking
ensures that type inference sees the right environment even when a
script has been imported. This is the bug that Zsolt identified in my
initial PEP 723 PR.
2. Today, the LSP only initialized scripts when the user opened them.
However, that means that their environment isn't available as part of
find references and workspace symbols. This can result in missed
references (note, making find references return all project and script
references when searching for references of a shared dependency requires
a few changes to find references as well, but this sets the foundation
for it). Initializing all environments when opening the project ensures
the environments are available when running find references (or the user
can rerun find references once all syncs are complete, but they don't
have to open every script just to make sure that find references will
show the references in them)
3. Knowing all scripts simplifies file watching because we now have
distinct points where script environments can change, and may require
updating the watched paths.
Now, there are a few downsides as well, but I think this is the right
trade off (honestly, I don't know how to fix 1 easily without it, and 3
was the main reason why I considered this approach even before Zsolt
found the bug).
* Unlike before, the LSP now indexes and reads all project files. This
can mean that startup now takes a second or so on very large projects
where it was instant before, because we didn't do any form of indexing.
This also means that the LSP retains more memory for projects that don't
use workspace diagnostics (or find references), because the LSP now
loads all files into memory to determine whether a file is a script.
This is inherent to the approach. I suspect that we can mitigate the
"all source in memory" in the future by only storing compressed source
text, which should reduce memory usage by about 70%.
* We now sync all scripts on LSP startup. I could see this become a
problem in workspaces with many (with 100 or 1000s scripts). If this
becomes a problem, I think we can a) limit script sync to on-open,
accepting that find-references etc. may require opening the relevant
scripts and/or b) allow users to selectively sync scripts by using a LSP
command. For now, I think syncing all scripts should be fine, given that
uv is fast (the first time might take a while)
* I had to make the uv sync queue unbounded. It's still bounded by the
number of scripts and projects. But we no longer apply back pressure if
the sync requests start piling up. The reason I had to remove it is
because we don't want a `didChange` or `didOpen` to block, only because
there are already 8 queued scripts. But more importantly, we don't want
to block the LSP startup until all scripts have synchronized, but
queuing them up during startup would just have done that (because the
main loop blocks everytime time the queue is full). I think this is
okay, and I'll add another PR on top of this one that adds cancellation
support for sync requests. I also think that our previous back pressure
wasn't as impactful as I thought
* We now sync scripts before checking the files. That means, ty no
longer makes progress on checking other project files while the syncs
are pending. This is the one I'm the most bumped about. It was so cool
that ty often checked the entire project while waiting on uv to finish
syncing. To mitigate, I increased the worker pool to N/4 where N is max
parallelism and, while testing, I also found that syncinc all scripts,
at least in Ruff's repo, is instant once uv is warm.
I think this also gets us into a position where adding support for
nested projects becomes much easier, because they'll naturally fit in.
We'll discover nested projects as part of indexing `Files` (we'd need to
move that off `Project`), synchronizing their environments can happen at
the exactly same points as synchronizing scripts. The biggest remaining
lift is to change `ty_project` and `ProjectDatabase` to no longer assume
that there's a single `Project` (but there is still one main project).
## Test Plan
CLI
https://github.com/user-attachments/assets/db370cde-d9db-4c85-8772-06b608ce6587https://github.com/user-attachments/assets/ae6e825a-6814-4dc0-8b2f-630469de1a8d
## Summary
This refactors the project's `uv workspace metadata` call to run in the
background, the same as for scripts. This ensures that the watch command
and the LSP remain responsive and show a progress indicator while uv is
running.
It should also make it easier to run `uv workspace metadata --sync` for
projects. Although, that does require some work so that we can surface
progress from the initial uv call (during project discovery) and not
have the LSP blocked until the command completes.
This PR does not change when we run `uv workspace metadata`. That is, it
doesn't add logic to detect when uv needs to rerun. The same as before,
`uv workspace metadata` is only called when the `pyproject.toml`
changed. It isn't run when a `uv.toml` or the virtual environment
changes. We should do so, but this should be its own PR.
This PR enables `TY_UV=1 ty check --watch`
## Review
I recommending reviewing this PR commit by commit and with whitespace
changes disabled.
## Test Plan
LSP:
https://github.com/user-attachments/assets/8c1ab9df-8209-44d5-b0b2-ba06cb824625
1. Simulate a long `uv metadata` command (e.g. when `uv sync` is holding
the lock for multiple seconds). You can see the progress report
2. Changes to a `pyproject.toml` trigger a refresh
CLI
https://github.com/user-attachments/assets/b3cde596-c7f5-4f2e-880b-d3885c02dcb8
## Summary
ty supports in-memory changes, but uv always reads the script's content
from disk. This discrepancy could lead to ty skipping a sync when,
between opening a file and saving it, the script metadata block wasn't
changed, but it changed compared to what was saved on disk when the file
was opened.
The root cause of this issue is that ty computes the script-fingerprint
from the current file content (which may include in-memory changes).
Instead, it should compute the fingerprint as it is seen on disk.
I considered a few different approaches:
* Support reading the persistent content in `ruff_db`: Add a new
`persisted_source_text` query, track the persisted `File` revision and
the in-memory revision separately. `ScriptsEnvironments` can then
compute the fingerprint based on the persisted source text.
* Use `system.read_text` in `ScriptsEnvironments`. I disregarded that
quickly because we proactively trigger the synchronization of all
scripts in a few places and we rely on this bailing quickly if the files
are unchanged (we don't want to read each script to bail).
* Introduce a new `Force` sync mode and use it in `didSave`.
I'd pick the first approach if we had multiple frontends that supported
in-memory changes. But, given that it's only the LSP, I started with the
`Force` sync mode, and I almost opened a PR for it. During the final
cleanups, I realized that there's a much simpler LSP-only fix (the CLI
doesn't have this problem).
We don't need a `persisted_source_text` query if we schedule the script
synchronization before we set the Script's in-memory content in
`didOpen`. That is, first trigger the synchronization, then open the
file and load its in-memory content. That's all we need.
Fixesastral-sh/ty#4250.
## Test Plan
**Before**
Reload VS Code with unsaved changes, hit save. Note how the script block
isn't synced (because we didn't make any changes to it)
https://github.com/user-attachments/assets/cd050b94-08a3-4c61-ba14-4a3f1b7a1e48
**This PR**
Same sequence, but the sync runs.
https://github.com/user-attachments/assets/c8acc477-7666-4411-a9b4-df33328ccb7e
## Summary
Each script with inline metadata (PEP 723) gets its own virtual
environment with its own dependencies. This is to guarantee isolation.
Before for this PR, go to definition only worked when resolving an
import directly in a script. E.g. `from mypyp_primer.projects import
get_projects`, clicking on projects resolved successfully. However, all
go to definition requests within `mypy_primer.projects` failed (unless
the main project had `mypy_primer` installed as a dependency).
The root cause of this was in `db.program_file(db, file)`. The method's
responsibility is to resolve the most likely program to use for `file`.
Today, `db.program_file(db, file)` only handled two cases:
* It's a first-party and it has inline metadata (PEP 723). Use the
script's `Program`. This is why clicking on `mypy_primer.projects`
worked from within the script
* Otherwise, use the `Project`'s program
Because of that, any script dependency used the `Project`'s `Program`
instead of the script's.
This PR extends `db.program_file(db, file)` with a heuristic for
third-party files (it's important that `db.program_file(db, file)` is
accurate for first-party files, for third-party files, the best we can
do is "guess"). That is, iterate over all known `Program`'s and use the
`Program` for which file appears on the search paths. Use `Project` as
the fallback (still the best guess if everything else fails).
This operation isn't expensive but it also isn't cheap. However, we only
call `db.program_file` at API boundaries. That is, we mainly call it
once for each LSP request. There are a few where the method is called
more than once (2 or three times, constant factor), but these are all
first-party files, for which `db.program_file` keeps using the
fast-path. That's why I don't think it's justified to Salsa cache this
resolution. But we can if this turns out to be a performance bottleneck
in the future.
Closes https://github.com/astral-sh/ty/issues/4253
## Test Plan
https://github.com/user-attachments/assets/5cff9f33-cb54-46d9-9f33-82f1ee63999b
Added language-server integration coverage and ran the affected test
suites and Clippy.
## Summary
This PR adds support for TOML syntax highlighting in PEP 723 inline
metadata blocks by emitting TOML specific tokens during semantic
highlighting. I know, this wasn't really something we planned on adding
but I got annoyed by starring at large gray blocks and it was too easy.
This approach is relatively straightforward, and has the advantage that
it works for all editors. However, it also comes with limitations:
* Parentheses matching and completions don't work
* It can't easily support inserting `# ` on enter (when adding a new
line)
* No support for go to def, hover etc (we could add them, but it becomes
a bit more questionable whether this is in ty's scope)
To support these features, we'd have to add an embedded grammar to
ty-vscode (and other editors). I think this is still something worth
considering, and the approach taken here doesn't prevent us from doing
this. That's why I decided to go with this easier and cross-editor
implementation for now.
The emitted tokens are inspired by the grammars of VS Code toml
extensions.
Note: I'm only aware of sublime supporting semantic highlighting of
inline metadata. I don't think pyrefly or pylance do.
## Test Plan
<img width="1245" height="603" alt="Screenshot 2026-08-14 at 11 07 41"
src="https://github.com/user-attachments/assets/120785d5-18a9-478e-af65-77819446b2d2"
/>
This completes https://github.com/astral-sh/ty/issues/4183 across all
surfaces by integrating the uv script environment synchronization into
the language server.
As explained in earlier PRs. The main challenge in the LSP are:
* Low latency: We don't want a uv synchronization to block unrelated
requests (for other files) or requests that don't need the initialized
virtual environment (go to def and semantic tokens already work decently
without an initialized virtual environment)
* Avoid flickering diagnostics: We want to avoid diagnostics
disappearing and reappearing. Specifically, diagnostics should not
disappear when you declare a new dependency only to then reappear when
the synchronization is complete. Instead, keep using the old virtual
environment while syncing the new environment so that completions,
diagnostics, semantic tokens etc remain unchanged until the sync is
complete. This gives a much nicer and more consistent experience than
throwing away the outdated virtual environment or simply returning empty
diagnostics while syncing.
* The LSP supports in-memory changes, uv does not. This PR avoids this
issue in most cases but not all. I opened a follow up issue to fix the
one instance that I'm aware of where this could lead to ty skipping a
synchronization when it should instead trigger a resync. I have ideas of
how to fix this, it just requires touching a few core infrastructure
pieces, which would unnecessarily bloat this PR (and I think the case is
uncommon enough that we can ship an initial version with this known
bug).
The script support hooks in to the following notifications:
* `didOpen`: Trigger a sync when the user opens a file in the
background. Use an empty virtual environment while the sync is running.
Skip diagnostics to avoid showing false-positives (we don't want to show
a bunch of unresolved import errors).
* `didSave`: Trigger a resync to update the virtual environment if the
inline metadata changed
* `didChangeWatchedFiles`: Trigger a resync for files that changed on
disk (that aren't open) but that ty analyzed before (e.g. when using
workspace diagnostics).
The notable omission is `didChange`. I decided not to sync on change.
Mainly because uv doesn't provide a way to pass the file content via
stdin. But also because syncing on every change might be a bit
excessive. This is something we can reconsider once uv supports syncing
in-memory changes.
Closes https://github.com/astral-sh/ty/issues/4183 (this entire stack)
## Opt In
This PR adds a new `experimantal.useUv` server setting. It has three
modes:
* `off`: default, today's behavior, ty doesn't use uv to sync script
environments
* `scripts`: Use uv to sync scripts, but not to retrieve project
metadata
* `on`: Use all supported uv integrations (the same as `scripts` for
now)
I expect that we'll iterate on the setting options, but it roughly is
the same as `TY_UV`, just for the LSP. It's also likely that this
setting will make it into ty's persistent configuration. But I decided
to defer that, because a persistent setting has a higher cost and I
don't know where to add this setting (new `uv` section or is it
`environment.use_uv` or something else)?
## Test plan
https://github.com/user-attachments/assets/2782ac93-1ea4-44ff-bc8b-a22ff33c89a7https://github.com/user-attachments/assets/8dabd518-0985-4fa5-adb2-24ced78c0469
## Summary
This PR adds `uv metadata` support to `ty check --watch`. The main
complexity introduced by this PR is that we can't synchronous scripts on
ty's main loop, because that would make `Ctrl + C` unresponsive, ty
would also stop responding to other incoming file changes (which can
fill the file watcher queue, which ultimately results in dropped events,
requiring rescanning).
This is why this PR introduces an API to start a background
synchronization, that notifies the main loop when a new completed sync
result is ready to be applied.
The CLI waits before scheduling a new check until all synchronization
have completed, to avoid running on a partial virtual environment.
Part of https://github.com/astral-sh/ty/issues/4183
## Test plan
https://github.com/user-attachments/assets/16034a0f-fc0f-4833-9747-eed5b5120c76
## Summary
This PR moves the `uv metadata` synchronization for scripts to a
background worker in preparation for watch and LSP support.
Up to now, the metadata synchronization always ran blocking (with some
form of polling). This works when checking an entire project, because we
can't finish checking until the synchronization finished. However, we'll
need an asynchronous (in the background) syncing mechanism for `--watch`
and the LSP to avoid blocking the main loops (which would result in
rust-analyzer like stalls where everything waits on `cargo check` to
complete).
* CLI: The initial synchronization continues to run in blocking mode
(because we need the environment to check the file). Updating the
environment after changes will run on the background thread to ensure ty
remains responsive to other files changing and `Ctrl + C`
* LSP: I'll talk more about this in the LSP PR but things are more
complicated here. Only the `diagnosticMode: Workspace` uses blocking
synchronization. All other synchronizations run async in the background
when a file is opened or saved to avoid blocking other LSP operations
(for other files) or operations that don't necessarily need a fully
initialized environment (e.g. semantic tokens, go to def within the
file, ...).
## Concurrency
I decided to limit concurrent `uv metadata` runs to 2 instances. There's
no good reason for 2, and we may want to play with this going forward.
It may also be something where uv will take more control because ty
really can't know what the ideal parallelism is. It depends on:
* How many venvs are warm vs how many are cold?
* How good is the network connectivity?
Ultimately, it feels like uv can make the best scheduling decisions.
## Test plan
https://github.com/user-attachments/assets/ce078f4f-508a-4fe4-a6c3-239bc4312da4
## Summary
This PR implements the first part of the script's uv integration. When
opting in with `TY_UV=1` or `TY_UV=scripts`, ty now creates a dedicated
virtual environment for scripts using `uv workspace metadata --script
<file> --sync`. This includes scripts with no dependencies (should we
skip those?).
This PR limits its scope to `ty check`. Integrating syncing into `ty
check --watch` and the LSP are handled as follow up PRs. I also decided
to move support for Ctrl+C (so that ty exit even when blocked on a uv
metadata call), and bounding the number of in-flight uv calls into
follow up PRs.
ty skips checking a file if its `uv workspace metadata` call failed,
similar to how it skips checking a file if the PEP 723 metadata is
invalid.
Part of https://github.com/astral-sh/ty/issues/4183
## Test Plan
https://github.com/user-attachments/assets/8da82ae4-0cc6-4484-863c-ec2f256951e7
## Ecosystem timing report
> The large regressions are real and consistent with per-script `uv`
subprocess overhead:
> The timing results look consistent with `uv` startup cost rather than
a general ty slowdown.
>
> - `scikit-build-core`: 3 PEP 723 scripts, +168 ms
> - `pybind11`: 3 scripts, +122 ms
> - `dd-trace-py`: 22 scripts, +711 ms
>
> This roughly scales with the number of `uv workspace metadata --sync
--script` invocations. The single-script CodSpeed result (+57 ms)
corroborates that range.
>
> Across all 162 projects, the average factor is 1.03×, with six
reported speedups and six slowdowns. The smaller changes therefore look
like measurement noise; material overhead is confined to repositories
containing PEP 723 scripts. If we want to optimize this later, the
target is reducing or amortizing the per-script uv process startup cost.
## Summary
Enable the new automatic "sync script virtual environment" in ecosystem
runs. This ensures that scripts will be checked with their own virtual
environment and own dependencies (at least, once PRs later in this stack
land).
I decided to only enable the new scripts behavior because enabling the
new automatic uv sync for all projects breaks mypy primer, because uv
now tries to install all dependencies of that project which may not
succeed. We'll need to find a solution for this, but I prefer to solve
this later
## Summary
Add a CodSpeed baseline for checking a PEP 723 script that imports a
declared dependency. The following CLI integration PR uses the same
benchmark to measure warm-cache uv synchronization overhead and catch
regressions.
Testing: Ran the benchmark in the profiling profile and checked the
affected benchmark target.
## Summary
Skip checking scripts with invalid PEP 723 metadata blocks, that are,
blocks that ty can't parse successfully or configurations that don't
resolve successfully. Instead, emit one diagnostic that points to the
invalid metadata.
ty still initializes a `Program` for each of those scripts so that LSP
operations continue to work. But we skip those scripts in all code paths
where we perform semantic checks.
Fixes https://github.com/astral-sh/ty/issues/4181.
## Test plan
Added tests
## Summary
Preserve source ranges for PEP 723 script metadata by building a compact
source map during extraction and applying it when deserializing ranged
values.
Closes https://github.com/astral-sh/ty/issues/4180
## Summary
Pass the s390x architecture flag to release builds so that stacker
builds successfully
## Test Plan
Testing: Focused repository hooks passed.
## Summary
This PR extends the PEP 723 configuration support to `ty.environment`
settings. That is:
* Scripts are checked with a separate virtual environment and search
paths
* Scripts can configure a different Python version
* Scripts can configure a different Python platform
Our long-term plan is to use uv to create the script's virtual
environment. We don't do this in this PR. Instead, we:
* Respect `VIRTUAL_ENV`, LSP fallback environment (the selected
environment), global available Python installation. Unlike for projects,
ty does not search for a local `.venv` folder
* Unlike for projects, scripts default to an empty `environment.root`
because scripts are considered to work in isolation. Users can customize
`environment.root`.
Out of scope for this PR but things we should look into next:
* Report a diagnostic if a script's metadata fails to parse instead of
falling back to the project's configuration.
* Add support for the Python environment extension's file-specific
virtual env (so that you can select a virtual env for a specific file)
* Preserve source ranges for configuration options from script metadata
(shouldn't be too hard but the PR is already large as it is)
* Integrating with uv. I expect that this will require significant
changes to the implementation to provide good UX, but I like to tackle
problems one step at a time (it also makes it easier to explain why we
changed something).
* We currently don't watch for changes in script search paths. That
means, results in the LSP or when using `ty check --watch` could get
outdated
## Test Plan
Testing: Added integration and language-server coverage and verified the
relevant unit, CLI, server, and Markdown suites.
Move standalone script handling and uv workspace metadata into dedicated
top-level project modules.
This establishes the module structure used by the stacked PEP 723 work
without changing behavior.
Testing: Ran the project test suite, all affected CLI/LSP tests, Clippy,
and repository hooks.
## Summary
`PythonVersionWithSource` contains from where ty derived the
`PythonVersion`. If the Python version comes from the project's
`pyproject.toml`, then the source includes the configuration file name
and the text range of the `requires-python` or `python-version`
key-value pair. Any change to that `pyproject.toml` would result in a
new and different `PythonVersionWithSource`, which, in turn, results in
a distinct interned `Program`. This results in invalidating every single
type inference query (more accurately, we now get a new instance of each
type inference query, but we loose all cached results).
Storing `PythonVersionWithSource` also has the downside that it prevents
scripts that otherwise have identical `Program` settings to share the
same `Program`, only because their `python-version` comes from different
script metadata blocks.
This PR moves `PythonVersionWithSource` out of `Program`. `Program`
should only contain information that changes type inference.
`PythonVersionWithSource` does not. `Db` now has a new
`python_version_with_source` method that, given a file, returns the
source for it (A script can return a different source than a project
file).
## Test Plan
Testing: Semantic, project, core, server, Markdown, and corpus tests
pass; repository hooks pass.
## Summary
This is the last PR that adds support for type checking multiple
programs with different Python Version, search paths, and platform using
a single Salsa DB.
This PR changes `Program` from a singleton input to an interned struct.
`ProgramFile` changes from `(File, ResolverEnvironment)` to `(File,
Program)`.
Changing `Program` from an input to an interned means that it now has a
`'db` lifetime. This requires changing how we construct and store
`Program`. That's why some of the db initialization code changes.
I doubt that it will be noticeable. But this should speed up mdtests
that contain tests with different Python versions (e.g. Py312, Py311,
Py312, Py311), because we now reuse parsed modules, and even type
inference results because the inference is now keyed by Program (instead
of updating an existing Program)
## Memory regression
I had codex look into the CodSpeed reported memory regression. The
reason is that `Program` now being an interned and being initialized
later changes the assigned `File` ids, which in turn changes in which
order we iterate over the project files. And this change the max peak
memory usage. It can also change the usage reported on the memory usage
report (we enter fixpoint cycles from different heads, etc).
The changes in the memory report are mainly from the new
`ProgramFile::python_version` (needed to mitigate a larger regression,
but we can reconsider), and that `Project` now stores `ProgramSettings`.
## Summary
This PR continues the refactor to make type inference and module
resolution support scripts that may have different settings than the
main project.
This PR introduces:
* `ResolverEnvironment(SearchPaths, PythonVersion)`: Uniquely identifies
the settings used to resolve a module
* `ResolverFile(File, ResolverEnvironment)`: A file with its
`ResolverEnvironment`. Used for desperate module resolution and
`module_to_file` (and all other places where the operation doesn't
depend on type-inference specific settings).
This PR also `ProgramFile` and updates the `Program` alias to point to
`ResolverEnvironment`. These changes are in preparation to reduce the
diff of the third PR (we don't want to rewrite all `python_file` call
sites to `resolver_file` to then rename them to `program_file`).
## Memory regression
This PR changes `Program` from `PythonVersion` (u16) to
`ResolverEnvironment` (u64). This increases the size of interned structs
and interned query arguments that contain `Program`. Memory usage is
still lower than before we made db `Program` aware (the base PR reduced
memory usage by about 1%).
## Test Plan
Testing: Ran workspace checks, resolver/core/semantic/IDE tests, and
repository hooks, including cross-version inference coverage.
## Summary
This is the first PR out of 3. It extends `ty_python_semantic` and lower
crates to support checking files that use different Python versions
while using a single Salsa db.
The motivation for supporting different Python versions is that scrolls
need to be checked independently. Initially, I planned to implement
scroll support in a similar way to how we support multiple workspace
folders in VS Code by using multiple dbs, but it turned out, that
they're different enough. The main difference is that scrolls are
checked as part of their enclosing project, and the diagnostics from
scrolls are merged with the diagnostics from the project. If we used
separate scroll dbs, then the scroll diagnostics would use `File`s
pointing into the scroll db and all other diagnostics would use `File`s
pointing into the main db. There's nothing that would allow us to tell
to which db any such `File`s belonged, and `File`s from different dbs
can even share the same value. There's also no lifetime that binds them
to one particular db because they're inputs. We could have decided to
rewrite the `Diagnostic`s for scrolls and convert the scroll db `File`s
to project db `File`s, but that felt very brittle. The other
disadvantage was that it became very difficult to tell which query is
cached because you now also need to reason about whether the current
query is cached for a specific argument and db (and not just the
argument).
This is why I decided to explore whether we could support checking files
with different Python version, search paths, and platforms, and this PR
is the first step towards this. It adds support for checking files with
different Python versions. Specifically, this PR removes the
`db.python_version` from `ruff_db::Db` and lifts it up to
`ty_project::Db`, proving that no semantic crate depends on it.
The first step is to make `parsed_module` Python version aware. That is,
instead of calling `db.python_version`, parametrize the query by file
and Python version. For this, I introduced a new `PythonFile` which is a
`(File, PythonVersion)` tuple. We now attach this `PythonFile` to
`Scope` (instead of `File`), because `::node` accessors need it to
retrieve their node. That in turn required `semantic_index` to take a
`PythonFile` instead of `File` as well. This was a mostly smooth ride.
Supporting multiple Python versions in the module resolver required
adding a `PythonVersion` argument to most queries. The next PR will
replace the `PythonVersion` with a new `ResolverSettings` (or similar
name) interned struct that wraps both Python version and search paths.
This was also mostly straightforward.
The hard part was to remove all `db.python_version` calls from type
inference and type operations. Today, we call `db.python_version` where
we need it, but we can't do this anymore. Instead, we have to pass the
Python version down to each function that needs it.
Now, this is not true for all functions. Functions that have access to a
scope (via `Definition`, `Expression` or similar) could retrieve the
Python version from there (because `Scope` stores a `PythonFile`). I
decided against that and, instead established the following policy:
* Explicitly pass the Python version to all semantic functions.
* The only exception to this are Salsa queries that have exactly one
argument other than `db`. For those, retrieving the Python version from
their `Scope` is allowed for performance reasons.
Ultimately what that means: The context should be initialized at the
query boundary and then not change, unless we cross into another Salsa
query (but assert that the python version does not change).
There are two motivations for this:
* I think it's easier to reason about what Python version a type
operation uses if there's a consistent way of retrieving it (always as
argument)
* At query boundaries, we can assert that the explicitly passed Python
version and the `Scope`'s Python version are identical, before calling
into the query (debug assert).
However, supporting different Python versions now requires that we
parametrize all type operations with the Python version under which the
operation should be performed (in the future, this will be Python
version, search paths, and platforms).
Now, passing `db` and `PythonVersion` as separate arguments not only
results in a huge diff (add another 7k changes on top of the current
diff), is cumbersome, it also is slower. That's why I this PR introduces
a new `SemanticContext` that wraps a `db` and a `PythonVersion` (Happy
to discuss names). The variable is consistently referred to as `ctx` to
reduce the diff (using `context` makes it more likely that lines break).
## Next steps
* Change `SemanticContext` to store a `db` and `SearchPathSettings` to
support different search paths
* After that, change `SemanticContext` to store a `db` and a
`ProgramFile` (which is a `(File, Program)` tuple.
* Use the new infrastructure for scroll support
## Performance
There's currently a 2-3% walltime regression. Codex reports a lower
local regression.
There are some options that we can explore if we think this regression
is too big. However, it would make the model a little more muddy. The
most promising is probably to have two versions of very hot queries.
E.g. `is_redundant_with` could have a `is_redundant_with_project` and
`is_redundant_with_script` query. `is_redundant_with_project` would use
`db.python_version` internally (yes, we'd have to add that back). I
don't like that. Not only does it reintroduce `db.python_version` and
make it very easy that we use it somewhere where we should not.
## Review
I, honestly, tried to split this PR into smaller, reviewable commits.
But I don't think I succeeded. But I also don't think it makes sense to
review this commit line by line. I'm sure there are places where we
still pass `db` where we should pass a `ctx`. I'm sure there are some
call sites where we should retrieve the `python_version` in a different
way. But I'm not sure it's worth finding all these places in a review.
Instead, I'd focus on the overall change. What's your reaction on
`SemanticContext`? That we now need to pass it everywhere? What about my
policy that we use a single `SemanticContext` within a query? Should we
relax this constraint, so that `SemanticIndex` must be passed in fewer
places, if the `PythonVersion` can be retrieved at the relevant call
site (because it has `Scope`)? The downside of that is that it becomes
more difficult to assert that we don't mix programs in a single
inference, but is this something we should really be concerned about? Do
you think the performance regression is to big for this feature?
## Notes
* Make associated tracked methods that don't take `python_version`
normal methods with an inner query instead. The method should take a
`python_version`, and we assert (before calling the query), that the
target's (self's) `python_version` matches the passed python version
## Summary
The `add` operations move the `Union`/`IntersectionBuilder`. This can be
noticeable in hot mappings and normalization where the fluent API also
isn't required. This PR uses `add_in_place` in more places and also adds
such an API to `IntersectionBuilder`.
This improves performance by about 1%. I extracted this out of the
scripts PR, so please remind that when considering the regression of
this PR. I discovered this optimization because the scripts refactor PR
increased the size of `UnionBuilder`, making the move more expensive.
Since I'm out Monday/Tuesday, feel free to merge.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
Micha Reiser