mirror of
https://github.com/pytorch/pytorch.git
synced 2026-10-25 18:53:39 +08:00
Bob Ren
10ca444331
**Problem.** When Dynamo gives up on a frame during a precompile capture, the capture runs that frame eagerly, but the package does not record it, so a loaded artifact refuses calls the capture handled fine.
**Fix.** The package records those frames as `eager_fallback`, and a loaded artifact runs a miss of one eagerly (with a once-per-frame warning) instead of refusing it, while still serving the variants that compiled.
**Summary.** Before this change, a frame Dynamo skipped mid-capture (a graph break it cannot resume from, a backend `SkipFrame`, a suppressed `Unsupported`/`UserError`) left no trace in the package. An installed or standalone artifact then refused every call the compiled variants missed ("no captured variant matches this call"), and a frame that fell back on its first call made the capture "incomplete", so the internal model had to pass `require_complete=False`. Now `PrecompileSummary.eager_fallbacks` names the flagged frames, `_RefuseRecompileCallback` and the standalone driver run their misses eagerly and warn once per frame, and they no longer count as uncovered. The eager miss lives only in the precompile load's own callback, not in a code-level `RUN_ONLY` strategy, so `DynamoCache`, other `torch.compile` callers and isolated regions are unaffected. An entry frame that fell back on its first call still has nothing to dispatch to, so `require_complete` (even when the entry is the only frame) and the artifact build both refuse it with a message naming the cause.
## The problem in detail
When Dynamo gives up on a frame during a capture (a graph break it cannot resume from, such as one inside a loop), it skips the whole code object and the capturing process runs every later call of it eagerly. Nothing in the package records that.
- A frame whose earlier calls had compiled keeps those variants, so an installed artifact refuses every other call of it ("no captured variant matches this call"), although the capture ran that very call eagerly.
- A frame that fell back on its first call holds no variant at all. It is served eagerly already, but `require_complete` counts it as an uncovered frame and refuses the artifact.
## What is recorded and how a load uses it
The package records Dynamo's own fallbacks: a compile that raises `SkipFrame` (Dynamo's own, or a backend declining the graph, such as Inductor's bf16 support check), or an `Unsupported` or `UserError` Dynamo suppresses by skipping the whole code object (a graph break it cannot resume from, or an entry generator, which it refuses to trace; a callee generator is refused before it gets an entry, so it is never flagged), flags the entry `eager_fallback`. Outside a precompile capture, which allows empty graphs, a frame that traced no ops raises `SkipFrame` too and is flagged; it has no variants, so install skips it as before. A recompile limit hit sets `RUN_ONLY` rather than a skip and stays a coverage gap (`truncated`). A compile error `suppress_errors=True` swallows also makes Dynamo skip the frame, but it is left unflagged so the failure stays visible as an uncovered frame rather than a silent eager fallback. These are exactly the soft failures Dynamo skips a frame for whatever `suppress_errors` says, so the flag means any frame Dynamo skips for the rest of the capture (a `torch._dynamo.skip_frame()` call included; the capture compiles in its own `isolate_recompiles` region, so the skip does not outlive it), and the docs say so. A backend that fails on the graph with one of the errors Dynamo falls back on (`exceptions_allowed_to_be_fallback`, such as `DynamicOutputShapeException`) arrives as such an `Unsupported` and is flagged too, and the cause lists name it.
- An installed artifact's `_RefuseRecompileCallback` is given the flagged frames' code objects: a call their variants miss runs eagerly for that call only (`ConvertFrameReturn(apply_to_code=False)`) instead of raising, and the variants keep serving. The check is a `_serve_miss` hook that `_FailOnRecompileCallback.__call__` calls (the base returns None, so the `fail_on_recompile` stance is unchanged), rather than an overriding `__call__`, because that method reads the missed frame's caller with `sys._getframe(1)` and an override would shift it. The hook runs first, so a miss of a flagged frame skips the trace-rules, tensor and nonrecursive-disable checks, none of which apply to a frame Dynamo compiled. A flagged frame without variants is skipped at install as before.
- The artifact's readable `FRAMES` list marks each flagged frame `# eager fallback`, which the serving notes point to. The `capture()` and `load()` docstrings, Note [precompile programming model] and the multigraph driver's docstring say a miss of such a frame runs eagerly rather than raising.
- The standalone driver runs a miss of a flagged frame as plain Python. It imports the warning helper after its Python and torch version checks, so an artifact loaded on another torch still fails with a clean `PrecompileError`.
- Both paths log a warning once per frame (`_warn_eager_fallback_miss`) naming it (under an installed artifact only that frame runs eagerly: its callees still dispatch to their own variants and refuse a miss; a standalone artifact runs the whole call as plain Python, since the frame's original bytecode never reaches its continuations), since every miss of a flagged frame runs eagerly, including call shapes the capture never ran, and a serving process should be able to see that it lost compiled code. The warning says how to get such calls compiled (recapture with a call that covers them, or fix what made Dynamo give up, via `TORCH_LOGS=graph_breaks`) and that `TORCH_LOGS=recompiles` does not report the miss. A flagged frame with no variants runs as plain Python from the start, as a trivial frame does, and logs nothing, since it has no compiled code to lose; the docs say so.
- `PrecompileSummary.eager_fallbacks` names the flagged frames (not a frame the capture serves as plain Python because its variants were lossy or bypassed: `_served_entry` clears the flag on it), and they no longer count as `uncovered_frames` or make a capture incomplete. The exception is an entry frame that fell back on its first call: it has no variant to dispatch to, so it stays in `uncovered_frames` and `complete` is False, matching the build's refusal below (a generator entry too, although Dynamo flags it before `_compile` and so it never gets a compile id). `require_complete` checks the entry's recorded flag (the same `_is_variantless_entry_fallback` the summary uses) and raises a refusal naming that cause, before every other gate (`require_no_risky_drops`, a raised call, no compiled code), each of which suggests a flag, so an entry that is the only frame gets it too, rather than the generic uncovered-frame message, and says `require_complete=False` does not get past it. The check runs whatever `require_complete` says, since no flag gets past it. `complete` also documents that it does not promise a load serves every call the capture ran: after a frame falls back the capture runs even a call an earlier variant matches eagerly and records nothing past it, so a load serves that call through the variant and a continuation or callee it reaches may refuse it.
- An entry frame Dynamo gave up on at its first call holds no variant, so the artifact build refuses it with a message naming the graph break, rather than the thin-wrapper advice; the message names the causes (a graph break it cannot resume from, a `skip_frame()` call, a frame Dynamo refuses to trace such as a generator, a suppressed `UserError`, or a backend `SkipFrame`). The build and the capture gate share that message (`_variantless_entry_fallback_message`) and the frame's `co_name`, and the gate's copy adds that the build refuses it regardless; since the gate now runs whatever `require_complete` says, the build refusal is now defence in depth; `_gated_summary` reads the entry `summary()` renders rather than fetching the cache entry again.
The eager miss lives in the precompile load's own callback, not in a code-level `RUN_ONLY` strategy, so nothing else changes: a `DynamoCache` (`caching_precompile`) package records the flag but its `install()` takes no hold for it and a miss recompiles as before, other `torch.compile` callers of the same code object are unaffected, and isolated regions see the same callback.
Unlike `torch.compile`, which after the fallback skips the frame for every call, including one an earlier variant matches, a load still serves the variants the capture compiled: their guards hold for exactly the calls they were compiled for.
## Why it is needed
A `torch.compiler.precompile` capture of an internal model shares one `forward` between two kinds of layer instance. The first layer's instances are FSDP units, which compile. Later layers' instances are plain modules whose children are FSDP units, and FSDP2's hooks break the graph inside the loop over them, so Dynamo gave up on that frame and ran it eagerly. The capture kept only the compiled variants, and every rank of the replay refused the first plain instance. Other frames the model's FSDP hooks broke on their first call (the layer dispatchers) made the capture "incomplete", so the model had to pass `require_complete=False`, which also hides genuinely missing coverage. Both stood in for what the capture already decided.
## Parent vs new
```python
@torch._dynamo.disable
def opaque(y): return y + 0
def dispatch(y, n):
for _ in range(n):
y = opaque(y) # graph break in a loop: Dynamo skips the whole frame
return y.sin()
# capture: dispatch(y, 0) compiles; dispatch(y, 1) falls back to eager
f = torch.compiler.precompile.load(artifact, cache) # in a fresh process
f(model, x) # served by the compiled variant (both)
f(model, x, 1) # parent: PrecompileError: no captured variant matches this call
# new: runs eagerly, as the capture did
# capture of only dispatch(y, 1):
# parent: PrecompileError: ... frame(s) that produced NO guarded code at all: ['dispatch']
# new: summary.eager_fallbacks == ('dispatch (...)',); the artifact is complete
```
## Tests
`test_an_installed_artifact_runs_a_frames_eager_fallback_eagerly` and `test_a_standalone_artifact_runs_a_continuations_eager_fallback_eagerly` (test_precompile.py) serve a fallen-back frame's variant and its eager miss through an installed artifact and through the standalone driver, assert the capture is `complete`, and replay the compiled call after the miss. `test_refuse_callback_runs_a_miss_of_an_eager_fallback_frame_eagerly` (test_package.py) pins that a plain `install()` takes no hold (a miss still reaches the callback) and that the refuse callback serves the variant and runs the miss eagerly only for a code in `eager_codes`, counting the backend's runs so a served variant is told apart from an eager run. `test_a_suppressed_user_error_is_an_eager_fallback` (which asserts the flagged exception really is a `UserError`), `test_a_backend_skip_is_an_eager_fallback` and `test_a_backend_fallback_error_is_an_eager_fallback` (a backend raising `DynamicOutputShapeException`, which runs once) pin the other ways a frame is flagged, and `test_a_recompile_limit_hit_or_suppressed_error_is_not_an_eager_fallback` the two exclusions. `test_an_installed_artifact_runs_an_entrys_eager_fallback_eagerly` serves an installed artifact whose entry frame compiled one variant and then fell back: the entry's miss runs eagerly while its callee `tagged_helper` is served by its own variant, and a call that misses the callee too is still refused, by the callee (the loader checks the refusal names `tagged_helper`). `test_no_dispatchable_graph_names_the_cause` covers the entry-frame refusal and `test_summary_complete_requires_every_term` that `eager_fallbacks` keeps a capture complete. `test_multigraph_frames_record_every_dynamo_frame` pins that a bypassed code is never an eager fallback. The installed test also checks the artifact's `FRAMES` row for `dispatch` carries the `# eager fallback` marker. `test_an_entry_dynamo_gave_up_on_at_its_first_call_is_uncovered` pins that a variant-less entry fallback whose callee compiled is uncovered: it asserts the summary lists the entry in `uncovered_frames`, not `eager_fallbacks`, and is not `complete`, and that `require_complete` refuses it. The refuse-callback test also asserts the eager miss warns exactly once, and `test_multigraph_driver_runs_an_eager_fallback_miss_eagerly` pins the same for the standalone driver in process. `test_a_lossy_graphless_frame_that_falls_back_is_not_an_eager_fallback` captures a graphless frame that drops a guard (so it is served as plain Python) and then falls back, and checks it is neither in `eager_fallbacks` nor flagged in the built artifact. `test_a_lone_entry_dynamo_gave_up_on_names_the_cause` captures an entry that is the only frame and falls back, alone and with a later call that raises, and `test_a_generator_entry_is_refused_naming_the_cause` a generator entry, with `require_complete` on and off (both hit the capture gate; `test_no_dispatchable_graph_names_the_cause` calls the build's refusal directly), and checks the generator entry is in `uncovered_frames` and the capture is not `complete`. The uncovered-entry test asserts the entry-specific refusal both with `require_complete` on and with `require_complete=False`. The generator-entry test is skipped on Python 3.10, where a generator frame first runs at `next()`, outside the capture. `test_a_standalone_artifact_runs_an_entrys_eager_fallback_eagerly` serves a standalone artifact whose entry frame compiled one variant and then fell back, and checks the driver runs a miss of the entry eagerly with its recorded defaults (a dtype miss that leaves two parameters to their defaults), then serves the variant again.
Test Plan:
```
python test/test_precompile.py -k eager_fallback -k dynamo_gave_up -k gave_up_on -k no_dispatchable_graph -k summary_complete -k lossy_graphless -k lone_entry -k generator_entry
python test/dynamo/test_package.py -k eager_fallback -k suppressed_user_error -k backend_skip -k recompile_limit_hit
```
Mutations: making the driver's `if eager_fallback:` branch dead fails the standalone test with "no captured variant ... matches this call"; not passing the flagged codes to `_RefuseRecompileCallback` fails the installed test; dropping the callback's `eager_codes` check fails the refuse-callback test; dropping the `record_eager_fallback` call in `_compile`'s `SkipFrame` handler fails the backend-skip test; recording only `Unsupported` fails the `UserError` test; dropping the `frame_exec_strategy is None` filter, or recording every suppressed error, fails the exclusions test; running the entry's eager miss without its recorded defaults fails the standalone entry test with "missing 1 required positional argument: 'scale'"; counting every eager-fallback entry as served fails the uncovered-entry test (`0 != 1` uncovered frames); dropping the warning fails the refuse-callback test with "no logs of level WARNING", and dropping the driver's warning fails the in-process driver test the same way; not clearing `eager_fallback` in `_served_entry` fails the lossy-graphless test (`1 != 0` eager fallbacks); disabling the capture gate's entry check fails the uncovered-entry test with the generic uncovered-frame message, and making it depend on `require_complete` again fails it with the build's message, which lacks "whatever require_complete says"; checking the entry after the no-compiled-code gates fails the lone-entry and generator-entry tests with "captured no compiled code", and after the raised-call gate fails the lone-entry test with "a captured call raised"; requiring a compile id for an uncovered frame again fails the generator-entry test (`0 != 1` uncovered frames); dropping the `FRAMES` marker fails the installed test; making `_serve_miss` serve every miss eagerly fails the installed entry test with "an uncovered call was served" (the callee's miss), and making it serve none fails it with "no captured variant matches this call".
This PR was authored with the help of an AI assistant (Claude).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@exported-using-ghexport
Pull Request resolved: https://github.com/pytorch/pytorch/pull/201314
Approved by: https://github.com/Mingming-Ding
ghstack dependencies: #201319