Module::getExceptionModel() now falls back to the triple's default
model instead of falling back to the TargetMachine.
The pass pipeline is built before the module is known, so schedule each
EH preparation pass when the target supports a model, and let the pass
check the module's model. Add TargetMachine::supportsExceptionModel so
the pass config can skip definitely irrelevant passes. SjLjEHPrepare is
now scheduled on every target with exception handling, and WinEHPrepare
only on COFF and Wasm targets. WinEHPrepare and WasmEHPrepare are
scheduled for Wasm by the base pass configs, replacing the WebAssembly
override.
This is almost the last step towards removing
TargetOptions::ExceptionModel.
Co-authored-by: Claude Opus <noreply@anthropic.com>
An IMPLICIT_DEF is erased by setting undef an all uses of the def in a
block. If there are uses outside of the block, it is kept. In the case of
physical registers, this was left as an unused def without a dead flag.
Look at the liveins of the successors to check if it's live, and add the
dead flag if not. We probably should just erase the instruction, but I've
left that for a future change.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
SplitCriticalEdge scanned every operand of every PHI in Succ twice per
split: once in replacePhiUsesWith to retarget the incoming block, then
again with LiveIntervals to find the registers now coming from the new
block. Collect the source registers while retargeting the operands, so
each split scans Succ's PHIs only once.
Instructions retired in phi-node-elimination, x86_64 -O3, on a generated
chain of N compare blocks branching to a shared PHI block:
N before after after/before
1k 89,072,376 69,208,524 0.78
2k 330,305,622 251,128,551 0.76
4k 1,140,509,526 818,125,849 0.72
8k 4,134,353,260 2,851,571,872 0.69
16k 15,538,463,961 10,437,522,736 0.67
gcc-c-torture compile/20001226-1.c (liveintervals,phi-node-elimination
on the pre-PHIElimination MIR): 1,131,964,337 -> 813,420,169.
Whole llc on the 16k input: 78,266,594,024 -> 73,171,746,875.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fill in the new block's live-out set in the same loop that updates the
live intervals, instead of walking the original block's set a second
time with another liveAt query per register. A register is live out of
the new block if it is a PHI source or is live into Succ, which the
update already checks.
Instructions retired in phi-node-elimination, x86_64 -O3, on a generated
chain of N compare blocks branching to a shared PHI block:
N before after after/before
1k 94,119,121 88,904,465 0.94
2k 353,559,942 330,642,394 0.94
4k 1,201,233,649 1,136,359,710 0.95
8k 4,305,218,922 4,136,184,186 0.96
16k 16,076,293,247 15,539,552,966 0.97
gcc-c-torture compile/20001226-1.c (liveintervals,phi-node-elimination
on the pre-PHIElimination MIR): 1,192,377,705 -> 1,132,130,646.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
To update LiveIntervals, SplitCriticalEdge checked every virtual
register in the function for liveness at the end of the split block.
That made PHIElimination's edge splitting O(splits * vregs).
PHIElimination now computes the set of virtual registers live out of
each block before the first split and passes it through
SplitCriticalEdgeAnalyses. Only those registers are visited, and the set
for the new block is added once the intervals are updated.
This essentially resurrects the per-block sparse register sets used to
update LiveVariables, which were removed in #228618 and #230147, applied
to the LiveIntervals update instead.
Instructions retired in phi-node-elimination, x86_64 -O3, on a generated
chain of N compare blocks branching to a shared PHI block:
N before after after/before
1k 224,562,067 94,012,986 0.42
2k 862,552,029 351,548,735 0.41
4k 3,279,201,709 1,199,726,325 0.37
8k 12,769,826,390 4,296,440,567 0.34
16k 50,117,918,793 16,061,464,876 0.32
gcc-c-torture compile/20001226-1.c (liveintervals,phi-node-elimination
on the pre-PHIElimination MIR): 3,274,667,934 -> 1,193,137,243.
Whole llc on the 16k input: 112,903,512,059 -> 78,878,832,446.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Nothing queries the LiveVariables analysis anymore, so remove uses that only
ran when the analysis was available. Remove the verifier handling and the
update in SplitCriticalEdge.
Co-authored-by: Claude (Opus 4.8) <noreply@anthropic.com>
SplitCriticalEdge collects the PHI sources coming from the new block so
the trimming loop below does not undo the segment just added for them.
An undef operand gets no segment, but was still added to the set, so a
register that is only an undef PHI operand on the split edge kept the
stale extension of its live range through the new block.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
SplitCriticalEdge trims the stale extension of a live range into the newly created
block. The subrange guard used overlaps(StartIndex, EndIndex), which only
happens to be equivalent to the containment removeSegment requires.
StartIndex is an index created for the new block, and EndIndex is the block end
index the live-out segments already end at, so no pre-existing segment endpoint
falls between them.
Check liveAt(PrevIndex) instead, mirroring the same condition on the main range.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Add guard to skip subranges that not live in the interval.
fix to: https://github.com/llvm/llvm-project/pull/227252
Passed local libc tests:
```
Total Discovered Tests: 814
Passed: 814 (100.00%)
```
assisted by: cursor
Fix missing serialization of another field. The alignment was
already handled. The name is a bit verbose. Some places call it
"MaxSkip" which matches the name of the 2nd operand to the .p2align
directive this corresponds to.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This fixes verifier errors when LiveIntervals is run before phi
elimination.
When extending a live interval across a newly split critical edge for a
PHI source operand, the subrange segments were added using the VNInfo of
the main range instead of the subrange's own value. Look up the
subrange's value at the predecessor's last index, as the other subrange
loop in the same function already does, and skip subranges with no value
there.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Resolve the exception model as the exception-model module flag if
present, otherwise the TargetOptions default.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply@anthropic.com>
After critical edge spliting replaces a terminator, clear referenced
physreg ranges. The block's terminators may be replaced, and may
reference physical registers (e.g. WebAssebly's ARGUMENTS. When
LiveIntervals is available, repairIntervalsInRange only repairs virtual
registers, so the tracked physreg units were stale.
It feels like a bug that repairIntervalsInRange doesn't touch physregs,
but I've left that for a future change.
Co-Authored-By: Claude claude-opus-4.8 <noreply@anthropic.com>
This relands 85e9030993 with fixes for
the buildbot failures that prompted its revert.
Move the MachineModuleSlotTracker debug-location query out of line so
DILocation is complete when SmallPtrSet pointer traits are instantiated
by MSVC.
The legacy DXIL pretty-printer renumbers persistent metadata IDs, which
changes module print state. Make the pass report the modification and
register it as a transformation rather than an analysis.
Repeated pass and debug printing currently rebuilds metadata slot
numbers by scanning the module. On large modules, this repeated setup
can take more time than printing the requested function.
Store a persistent print ID in each metadata node and use that ID
whenever metadata is printed. Before final assembly output, renumber the
IDs into canonical compact order so normal textual IR output remains
unchanged.
This keeps metadata numbers stable across pass snapshots while avoiding
repeated module-wide metadata scans.
Buildbot failure:
https://lab.llvm.org/buildbot/#/builders/2/builds/60795
Buildbot failure:
https://lab.llvm.org/buildbot/#/builders/187/builds/23268
Repeated pass and debug printing currently rebuilds metadata slot
numbers
by scanning the module. On large modules, this repeated setup can take
more time than printing the requested function.
Store a persistent print ID in each metadata node and use that ID
whenever
metadata is printed. Before final assembly output, renumber the IDs into
canonical compact order so normal textual IR output remains unchanged.
This keeps metadata numbers stable across pass snapshots while avoiding
repeated module-wide metadata scans.
MachineBasicBlock::splitAt() moves the tail of a block -- including any
call instructions carrying register-mask operands -- into a newly
created block. When LiveIntervals is attached it calls
LIS->insertMBBInMaps() for the new block, which records the block with
zero regmask slots. That is only correct for a fresh, empty block:
because the tail (and its regmasks) was *moved* out of the original
block, the per-block RegMaskBlocks index for both blocks is left stale,
so checkRegMaskInterference() can miss a call clobber and the register
allocator can assign a live value to a clobbered physical register.
Add a LiveIntervals::splitAt(Orig, SplitBB) entry point that inserts
SplitBB into the SlotIndexes/regmask maps and then re-slices the moved
regmask slots out of Orig's RegMaskBlocks entry into SplitBB's
(RegMaskSlots is sorted, so the slots that moved are those at/after
SplitBB's start index). MachineBasicBlock::splitAt() now calls it in
place of insertMBBInMaps().
The re-slicing helper (reassignRegMaskSlots) and the shared
map-insertion helper are private; insertMBBInMaps() is documented as
expecting an empty block and, under EXPENSIVE_CHECKS, asserts the block
has no regmask operands so that misuse is caught.
Adds a LiveIntervals unit test (SplitAtMovesRegMaskInterference) that
splits a block so a value becomes block-local yet still crosses a call
regmask, and checks that checkRegMaskInterference() still reports the
clobber.
**AI assistance disclosure** (per the LLVM AI Tool Use Policy): the MIR
test and parts of this commit message were drafted with an AI tool
(Claude, Anthropic). I have reviewed and verified all code and tests and
take full responsibility for this contribution.
Currently this is sourced from TargetOptions. In the future
this should come from program state, so needs to be passed in
from a value derived from the module flag. Thread through the
argument for future use. For now it's still sourced directly
from the TargetOptions.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply@anthropic.com>
`MachineBasicBlock::printName` writes the IR basic-block name after
`bb.<N>.` unquoted, and the
MIR lexer reads it back with `isIdentifierChar` only. If the name
contains anything else -- a
comma, say -- the name is cut short on the way in and the remainder is
treated as syntax, so `llc`
cannot parse the MIR that `llc` just wrote:
```console
$ llc -mcpu=gfx90a -stop-before=greedy bb-comma-name.ll -o out.mir
$ llc -mcpu=gfx90a -x mir -start-before=greedy out.mir -o /dev/null
error: out.mir:170:8: expected ':'
```
because the printer emitted
```
bb.1., bb71:
```
This breaks the `-stop-before` / `-start-before` round-trip normally
used to reduce a backend bug
to a MIR test. Fortran front ends emit such names routinely, which is
where I hit it. Fixes#212785.
Both sides need the change:
* `MachineBasicBlock::printName` now uses `printLLVMNameWithoutPrefix`,
which quotes and escapes
only when the name is not a plain identifier. Names that already
round-tripped are printed
exactly as before, so existing MIR output is unchanged.
* `maybeLexMachineBasicBlock` accepts a quoted name after `bb.<id>.`,
via the same
`lexStringConstant` / `unescapeQuotedString` pair that `lexName` already
uses for `%ir-block.`
and that the printer's `printEscapedString` output matches.
The `%ir-block.` reference printed by `PrintBBRef` had the same hole and
is fixed the same way.
The parse side there was already correct, since `maybeLexIRBlock` goes
through `lexName`.
### Testing
New test round-trips a module whose block names contain commas and
spaces, checking both that the
name is quoted on the way out and that the result parses back.
Verified the test fails without the patch: the unpatched `llc` emits
`bb.1., bb71:` and then
rejects it with `expected ':'`.
`llvm/test/CodeGen/X86`, `llvm/test/CodeGen/AMDGPU` and
`llvm/test/CodeGen/MIR` are clean, 10902
tests, no failures. Those are the targets my build enables; I have not
run the other backends.
---
Parts of this change were written or audited with Claude Code. I have
reviewed all of it and take
full responsibility for the contribution. See
`llvm/docs/AIToolPolicy.md`.
Breaks some buildbots with gcc. Looks like constexpr member definitions
need an extern for old gcc otherwise they get local linkage...
Reverts llvm/llvm-project#208407
Largely by converting relevant functions or variables to constexpr,
changing the type, converting to static locals, or by outright removal.
PunnedPointer assignment can never be constexpr, because std::memcpy is
not constexpr and type punning is not permitted in a constant-evaluated
context.
In many passes involving CFG updates, it is a common pattern to process
the Predecessors vector from back to front for efficiency. However, the
current forward search in removePredecessor often results in an O(N)
complexity.
So this patch tries to change the search logic to a reverse search to
better align with the majority of actual CFG manipulation scenarios.
And in a real-world case (with ~16k predecessors), this modification can
help to reduce the execution time of the BranchFolder pass from
166.4951s to 6.0717s.
---------
Co-authored-by: Reid Kleckner <rkleckner@nvidia.com>
Block numbers are updated too frequently, which makes it difficult to
keep analyses up to date. Therefore, introduce a second number per basic
block that is used for analyses and is renumbered less often. This frees
analyses from providing somewhat efficient facilities for dealing with
changed block numbers, making it simpler to implement in e.g. LoopInfo
or CycleInfo.
(Currently, "less often" means not at all, but we might want to renumber
after certain passes if the numbering gets too sparse and no analyses
are preserved anyway.)
When we introduced a more general use of block numbers some time ago,
using the existing numbers seemed to be a somewhat obvious choice, but I
now think that this was a bad decision, as it conflates a number that is
used for ordering with a number that should be more stable.
MachineBasicBlock isn't particularly size-optimized and there's a fair
amount of padding where we can add another number.
There should be no performance impact,
Fixes https://github.com/llvm/llvm-project/issues/170051
The previous implementation allows splitting the successor if it's the
loop header, regardless of whether the terminator of `this` is
analyzable.
This addresses a performance issue for our downstream GPU target that
sets requiresStructuredCFG to true. The issue is that EarlyMachineLICM
pass does not hoist loop invariants because a critical edge is not
split.
The critical edge's destination a loop header. Splitting the critical
edge will not break structured CFG.
Add a nvptx test to demonstrate the issue since the target also
requires structured CFG.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Post-RA machine sinking could sink a copy of sub-register into
a successor. However, the sub-register might not be removed from the
live-in bitmask of its super register in successor and then a later
pass, e.g, if-converter, may add an implicit use of the register from
live-in resulting in an use of an undefined register. This change makes
sure subrange of live-ins from super register could be removed as well.
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
Part of the coverage-tracking feature, following #107279.
In order for DebugLoc coverage testing to work, we firstly have to set
annotations for intentionally-empty DebugLocs, and secondly we have to
ensure that we do not drop these annotations as we propagate DebugLocs
throughout compilation. As the annotations exist as part of the DebugLoc
class, and not the underlying DILocation, they will not survive a
DebugLoc->DILocation->DebugLoc roundtrip. Therefore this patch modifies
a number of places in the compiler to propagate DebugLocs directly
rather than via the underlying DILocation. This has no effect on the
output of normal builds; it only ensures that during coverage builds, we
do not drop incorrectly annotations and therefore create false
positives.
The bulk of these changes are in replacing
DILocation::getMergedLocation(s) with a DebugLoc equivalent, and in
changing the IRBuilder to store a DebugLoc directly rather than storing
DILocations in its general Metadata array. We also use a new function,
`DebugLoc::orElse`, which selects the "best" DebugLoc out of a pair
(valid location > annotated > empty), preferring the current DebugLoc on
a tie - this encapsulates the existing behaviour at a few sites where we
_may_ assign a DebugLoc to an existing instruction, while extending the
logic to handle annotation DebugLocs at the same time.
These are identified by misc-include-cleaner. I've filtered out those
that break builds. Also, I'm staying away from llvm-config.h,
config.h, and Compiler.h, which likely cause platform- or
compiler-specific build failures.
- Move `MIPrinter` class to anonymous namespace, and remove it as a
friend of `MachineBasicBlock`.
- Move `canPredictBranchProbabilities` to `MachineBasicBlock` and change
it to use the new `BranchProbability::normalizeProbabilities` function
that accepts a range, and also to use `llvm::equal()` to check equality
of the two vectors.
- Use `ListSeparator` to print comma separate lists instead of manual
code to do that.
