Building on top of the work to convert deinterleave + extend to tbl,
this PR will search for a reverse on the other side of binops that use
the extended value, which are themselves used in an fadd reduction. We
can then fold the reverse into the tbls and remove the reverse from the
loop.
`optimizeExtUses` rewrites uses of an extension's source in other blocks
to read `trunc` of the extension, which is only correct where the
extension isn't `poison`. A `zext nneg` can be valid only where it was
originally used, like behind a check that its source isn't negative. If
that check is one of the rewritten uses, it ends up reading `poison`,
and on `x86_64-pc-windows-msvc`, InstSimplify in WinEHPrepare's cleanup
folds it to true and causes a miscompile.
Drop the extension's poison-generating flags once a use has been
rewritten, like #72103 and #90382 did for similar cases in
`CodeGenPrepare`.
Fixes#234098.
The addressing-mode matcher only looks through `add`. When the address
is used in another block, the `or` is computed separately instead of
folding into the memory operand:
```
; before
leal 1(,%ecx,4), %ecx
movzbl (%eax,%ecx), %eax
; after
movzbl 1(%ecx,%eax,4), %eax
```
Treat `or disjoint` like `add`.
InstCombine canonicalizes `add X, C` to `or disjoint X, C` when the
operands have no common set bits
LLM-aided
This patch renames the functions in BasicBlock as follows because they
no longer check for debug intrinsics:
- getFirstNonPHIOrDbg -> getFirstNonPHIOrPseudoProbe
- getFirstNonPHIOrDbgOrLifetime -> getFirstNonPHIOrLifetime
- getFirstNonPHIOrDbgOrAlloca -> getFirstNonPHIOrAlloca
This patch also drops the SkipPseudoOp parameter from
getFirstNonPHIOrPseudoProbe and getFirstNonPHIOrLifetime because every
caller uses the default value of true.
Note that all three functions still return iterators with the head bit
unset so that insertions at the returned position are placed after any
DbgRecords attached to that instruction.
Assisted-by: Antigravity
Follow-up to #232210: drop perContext = false from CodeGenOptions and
MCCLOptions.
* TargetPassConfig stores `Opts` as a member; a callDefaultCtor
specialization replaces its default constructor.
* MCSchedModel::getResourceBufferSize takes the reservation station
scale factor. llvm-mca passes it through mca::PipelineOptions to the
ResourceManager and to its scheduler and timeline views. InstrBuilder
and TargetSchedModel only distinguish sizes -1, 0, and 1, which
scaling preserves, so they read the unscaled BufferSize.
This patch replaces Instruction::isDebugOrPseudoInst with
isa<PseudoProbeInst> at most call sites and drops the check altogether
where PseudoProbeInst is already handled or cannot occur.
Note that debug info is now represented with DbgRecord rather than
debug intrinsics outside the DirectX backend. None of the callers are
in the DirectX backend, so the DbgInfoIntrinsic check is dead.
Assisted-by: Antigravity
CodeGenPrepare::splitLargeGEPOffsets collects GEP candidates with large
constant offsets and rebases them to a shared common base, but only for
positive offsets (`ConstantOffset > 0`). Negative far-offset accesses
are left with independent `SUBXri` base materializations per access,
giving them different base registers and preventing the
`LoadStoreOptimizer` from pairing them into `LDP/STP`.
Relax the gate to `ConstantOffset != 0`, and extend
`AArch64TargetLowering::getPreferredLargeGEPBaseOffset` to handle
negative offsets using the same `HighPart = MinOffset & ~0xfff` rebase
as positive ones. For negative `MinOffset`, `HighPart` is the next lower
4096-aligned address, so all residuals (`Offset - HighPart`) are
non-negative and fit `LDR/STR`'s 12-bit unsigned scaled immediate. When
the residual also falls within `LDP/STP`'s 7-bit pairing range, `LSO`
pairs directly; otherwise `LSO`'s base-adjust (#223684) folds in an
extra `ADDXri` (or merges it into a preceding `SUBXri`/`ADDXri` when
adjacent), recovering the same instruction count as a direct rebase to
`MinOffset`.
This subsumes the MIR-level `shareBaseAddresses` pass (#223682) for the
negative-offset case, doing the equivalent work earlier at IR level. It
also benefits cases where LDP pairing isn't possible (non-adjacent
siblings) — the shared base alone saves instructions via CSE, which a
per-BB MIR-level pass can't do.
Example (from https://godbolt.org/z/sjo8eMW6j, 3 non-adjacent
negative-offset siblings, no LDP pairing):
; Before (upstream/main):
mov x8, #-4
mov x9, #-64
mov x10, #-1024
movk x8, #65531, lsl #16
movk x9, #65531, lsl #16
movk x10, #65531, lsl #16
ldr w8, x0, x8
ldr w9, x0, x9
ldr w10, x0, x10
; 9 instructions
; After (this patch):
sub x8, x0, #65, lsl #12 ; =266240
ldr w9, [x8, #4092]
ldr w10, [x8, #4032]
ldr w8, [x8, #3072]
; 4 instructions vs 9
Testing:
- Full AArch64 CodeGen test suite with assertions enabled (4301 tests, 0
regressions; 2 pre-existing upstream failures in
`arm64-codegen-prepare-extload.ll` and `and-sink.ll`, unrelated to this
change)
- `llvm-test-suite` (3535 passed, 0 failures, 0 errors)
- Bootstrap: building the entire LLVM+Clang with the patched clang
(assertions on, no crashes)
Note: `AArch64` is the only target that overrides
`shouldConsiderGEPOffsetSplit` and `getPreferredLargeGEPBaseOffset`, so
the gate change only affects AArch64 in practice.
Assisted-by: Claude (OpenCode)
Add new constructor overloads that accept just an insertion point
iterator, without the redundant basic block argument. The overload with
the basic block will be deprecated in a followup.
This is #228476, but for the constructor instead of `SetInsertPoint()`.
Mass replacement was AI assisted.
Rename the CodeGenPrepare options so that they start with cgp-, and turn
the -disable-* options into positive options that default to true:
The section prefix options (-profile-guided-section-prefix, etc.) are
kept for now.
Aided by Opus 5.5
Move the cl::opts of TargetPassConfig.cpp and CodeGenPrepare.cpp into
CodeGenOptions.td, private to lib/CodeGen; other files will follow.
-enable-machine-outliner (cl::ValueOptional) and -regalloc
(RegisterPassParser) stay cl::opt.
CodeGenPrepare and its addressing-mode helpers hold
`const CodeGenOptions &Opts`; TargetPassConfig functions read
CodeGenOptions::Global. getCGPassBuilderOption() converts BoolOrDefault
members to the cl::boolOrDefault and std::optional<bool> fields of the
public CGPassBuilderOption. -basic-block-section-match-infer, which was
not cl::Hidden, is now listed by -help-hidden only.
Aided by Opus 5.5
Pass insertion point directly to the ctor instead of calling
SetInsertPoint(). Also remove redundant SetCurrentDebugLocation()
calls, which SetInsertPoint() already does.
`store-extract` promotion was widening sdiv/srem to the vector type
without checking if thats safe or not... and INT_MIN lane / -1 is UB.
skip sdiv/srem by -1 in `shouldPromote`, rest still promotes fine.
Fixes#218570
Remove the shim methods from the TargetMachine's copy of the DataLayout,
which will soon be eliminated. The Module owns the authoritative DataLayout,
so callers should read the value from the contextual Module.
Completely unreasonably, Mips's ABI name can change the pointer size which
we probably should just not support. Many other triple checks will never be
correct. This avoids potential mismatches in these contexts, but I still expect
this to be widely broken.
Some of the TargetLowering constructor changes and AMDGPULegalizerInfo
changes are kind of annoying. We could pass in the DataLayout through the
subtarget constructors but it didn't seem worth the effort and the information
should be derivable from the triple anyway.
Co-authored-by: Claude (Claude-Opus-4.8) <noreply@anthropic.com>
This intrinsic was originally introduced to make sure that if we have a
dominating condition like `a == launder.invariant.group(a)`, we don't
end up replacing the latter with the former, as that would allow the
optimizer to assume that the memory stays invariant, despite going
through a launder operation. Wrapping both comparison operands in
strip.invariant.group avoids the issue, because replacing one stripped
pointer with another is safe.
The underlying issue was fixed in
https://github.com/llvm/llvm-project/pull/224281 (and prior work), which
ensures that the problematic replacement cannot happen in the first
place.
https://github.com/llvm/llvm-project/pull/225072 stopped emitting
llvm.strip.invariant.group on the clang side.
This PR now removes the intrinsic entirely. To be on the safe side, it
is upgraded to llvm.launder.invariant.group. For Clang's specific usage,
just dropping it would be fine.
Followup to https://github.com/llvm/llvm-project/pull/222314. That
PR only dropped the flags for the case where a new freeze instruction
was created, but of course we also need to do this if both operands
are constant and the freeze is dropped entirely.
We need to clear the samesign flag when pushing freeze through icmp.
In the last commit, I also adjusted the handling of FMF on fcmp to work
the same way -- previously the transform would be skipped instead if the
fcmp had fast math flags. I think just clearing the poison-generating
flags is preferable (and what we'd do anyway in InstCombine and
DAGCombine).
Fixes https://github.com/llvm/llvm-project/issues/222309.
Users of an Instruction have a special property: they're guaranteed to
be Instructions themselves. Add a specialized iterator to take advantage
of this property.
See the change to IVDescriptors.cpp for an example of how this
simplifies code.
FIX#179950
Extracted the core logic from CodeGenPrepare.cpp into a new optimizeBitCast function.
I have also included tests covering both legal and illegal types.
While iterating users of a {s|z}ext's source operand in optimizeExtUses
we explicitly replace the use with a trunc instruction thereby
invalidating iterator of the loop. Update the loop to use
make_early_inc_range to avoid invalidation.
getLibFunc returned a bool and wrote the result to a LibFunc output
argument, which is only set when the lookup succeeds. Callers that
ignored the result read an uninitialized LibFunc, and a few in-tree
callers did exactly that.
Return the LibFunc instead, and NotLibFunc when the name is not a known
library function, so there is no way to get an uninitialized value out
of it.
Assisted-by: Opus
The test case crashes on the release/23.x. What happens is the
`BranchProbabilityInfo` becomes outdated because of a change of CFG in
`bypassSlowDivision` but no update to `BranchProbabilityInfo`.
So when it arrives to `SplitIndirectBrCriticalEdges` and hit the assert
in `setEdgeProbability`, because the Branch probabilities no-longer make
sense.
BranchProbabilityAnalysis uses CycleInfo, but BFI doesn't, causing the
somewhat redundant construction of an extra LoopInfo. Avoid this by
porting BFI to use CycleInfo.
This requires a minor change to the BFI implementation to avoid
incorrect results with irreducible loops that show up as nested but
don't show up as nested loops -- such cycle entries are skipped now.
(Such an entry heads a loop the cycle absorbed, and which entry keeps a
nested cycle of its own depends on the order the search found the entries
in.) @crossloops reaches c1 and c2 alike, yet only c1 heads a nested cycle,
so seeding it gave c1 a loop scale c2 never got and their frequencies came
out 0.68571 and 1.1429 where the test derives 1.0 for both. Represent none
of those entries and leave the region to computeIrreducibleMass, which
decomposes it from the reverse postorder as it does when LoopInfo finds
no natural loop there.
Passing the parent down also fixes the loop nest: &Loops.back() is whichever
loop was created last, not the enclosing one, and every crash on this branch
came from that.
Co-authored-by: Fangrui Song <i@maskray.me>
Instead of computing a cycle info (SccInfo) internally in
BranchProbabilityInfo, use CycleInfo and remove the use of LoopInfo.
After recent improvements to CycleInfo, the extra analysis is cheaper
than computing the same information internally.
Replacing LoopInfo with CycleInfo has some impliciation that the
loop/cycle forest can differ if the header of an irreducible loop (which
is ignored by loop info) also happens to be the header of a different
natural loop. (Simple example: S -> A, B; A -> A, B; B -> A, B --
LoopInfo will find {{A}, {B}}; CycleInfo will find {A, {B}}].) In these
cases, the default weights will differ.
This change also makes the handling of entering/exiting edges
consistent between reducible and irreducible loops, therefore,
there are also minor changes for reducible CFGs.
This patch fixes a compile-time issue in CodeGenPrepare for huge
functions.
`DeleteDeadPHIs` may repeatedly prove overlapping PHI chains non-dead.
For very large functions, many PHIs can share the same non-dead def-use
suffix, causing the same suffix to be scanned many times.
Add an `KnownNonDeadPHIs` cache to `RecursivelyDeleteDeadPHINode`
and `DeleteDeadPHIs`. When a chain is proven non-dead, visited PHIs are
recorded so later queries can stop once they reach one of them.
This reduces the pathological CodeGenPrepare case from ~30mins to ~30s.
splitBranchCondition computes new branch weights after splitting an
and/or condition into two branches, but then passed the original weights
to createBranchWeights at each metadata update. The recomputed values
were discarded.
Pass the scaled NewTrueWeight/NewFalseWeight values when installing
metadata on both generated branches.
This bug was found by a large run of Opus 4.7 looking for bugs in LLVM.
splitMergedValStore notices when you do e.g.
z = x | (y << 32)
store z
and may split this up into 32-bit two stores, of x and y, depending
e.g. on the type of x and y.
It skips this optimization for volatile stores, but currently does NOT
skip it for atomics (!!). So an atomic store can be split up into two
(non-atomic!) stores.
This bug was found by a large run of Opus 4.7 looking for bugs in LLVM.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CodeGenPreparePass can't declare ProfileSummaryAnalysis as required,
because PSA is a module-level analysis, but CFP is a function-level pass.
Therefore it accesses PSA using getCachedResult, and PSA might be null.
In practice this doesn't happen, because the CGP pass pipeline
preparation code ensures that PSA is present. But if you invoke
CGP via opt -passes=codegenprepare, then it's not
there, and we segfault.
Fix for https://github.com/llvm/llvm-project/issues/173360.
This bug was found by a large run of Opus 4.7 looking for bugs in LLVM.
After #172329, we noticed that some sources compiled with MSan take
1000x longer to compile. This is caused by quadratic complexity in
tryToSinkFreeOperands, which can be called on a significant number
of instructions within huge basic blocks.
This inefficiency was introduced in 9cfa9b4, which manually iterates
and creates a DenseMap of entire basic blocks for each interesting
instruction.
This patch avoids the manual ordering by using
Instruction::comesBefore(), which provides the exact same
ordering much more efficiently.
Alternative approach to the same goals as #162407
This takes `TargetLoweringBase::getLoadExtAction`, renames it to
`TargetLoweringBase::getLoadAction`, merges `getAtomicLoadExtAction`
into it, and adds more inputs for relavent information (alignment,
address space).
The `isLoadExtLegal[OrCustom]` helpers are also modified in a matching
manner.
This is fully backwards compatible, with the existing `setLoadExtAction`
working as before. But this allows targets to override a new hook to
allow the query to make more use of the information. The hook
`getCustomLoadAction` is called with all the parameters whenever the
table lookup yields `LegalizeAction::Custom`, and can return any other
action it wants.
BranchProbabilityInfo will compute it's own dominator tree and
post-dominator tree if none is specified; avoid this by using the
analysis manager/pass manager to get the analysis, which will reuse the
previously computed DomTree.
Largely a straight-forward replacement with occasional simplifcations.
For AMDGPU, I assumed that unconditional branches are always uniform and
therefore "simplified"/changed AMDGPUAnnotateUniformValues to only
annotate conditional branches.
Target-specific FastISel only selects conditional branches,
unconditional branches are already handled by the non-target-specific
code.
BranchInst currently represents both unconditional and conditional
branches. However, these are quite different operations that are often
handled separately. Therefore, split them into separate opcodes and
classes to allow distinguishing these operations in the type system.
Additionally, this also slightly improves compile-time performance.
https://github.com/llvm/llvm-project/pull/178895 caused a clang
crash(see https://lab.llvm.org/buildbot/#/builders/210/builds/8229),
reverted in 6d52d2683c.
The crash is assertion `DT && "DT should be available to update
LoopInfo!"' failed.
https://github.com/llvm/llvm-project/blob/ad8d5349d46734826aaeae4a2ebdc6f427a5bad8/llvm/lib/Transforms/Utils/BasicBlockUtils.cpp#L1106
```
#7 0x00007f5a380254e3 __assert_perror_fail (/usr/lib/libc.so.6+0x254e3)
#8 0x0000563df5d8fde1 UpdateAnalysisInformation(llvm::BasicBlock*, llvm::BasicBlock*, llvm::ArrayRef<llvm::BasicBlock*>, llvm::DomTreeUpdater*, llvm::DominatorTree*, llvm::LoopInfo*, llvm::MemorySSAUpdater*, bool, bool&) BasicBlockUtils.cpp:0:0
#9 0x0000563df5d8f3bb llvm::splitBlockBefore(llvm::BasicBlock*, llvm::ilist_iterator_w_bits<llvm::ilist_detail::node_options<llvm::Instruction, true, false, void, true, llvm::BasicBlock>, false, false>, llvm::DomTreeUpdater*, llvm::LoopInfo*, llvm::MemorySSAUpdater*, llvm::Twine const&)
#10 0x0000563df5d8cb08 llvm::SplitEdge(llvm::BasicBlock*, llvm::BasicBlock*, llvm::DominatorTree*, llvm::LoopInfo*, llvm::MemorySSAUpdater*, llvm::Twine const&)
#11 0x0000563df4ff5b59 (anonymous namespace)::CodeGenPrepare::splitLargeGEPOffsets()::$_1::operator()(long, llvm::Value*, llvm::GetElementPtrInst*) const CodeGenPrepare.cpp:0:0
#12 0x0000563df4fc0ec8 (anonymous namespace)::CodeGenPrepare::_run(llvm::Function&) CodeGenPrepare.cpp:0:0
#13 0x0000563df4fbb36c (anonymous namespace)::CodeGenPrepareLegacyPass::runOnFunction(llvm::Function&) CodeGenPrepare.cpp:0:0
```
I think this happened when we get DominatorTree with `DT.get()` in
`splitLargeGEPOffsets()` but `DT.reset()` already setting it to nullptr
in
https://github.com/llvm/llvm-project/blob/ad8d5349d46734826aaeae4a2ebdc6f427a5bad8/llvm/lib/CodeGen/CodeGenPrepare.cpp#L660.
To fix this assertion failure, use `getDT()` for
`splitLargeGEPOffsets()` to build the DominatorTree if it is set to
nullptr by `DT.reset()`.
I don't have a RSIC-V environment, so no reproducer. Checked that the
crash is fixed by rerunning buildbot with this patch
https://lab.llvm.org/buildbot/#/builders/210/builds/8248
We never need to use this conditionally (and it doesn't really make
sense, as the behavior is substantially different). Force the use of
separate APIs instead of a boolean argument.
Returns uint64_t to simplify callers. The goal is eventually replace
getValueType with this query, which should return the known minimum
reference-able size, as provided (instead of a Type) during create.
Additionally the common isSized query would be replaced with an
isExactKnownSize query to test if that size is an exact definition.
OptimizeNoopCopyExpression was sinking same-type bitcasts (e.g. bitcast
i32 to i32) which would then be reintroduced by optimizePhiType, causing
an infinite loop.
Fix by adding a check (PhiTy == ConvertTy) in optimizePhiType to skip
the conversion when types are already identical.
Fixes#176688.
- Remove pass initialization calls from pass constructors.
- For some passes, add the initialization to `initializeCodeGen` or
`initializeGlobalISel`.
- Remove redundant initializations from llc and X86 target for some
passes.
Replace patterns that manually compute allocation sizes by multiplying
getTypeAllocSize(getAllocatedType()) by the array size with calls to the
getAllocationSize(DL) API, which handles this correctly and concisely,
returning nullopt for VLAs.
This fixes several places that were not accounting for array allocations
when computing sizes, simplifies code that was doing this manually, and
adds some explicit isFixed checks where implied convert was being used.
This PR is because now that we have opaque pointers, I hate that some
AllocaInst still has type information being consumed by some passes
instead of just using the size, since passes rarely handle that type
information well or correctly. I hope this will grow into a sequence of
commits to slowly eliminate uses of getAllocatedType from AllocaInst.
And similarly later to remove type information from GlobalValue too (it
can be replaced with just dereferenceable bytes, similar to arguments).
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The test case's call instruction was being sank past the point where the
memory
it accessed was valid. Add a check that CGP does not try to sink
instruction that
might be invalid to move.
Fixes#176095
Graham Hunter