The exception model now comes only from the "exception-model" module
flag. Remove the TargetOptions field and its copy into MCAsmInfo, so
MCAsmInfo only knows the target's default model.
The WebAssembly model checks move from the TargetMachine constructor
into the Emscripten EH/SjLj lowering pass. Clang relied on the backend
to reject non-wasm models for wasm targets, so it now rejects them itself.
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
MachineFunction::clear() deletes the instructions but keeps the call
site info and called globals that refer to them. After
ResetMachineFunction, which GlobalISel uses to fall back to
SelectionDAG, the entries point to deleted instructions, and printing
the MIR hangs. Clear both maps with the rest of the function, as is done
for the variable debug info that GlobalISel records.
No GlobalISel target records call site info yet, so the test resets a
MIR function marked failedISel.
This reapplies commit 21740d6b79 (reverted
in
627b8cf3d4) with test updates for 5fda7711c6 (#233298), which landed
shortly beforehand and updated the IR verifier to reject functions that
carry both the `hot` and `cold` attributes. Remove the `hot`+`cold` test
cases in `hot-attribute.ll` and `ProfileSummaryInfoTest.cpp`, and update
the comment in `ProfileSummaryInfo.cpp` accordingly.
Original commit message:
LangRef documents the `hot` function attribute as taking precedence over
profile information and acting as a manual alternative to PGO hotness
data when training inputs lack coverage for hot functions. However,
ProfileSummaryInfo only checked `Attribute::Cold` (in
isFunctionEntryCold) and never consulted `Attribute::Hot`. As a result,
functions marked `hot` that had zero entry counts in PGO profiles (for
example, newly introduced functions not executed during profile
collection) were still treated as cold by PSI queries, causing passes
such as TailCallElim and PGOForceFunctionAttrs to apply cold-function
heuristics.
Check `Attribute::Hot` in `isFunctionEntryHot`,
`isFunctionHotInCallGraph`, `isFunctionColdInCallGraph`, and
`isFunctionEntryCold` before checking `hasProfileSummary()`. This
ensures
the attribute is honored both with and without profile data.
Assisted-by: Gemini
LangRef documents the `hot` function attribute as taking precedence over
profile information and acting as a manual alternative to PGO hotness
data when training inputs lack coverage for hot functions. However,
ProfileSummaryInfo only checked `Attribute::Cold` (in
isFunctionEntryCold) and never consulted `Attribute::Hot`. As a result,
functions marked `hot` that had zero entry counts in PGO profiles (for
example, newly introduced functions not executed during profile
collection) were still treated as cold by PSI queries, causing passes
such as TailCallElim and PGOForceFunctionAttrs to apply cold-function
heuristics.
Check `Attribute::Hot` in `isFunctionEntryHot`,
`isFunctionHotInCallGraph`, `isFunctionColdInCallGraph`, and
`isFunctionEntryCold` before checking `hasProfileSummary()` or
`Attribute::Cold`. This matches the precedence in CodeGenPrepare and
ensures the attribute is honored both with and without profile data.
Assisted-by: Gemini
Whether unwind information is encoded with SEH opcodes (.seh_*, .pdata
and .xdata) is a property of the object format. Whether a function uses
that encoding depends on the exception model, but usesWindowsCFI()
conflated the two. Make it report only the format, and add
MachineFunction::usesWindowsUnwindInfo() for the combined question and
needsWinCFI() for the common case that also requires an unwind table
entry.
This moves the exception model check to the point of use, in preparation
for removing ExceptionModel from TargetOptions. The assembler now
accepts
.seh_* directives whenever the object format supports them (i.e., MC
only tools will no longer have a notion of ExceptionModel).
Co-authored-by: Claude Opus <noreply@anthropic.com>
Many call sites of the `Align` constructor uses it as `Align(1ULL << n)`, and
the constructors takes the `Log2` of the value. Add a new static function to
allow `Align::fromLog2(n)` and update old call sites. This avoids the
unnecessary `Log2(1ULL << n)` computations when constructing `Align`s.
These used to depend on a TargetOptions field, which was removed
at some point. These are now only depend on MachineFunction,
so move the attribute check there.
Co-authored-by: Claude (Claude-Opus-4.8)
Previously we printed something like `cp#2: splat (i8 -8), align=64`
which doesn't give any indication of total size.
This would have made debugging
https://github.com/llvm/llvm-project/pull/220448 easier.
This is the follow up for commit
https://github.com/llvm/llvm-project/commit/0fc5d0ab0cea3afe32592f24af17b5d7e02c9dfe,
which added IR support for cache hint metadata, as described in
https://discourse.llvm.org/t/rfc-composable-and-extensible-memory-cache-control-hints-in-llvm-ir/89443.
See previous https://github.com/llvm/llvm-project/pull/175901 that I
closed in favor of this one.
This PR adds support in SelectionDAG and lowering in NVPTX.
Supported cache hints:
L1 eviction: L1::evict_first, L1::evict_last, L1::evict_unchanged,
L1::no_allocate (requires SM 70+)
L2 eviction: L2::evict_first, L2::evict_last (requires SM 70+)
L2 prefetch: L2::64B, L2::128B (SM 75+), L2::256B (SM 80+)
L2::cache_hint with 64-bit cache policy descriptor (SM 80+, PTX 7.4+)
L1 eviction: L1::evict_first, L1::evict_last, L1::evict_unchanged,
L1::no_allocate (requires SM 70+ and PTX 7.4+)
L2 eviction: L2::evict_first, L2::evict_last (requires SM 100+, PTX
8.8+, and a 256-bit .v8.b32 or .v4.b64 memory operation)
L2 prefetch: L2::64B, L2::128B (requires SM 75+ and PTX 7.4+), L2::256B
(requires SM 80+ and PTX 7.4+)
L2 cache policy: L2::cache_hint with a 64-bit cache policy descriptor
(requires SM 80+ and PTX 7.4+)
I know this is a fairly large PR, but I needed to plug the support all
the way through the backend. I supposed I could add machinery in
SelectionDAG as an initial PR, but I'm not sure if that makes sense. I
implemented lowering on load + store + memcpy. I supposed memcpy can go
in a follow-up, but it doesn't really reduce the size that much.
TODO:
- implement support in global ISEL
- Implement lowering in NVPTX for atomics + other intrinsics
(llvm.masked.load/store)
- Better handling of cache hint metadata in legalization. Today we will
often drop the metadata. If we legalize a wide vector load into a bunch
of smaller loads, for example, we should preserve the metadata.
- Preserve L2 eviction metadata for suitably aligned, 32-byte-multiple
memcpy operations
Design decisions:
- I encoded the metadata in the MachineMemOperand. I hope it's ok to
increase the size. I think this is the best place because similar
information (atomic info, range metadata, etc...) is all stored there.
- When the metadata node values (strings which are target dependent) are
not valid for NVPTX, I call `ctx.emitError` to emit a diagnostic. I
think this is best because we wouldn't want front-ends to accidentally
emit incorrect metadata and for it to silently be dropped.
Co-authored-by: Fiigii <feipeng.compiler@gmail.com>
Assisted by AI
When -fexperimental-call-graph-section flag is set, it adds type
metadata to all the functions whose addresses are taken and does not
have local linkage. When this flag is set along with CFI, the type
metadata is added to all the vtable functions including destructors.
This changes which functions are to be treated as CFI functions and
includes such vtable entries to become part of the CFI check jumptables.
To disambiguate intentions of CFI and callgraph mechanisms, this patch
renames metadata set by callgraph mechanism to !callgraph
(MD_callgraph). This prevents inflating the list of CFI functions when
callgraph section is enabled along with CFI.
This only exists to differentiate between real and synthetic profiles.
Remove the abstraction now that we plan to fully remove synthetic
profiles.
Reviewers: mtrofin, david-xl
Reviewed By: mtrofin
Pull Request: https://github.com/llvm/llvm-project/pull/204770
Now that synthetic entry counts are being removed, stop using the
ProfileCount wrapper around entrycounts given it only exists to
distinguish between synthetic and real profile counts.
Reviewers: teresajohnson, david-xl, mtrofin
Pull Request: https://github.com/llvm/llvm-project/pull/204769
Adds two optional fields to the per-vreg YAML record so MIR tests can
express VirtRegMap state that previously had no representation:
registers:
- { id: 1, class: vgpr_32, split-from: '%0', assigned-phys: '$vgpr5' }
Testing passes that consume sibling-register information (e.g.
InlineSpiller) requires constructing a VirtRegMap with split
relationships from a MIR test, which implies triggering live-range
splitting at minimum and make reproducers unnecessarily complicated.
So this change introduces a mechanism to serialize/deserialize the state
of the VirtRegMap pass.
Mechanism:
- For serialization:
- MIRPrinter emits the new fields only when the VirtRegMap is available.
- For deserialization:
- MIRParser stashes parsed entries in the MachineRegisterInfo object
- VirtRegMap::init() drains the stash via assignVirt2Phys and
setIsSplitFromReg, then clears it.
Validation at parse time:
- 'assigned-phys' must be a physical register.
- 'split-from' must reference a different vreg than 'id'.
- 'split-from' physregs / named-vreg references already rejected
by parseStandaloneVirtualRegister.
I'm not super happy about stashing the VRM info in the MRI, but that's
a small price to pay.
This removes `WasmEHFuncInfo` class.
This class was created to maintain the information of, "If an exception
is not caught by EHPad A, what is its next unwind destination?". Turns
out this information is already in the CFG.
After #130374, we use the common `findUnwindDestination`:
https://github.com/llvm/llvm-project/blob/113479d119a997e4c4c3eae63e087588c9662121/llvm/lib/CodeGen/SelectionDAG/SelectionDAGBuilder.cpp#L2107-L2164
Note that in case of `catchswitch`, we follow its unwind destination
chain and add all of them to the invoke BB's successors until it meets a
`cleanuppad`, which always catches an exception. And the order of the
successor is the order of the unwind destination chain. So an invoke
BB's successor list would be like: [normal destination, unwind EHPad 1,
unwind EHPad 2, unwind EHPad 3, ...] where EHPad 2 is the next unwind
destination if EHPad 1 does not catch an exception and so on. So if we
want to know what the current EHPad's next unwind destination is, we can
examine the EHPad's predecessor (invoke BB)'s successor list, and find
the next successor of EHPad. If there is no further next unwind
destination, the list should end with that EHPad.
How we do this is in the added lambda functions in CFGStackify, which
replaces `WasmEHFuncInfo`'s `hasUnwindDest`/`getUnwindDest`.
This depends on the order of successors of a CFG, which someone can
argue may be not guaranteed, in case someone messes up the successor
order in later passes. But I think in practice this should be fine. Also
maintaining `WasmEHFuncInfo` has more risks, because if other passes
change EH-related CFGs, they have to make sure `WasmEHFuncInfo` stays
synced. I haven't personally found (so far) those passes and haven't had
to any adjustments to other passes, but it was always an inherent risk
when you maintain a separate data structure.
I wasn't able to completely delete `WasmEHFuncInfo.h` because it
contains the `Tag` enum. I renamed it to `WasmEHInfo.h`, because we
dont' have `WasmEHFuncInfo` class anymore.
The existing datalayout incompatible error assert does not help with
debugging, as it does not print the datalayouts in question.
This change makes this failure give more useful information.
Block numbers are updated too frequently, which makes it difficult to
keep analyses up to date. Therefore, introduce a second number per basic
block that is used for analyses and is renumbered less often. This frees
analyses from providing somewhat efficient facilities for dealing with
changed block numbers, making it simpler to implement in e.g. LoopInfo
or CycleInfo.
(Currently, "less often" means not at all, but we might want to renumber
after certain passes if the numbering gets too sparse and no analyses
are preserved anyway.)
When we introduced a more general use of block numbers some time ago,
using the existing numbers seemed to be a somewhat obvious choice, but I
now think that this was a bad decision, as it conflates a number that is
used for ordering with a number that should be more stable.
MachineBasicBlock isn't particularly size-optimized and there's a fair
amount of padding where we can add another number.
There should be no performance impact,
The "private global" terminology, likely came from
llvm/lib/IR/Mangler.cpp, is misleading: "private" is the opposite of
"global", and these prefixed symbols are not global in the object file
format sense (e.g. ELF has STB_GLOBAL while these symbols are always
STB_LOCAL). The term "internal symbol" better describes their purpose:
symbols for internal use by compilers and assemblers, not meant to be
visible externally.
This rename is a step toward adopting the "internal symbol prefix"
terminology agreed with GNU as
(https://sourceware.org/pipermail/binutils/2026-March/148448.html).
Given the test case:
struct CBase {
virtual void foo();
};
void bar(CBase *Base) {
Base->foo();
}
and using '-emit-call-site-info' with llc, the following DWARF
is produced for the indirect call 'Base->foo()':
1$: DW_TAG_structure_type "CBase"
...
2$: DW_TAG_subprogram "foo"
...
3$: DW_TAG_subprogram "bar"
...
4$: DW_TAG_call_site
...
We add DW_AT_LLVM_virtual_call_origin to existing call-site
information, linking indirect calls to the function-declaration
they correspond to.
4$: DW_TAG_call_site
...
DW_AT_LLVM_virtual_call_origin (2$ "_ZN5CBase3fooEv")
The new attribute DW_AT_LLVM_virtual_call_origin helps to
address the ambiguity to any consumer due to the usage of
DW_AT_call_origin.
The functionality is available to all supported debuggers and
it is generated only for DWARF version 5 or greater.
…n in DWARF. (#167666)"
This reverts commit 418ba6e8ae.
The commit caused an ICE due to hitting unreachable in
llvm/lib/CodeGen/AsmPrinter/DwarfCompileUnit.cpp:1307
Fixes#182337
Given the test case:
struct CBase {
virtual void foo();
};
void bar(CBase *Base) {
Base->foo();
}
and using '-emit-call-site-info' with llc, the following DWARF
is produced for the indirect call 'Base->foo()':
1$: DW_TAG_structure_type "CBase"
...
2$: DW_TAG_subprogram "foo"
...
3$: DW_TAG_subprogram "bar"
...
4$: DW_TAG_call_site
...
We add DW_AT_LLVM_virtual_call_origin to existing call-site
information, linking indirect calls to the function-declaration
they correspond to.
4$: DW_TAG_call_site
...
DW_AT_LLVM_virtual_call_origin (2$ "_ZN5CBase3fooEv")
The new attribute DW_AT_LLVM_virtual_call_origin helps to
address the ambiguity to any consumer due to the usage of
DW_AT_call_origin.
The functionality is available to all supported debuggers.
As per @arsenm 's instructions, I've separated the non-functional
changes from https://github.com/llvm/llvm-project/pull/169958.
Afterwards I'll tackle the functional ones one by one. I hope I did
everything right this time.
Full descriptions in the article:
https://pvs-studio.com/en/blog/posts/cpp/1318/
3. Array overrun is possible.
The PVS-Studio warning: V557 Array overrun is possible. The value of
'regIdx' index could reach 31. VEAsmParser.cpp 696
10. Excessive check.
The PVS-Studio warning: V547 Expression 'IsLeaf' is always false.
PPCInstrInfo.cpp 419
11. Doubling the same check.
The PVS-Studio warning: V581 The conditional expressions of the 'if'
statements situated alongside each other are identical. Check lines:
5820, 5823. PPCInstrInfo.cpp 5823
15. Excessive check.
The PVS-Studio warning: V547 Expression 'i != e' is always true.
MachineFunction.cpp 1444
17. Excessive assignment.
The PVS-Studio warning: V1048 The 'FirstOp' variable was assigned the
same value. MachineInstr.cpp 1995
18. Excessive check.
The PVS-Studio warning: V547 Expression 'AllSame' is always true.
SimplifyCFG.cpp 1914
19. Excessive check.
The PVS-Studio warning: V547 Expression 'AbbrevDecl' is always true.
LVDWARFReader.cpp 398
This reverts commit 05e08cdb3e.
Adding the missing -mtriple flags in MIR/X86 test files which caused
these tests to fail which was the reason for reverting the patch.
Introducing `EnableCallGraphSection` target option to add
CalleeTypeIds field in CallSiteInfo. Read the callee type ids
in and out by the MIR parser/printer.
Reviewers: ilovepi
Reviewed By: ilovepi
Pull Request: https://github.com/llvm/llvm-project/pull/87574
Deleting a basic block removes all references from jump tables, which
is O(n). When freeing a MachineFunction, all basic blocks are deleted
before the jump tables, causing O(n^2) runtime. Fix this by deallocating
the jump table first.
Test case generator:
import sys
n = int(sys.argv[1])
print("define void @f(i64 %c, ptr %p) {")
print(" switch i64 %c, label %d [")
for i in range(n):
print(f" i64 {i}, label %h{i}")
print(f" ]")
for i in range(n):
print(f'h{i}:')
print(f' store i64 {i*i}, ptr %p')
print(f' ret void')
print('d:')
print(' ret void')
print('}')
Improvement at 5000 entries:
Benchmark 1: ./llc.pre -filetype=obj -O0 <switch5k.bc
Time (mean ± σ): 49.7 ms ± 1.0 ms
Range (min … max): 48.0 ms … 52.1 ms 57 runs
Benchmark 2: ./llc.post -filetype=obj -O0 <switch5k.bc
Time (mean ± σ): 39.4 ms ± 0.8 ms
Range (min … max): 37.1 ms … 41.1 ms 72 runs
Summary
./llc.post -filetype=obj -O0 <switch5k.bc ran
1.26 ± 0.04 times faster than ./llc.pre -filetype=obj -O0 <switch5k.bc
Improvement at 20000 entries:
Benchmark 1: ./llc.pre -filetype=obj -O0 <switch20k.bc
Time (mean ± σ): 281.7 ms ± 1.0 ms
Range (min … max): 280.2 ms … 283.0 ms 10 runs
Benchmark 2: ./llc.post -filetype=obj -O0 <switch20k.bc
Time (mean ± σ): 123.9 ms ± 1.5 ms
Range (min … max): 121.4 ms … 129.2 ms 23 runs
Summary
./llc.post -filetype=obj -O0 <switch20k.bc ran
2.27 ± 0.03 times faster than ./llc.pre -filetype=obj -O0 <switch20k.bc
Pull Request: https://github.com/llvm/llvm-project/pull/144108
The LiveDebugValues pass and the instruction selector (which calls
salvageCopySSA) need to be consistent on what they consider a copy
instruction. With https://github.com/llvm/llvm-project/pull/75184, the
definition of what a copy instruction is was narrowed for AArch64 to
exclude a w->x ORR and treat it as a zero-extend rather than a copy
However, to make sure LiveDebugValues still treats a w->x ORR as a copy,
the new function, isCopyLikeInstr was created. We need to make sure that
salvageCopySSA also calls that function.
This patch addresses this mismatch.
As part of the "RemoveDIs" project, BasicBlock::iterator now carries a
debug-info bit that's needed when getFirstNonPHI and similar feed into
instruction insertion positions. Call-sites where that's necessary were
updated a year ago; but to ensure some type safety however, we'd like to
have all calls to getFirstNonPHI use the iterator-returning version.
This patch changes a bunch of call-sites calling getFirstNonPHI to use
getFirstNonPHIIt, which returns an iterator. All these call sites are
where it's obviously safe to fetch the iterator then dereference it. A
follow-up patch will contain less-obviously-safe changes.
We'll eventually deprecate and remove the instruction-pointer
getFirstNonPHI, but not before adding concise documentation of what
considerations are needed (very few).
---------
Co-authored-by: Stephen Tozer <Melamoto@gmail.com>
Fixes the "use after poison" issue introduced by #121516 (see
<https://github.com/llvm/llvm-project/pull/121516#issuecomment-2585912395>).
The root cause of this issue is that #121516 introduced "Called Global"
information for call instructions modeling how "Call Site" info is
stored in the machine function, HOWEVER it didn't copy the
copy/move/erase operations for call site information.
The fix is to rename and update the existing copy/move/erase functions
so they also take care of Called Global info.
Try to use DiagnosticInfo if every register in the class is reserved
by forcing assignment to a reserved register. Also reduces the number
of redundant errors emitted, particularly with fast.
This is still broken in the case of undef uses. There are additional
complications in greedy and fast, so leave it for a separate fix.
Following discussions in #110443, and the following earlier discussions
in https://lists.llvm.org/pipermail/llvm-dev/2017-October/117907.html,
https://reviews.llvm.org/D38482, https://reviews.llvm.org/D38489, this
PR attempts to overhaul the `TargetMachine` and `LLVMTargetMachine`
interface classes. More specifically:
1. Makes `TargetMachine` the only class implemented under
`TargetMachine.h` in the `Target` library.
2. `TargetMachine` contains target-specific interface functions that
relate to IR/CodeGen/MC constructs, whereas before (at least on paper)
it was supposed to have only IR/MC constructs. Any Target that doesn't
want to use the independent code generator simply does not implement
them, and returns either `false` or `nullptr`.
3. Renames `LLVMTargetMachine` to `CodeGenCommonTMImpl`. This renaming
aims to make the purpose of `LLVMTargetMachine` clearer. Its interface
was moved under the CodeGen library, to further emphasis its usage in
Targets that use CodeGen directly.
4. Makes `TargetMachine` the only interface used across LLVM and its
projects. With these changes, `CodeGenCommonTMImpl` is simply a set of
shared function implementations of `TargetMachine`, and CodeGen users
don't need to static cast to `LLVMTargetMachine` every time they need a
CodeGen-specific feature of the `TargetMachine`.
5. More importantly, does not change any requirements regarding library
linking.
cc @arsenm @aeubanks
