llvm.org GIT mirror llvm / afe3498
Remove gc.root's performCustomLowering This is a refactoring to restructure the single user of performCustomLowering as a specific lowering pass and remove the custom lowering hook entirely. Before this change, the LowerIntrinsics pass (note to self: rename!) was essentially acting as a pass manager, but without being structured in terms of passes. Instead, it proxied calls to a set of GCStrategies internally. This adds a lot of conceptual complexity (i.e. GCStrategies are stateful!) for very little benefit. Since there's been interest in keeping the ShadowStackGC working, I extracting it's custom lowering pass into a dedicated pass and just added that to the pass order. It will only run for functions which opt-in to that gc. I wasn't able to find an easy way to preserve the runtime registration of custom lowering functionality. Given that no user of this exists that I'm aware of, I made the choice to just remove that. If someone really cares, we can look at restoring it via dynamic pass registration in the future. Note that despite the large diff, none of the lowering code actual changes. I added the framing needed to make it a pass and rename the class, but that's it. Differential Revision: http://reviews.llvm.org/D7218 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@227351 91177308-0d34-0410-b5e6-96231b3b80d8 Philip Reames 5 years ago
9 changed file(s) with 490 addition(s) and 511 deletion(s). Raw diff Collapse all Expand all
720720 Custom lowering of intrinsics: ``CustomRoots``, ``CustomReadBarriers``, and ``CustomWriteBarriers``
721721 ---------------------------------------------------------------------------------------------------
722722
723 For GCs which use barriers or unusual treatment of stack roots, these flags
724 allow the collector to perform arbitrary transformations of the LLVM IR:
723 For GCs which use barriers or unusual treatment of stack roots, these
724 flags allow the collector to perform arbitrary transformations of the
725 LLVM IR:
725726
726727 .. code-block:: c++
727728
732733 CustomReadBarriers = true;
733734 CustomWriteBarriers = true;
734735 }
735
736 virtual bool initializeCustomLowering(Module &M);
737 virtual bool performCustomLowering(Function &F);
738736 };
739737
740 If any of these flags are set, then LLVM suppresses its default lowering for the
741 corresponding intrinsics and instead calls ``performCustomLowering``.
742
743 LLVM's default action for each intrinsic is as follows:
744
745 * ``llvm.gcroot``: Leave it alone. The code generator must see it or the stack
746 map will not be computed.
747
748 * ``llvm.gcread``: Substitute a ``load`` instruction.
749
750 * ``llvm.gcwrite``: Substitute a ``store`` instruction.
751
752 If ``CustomReadBarriers`` or ``CustomWriteBarriers`` are specified, then
753 ``performCustomLowering`` **must** eliminate the corresponding barriers.
754
755 ``performCustomLowering`` must comply with the same restrictions as
756 :ref:`FunctionPass::runOnFunction `
757 Likewise, ``initializeCustomLowering`` has the same semantics as
758 :ref:`Pass::doInitialization(Module&)
759 `
760
761 The following can be used as a template:
762
763 .. code-block:: c++
764
765 #include "llvm/IR/Module.h"
766 #include "llvm/IR/IntrinsicInst.h"
767
768 bool MyGC::initializeCustomLowering(Module &M) {
769 return false;
770 }
771
772 bool MyGC::performCustomLowering(Function &F) {
773 bool MadeChange = false;
774
775 for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)
776 for (BasicBlock::iterator II = BB->begin(), E = BB->end(); II != E; )
777 if (IntrinsicInst *CI = dyn_cast(II++))
778 if (Function *F = CI->getCalledFunction())
779 switch (F->getIntrinsicID()) {
780 case Intrinsic::gcwrite:
781 // Handle llvm.gcwrite.
782 CI->eraseFromParent();
783 MadeChange = true;
784 break;
785 case Intrinsic::gcread:
786 // Handle llvm.gcread.
787 CI->eraseFromParent();
788 MadeChange = true;
789 break;
790 case Intrinsic::gcroot:
791 // Handle llvm.gcroot.
792 CI->eraseFromParent();
793 MadeChange = true;
794 break;
795 }
796
797 return MadeChange;
798 }
738 If any of these flags are set, LLVM suppresses its default lowering for
739 the corresponding intrinsics. Instead, you must provide a custom Pass
740 which lowers the intrinsics as desired. If you have opted in to custom
741 lowering of a particular intrinsic your pass **must** eliminate all
742 instances of the corresponding intrinsic in functions which opt in to
743 your GC. The best example of such a pass is the ShadowStackGC and it's
744 ShadowStackGCLowering pass.
745
746 There is currently no way to register such a custom lowering pass
747 without building a custom copy of LLVM.
799748
800749 .. _safe-points:
801750
100100 const std::string &getName() const { return Name; }
101101
102102 /// By default, write barriers are replaced with simple store
103 /// instructions. If true, then performCustomLowering must instead lower
104 /// them.
103 /// instructions. If true, you must provide a custom pass to lower
104 /// calls to @llvm.gcwrite.
105105 bool customWriteBarrier() const { return CustomWriteBarriers; }
106106
107107 /// By default, read barriers are replaced with simple load
108 /// instructions. If true, then performCustomLowering must instead lower
109 /// them.
108 /// instructions. If true, you must provide a custom pass to lower
109 /// calls to @llvm.gcread.
110110 bool customReadBarrier() const { return CustomReadBarriers; }
111111
112112 /// Returns true if this strategy is expecting the use of gc.statepoints,
142142 }
143143
144144 /// By default, roots are left for the code generator so it can generate a
145 /// stack map. If true, then performCustomLowering must delete them.
145 /// stack map. If true, you must provide a custom pass to lower
146 /// calls to @llvm.gcroot.
146147 bool customRoots() const { return CustomRoots; }
147148
148149 /// If set, gcroot intrinsics should initialize their allocas to null
157158 bool usesMetadata() const { return UsesMetadata; }
158159
159160 ///@}
160
161 /// initializeCustomLowering/performCustomLowering - If any of the actions
162 /// are set to custom, performCustomLowering must be overriden to transform
163 /// the corresponding actions to LLVM IR. initializeCustomLowering is
164 /// optional to override. These are the only GCStrategy methods through
165 /// which the LLVM IR can be modified. These methods apply mostly to
166 /// gc.root based implementations, but can be overriden to provide custom
167 /// barrier lowerings with gc.statepoint as well.
168 ///@{
169 virtual bool initializeCustomLowering(Module &F) {
170 // No changes made
171 return false;
172 }
173 virtual bool performCustomLowering(Function &F) {
174 llvm_unreachable("GCStrategy subclass specified a configuration which"
175 "requires a custom lowering without providing one");
176 }
177161 };
178162
179163 /// Subclasses of GCStrategy are made available for use during compilation by
516516 /// information.
517517 extern char &MachineBlockPlacementStatsID;
518518
519 /// GCLowering Pass - Performs target-independent LLVM IR transformations for
520 /// highly portable strategies.
521 ///
519 /// GCLowering Pass - Used by gc.root to perform its default lowering
520 /// operations.
522521 FunctionPass *createGCLoweringPass();
522
523 /// ShadowStackGCLowering - Implements the custom lowering mechanism
524 /// used by the shadow stack GC. Only runs on functions which opt in to
525 /// the shadow stack collector.
526 FunctionPass *createShadowStackGCLoweringPass();
523527
524528 /// GCMachineCodeAnalysis - Target-independent pass to mark safe points
525529 /// in machine code. Must be added very late during code generation, just
244244 void initializeScalarEvolutionAliasAnalysisPass(PassRegistry&);
245245 void initializeScalarEvolutionPass(PassRegistry&);
246246 void initializeSimpleInlinerPass(PassRegistry&);
247 void initializeShadowStackGCLoweringPass(PassRegistry&);
247248 void initializeRegisterCoalescerPass(PassRegistry&);
248249 void initializeSingleLoopExtractorPass(PassRegistry&);
249250 void initializeSinkingPass(PassRegistry&);
9595 ScheduleDAGPrinter.cpp
9696 ScoreboardHazardRecognizer.cpp
9797 ShadowStackGC.cpp
98 ShadowStackGCLowering.cpp
9899 SjLjEHPrepare.cpp
99100 SlotIndexes.cpp
100101 SpillPlacement.cpp
110110 C.initializeRoots();
111111 }
112112
113 static bool NeedsCustomLoweringPass(const GCStrategy &C) {
114 // Custom lowering is only necessary if enabled for some action.
115 return C.customWriteBarrier() || C.customReadBarrier() || C.customRoots();
116 }
117
118113 /// doInitialization - If this module uses the GC intrinsics, find them now.
119114 bool LowerIntrinsics::doInitialization(Module &M) {
120 // FIXME: This is rather antisocial in the context of a JIT since it performs
121 // work against the entire module. But this cannot be done at
122 // runFunction time (initializeCustomLowering likely needs to change
123 // the module).
124115 GCModuleInfo *MI = getAnalysisIfAvailable();
125116 assert(MI && "LowerIntrinsics didn't require GCModuleInfo!?");
126117 for (Module::iterator I = M.begin(), E = M.end(); I != E; ++I)
127118 if (!I->isDeclaration() && I->hasGC())
128119 MI->getFunctionInfo(*I); // Instantiate the GC strategy.
129120
130 bool MadeChange = false;
131 for (GCModuleInfo::iterator I = MI->begin(), E = MI->end(); I != E; ++I)
132 if (NeedsCustomLoweringPass(**I))
133 if ((*I)->initializeCustomLowering(M))
134 MadeChange = true;
135
136 return MadeChange;
121 return false;
137122 }
138123
139124 /// CouldBecomeSafePoint - Predicate to conservatively determine whether the
209194
210195 if (NeedsDefaultLoweringPass(S))
211196 MadeChange |= PerformDefaultLowering(F, S);
212
213 bool UseCustomLoweringPass = NeedsCustomLoweringPass(S);
214 if (UseCustomLoweringPass)
215 MadeChange |= S.performCustomLowering(F);
216
217 // Custom lowering may modify the CFG, so dominators must be recomputed.
218 if (UseCustomLoweringPass) {
219 if (DominatorTreeWrapperPass *DTWP =
220 getAnalysisIfAvailable())
221 DTWP->getDomTree().recalculate(F);
222 }
223197
224198 return MadeChange;
225199 }
418418 addPass(createPrintFunctionPass(dbgs(), "\n\n*** Code after LSR ***\n"));
419419 }
420420
421 // Run GC lowering passes for builtin collectors
422 // TODO: add a pass insertion point here
421423 addPass(createGCLoweringPass());
424 addPass(createShadowStackGCLoweringPass());
422425
423426 // Make sure that no unreachable blocks are instruction selected.
424427 addPass(createUnreachableBlockEliminationPass());
3737 #define DEBUG_TYPE "shadowstackgc"
3838
3939 namespace {
40
4140 class ShadowStackGC : public GCStrategy {
42 /// RootChain - This is the global linked-list that contains the chain of GC
43 /// roots.
44 GlobalVariable *Head;
45
46 /// StackEntryTy - Abstract type of a link in the shadow stack.
47 ///
48 StructType *StackEntryTy;
49 StructType *FrameMapTy;
50
51 /// Roots - GC roots in the current function. Each is a pair of the
52 /// intrinsic call and its corresponding alloca.
53 std::vector> Roots;
54
5541 public:
5642 ShadowStackGC();
57
58 bool initializeCustomLowering(Module &M) override;
59 bool performCustomLowering(Function &F) override;
60
61 private:
62 bool IsNullValue(Value *V);
63 Constant *GetFrameMap(Function &F);
64 Type *GetConcreteStackEntryType(Function &F);
65 void CollectRoots(Function &F);
66 static GetElementPtrInst *CreateGEP(LLVMContext &Context, IRBuilder<> &B,
67 Value *BasePtr, int Idx1,
68 const char *Name);
69 static GetElementPtrInst *CreateGEP(LLVMContext &Context, IRBuilder<> &B,
70 Value *BasePtr, int Idx1, int Idx2,
71 const char *Name);
7243 };
7344 }
7445
7546 static GCRegistry::Add
7647 X("shadow-stack", "Very portable GC for uncooperative code generators");
7748
78 namespace {
79 /// EscapeEnumerator - This is a little algorithm to find all escape points
80 /// from a function so that "finally"-style code can be inserted. In addition
81 /// to finding the existing return and unwind instructions, it also (if
82 /// necessary) transforms any call instructions into invokes and sends them to
83 /// a landing pad.
84 ///
85 /// It's wrapped up in a state machine using the same transform C# uses for
86 /// 'yield return' enumerators, This transform allows it to be non-allocating.
87 class EscapeEnumerator {
88 Function &F;
89 const char *CleanupBBName;
90
91 // State.
92 int State;
93 Function::iterator StateBB, StateE;
94 IRBuilder<> Builder;
95
96 public:
97 EscapeEnumerator(Function &F, const char *N = "cleanup")
98 : F(F), CleanupBBName(N), State(0), Builder(F.getContext()) {}
99
100 IRBuilder<> *Next() {
101 switch (State) {
102 default:
103 return nullptr;
104
105 case 0:
106 StateBB = F.begin();
107 StateE = F.end();
108 State = 1;
109
110 case 1:
111 // Find all 'return', 'resume', and 'unwind' instructions.
112 while (StateBB != StateE) {
113 BasicBlock *CurBB = StateBB++;
114
115 // Branches and invokes do not escape, only unwind, resume, and return
116 // do.
117 TerminatorInst *TI = CurBB->getTerminator();
118 if (!isa(TI) && !isa(TI))
119 continue;
120
121 Builder.SetInsertPoint(TI->getParent(), TI);
122 return &Builder;
123 }
124
125 State = 2;
126
127 // Find all 'call' instructions.
128 SmallVector Calls;
129 for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)
130 for (BasicBlock::iterator II = BB->begin(), EE = BB->end(); II != EE;
131 ++II)
132 if (CallInst *CI = dyn_cast(II))
133 if (!CI->getCalledFunction() ||
134 !CI->getCalledFunction()->getIntrinsicID())
135 Calls.push_back(CI);
136
137 if (Calls.empty())
138 return nullptr;
139
140 // Create a cleanup block.
141 LLVMContext &C = F.getContext();
142 BasicBlock *CleanupBB = BasicBlock::Create(C, CleanupBBName, &F);
143 Type *ExnTy =
144 StructType::get(Type::getInt8PtrTy(C), Type::getInt32Ty(C), nullptr);
145 Constant *PersFn = F.getParent()->getOrInsertFunction(
146 "__gcc_personality_v0", FunctionType::get(Type::getInt32Ty(C), true));
147 LandingPadInst *LPad =
148 LandingPadInst::Create(ExnTy, PersFn, 1, "cleanup.lpad", CleanupBB);
149 LPad->setCleanup(true);
150 ResumeInst *RI = ResumeInst::Create(LPad, CleanupBB);
151
152 // Transform the 'call' instructions into 'invoke's branching to the
153 // cleanup block. Go in reverse order to make prettier BB names.
154 SmallVector Args;
155 for (unsigned I = Calls.size(); I != 0;) {
156 CallInst *CI = cast(Calls[--I]);
157
158 // Split the basic block containing the function call.
159 BasicBlock *CallBB = CI->getParent();
160 BasicBlock *NewBB =
161 CallBB->splitBasicBlock(CI, CallBB->getName() + ".cont");
162
163 // Remove the unconditional branch inserted at the end of CallBB.
164 CallBB->getInstList().pop_back();
165 NewBB->getInstList().remove(CI);
166
167 // Create a new invoke instruction.
168 Args.clear();
169 CallSite CS(CI);
170 Args.append(CS.arg_begin(), CS.arg_end());
171
172 InvokeInst *II =
173 InvokeInst::Create(CI->getCalledValue(), NewBB, CleanupBB, Args,
174 CI->getName(), CallBB);
175 II->setCallingConv(CI->getCallingConv());
176 II->setAttributes(CI->getAttributes());
177 CI->replaceAllUsesWith(II);
178 delete CI;
179 }
180
181 Builder.SetInsertPoint(RI->getParent(), RI);
182 return &Builder;
183 }
184 }
185 };
186 }
187
188 // -----------------------------------------------------------------------------
189
19049 void llvm::linkShadowStackGC() {}
19150
192 ShadowStackGC::ShadowStackGC() : Head(nullptr), StackEntryTy(nullptr) {
51 ShadowStackGC::ShadowStackGC() {
19352 InitRoots = true;
19453 CustomRoots = true;
19554 }
196
197 Constant *ShadowStackGC::GetFrameMap(Function &F) {
198 // doInitialization creates the abstract type of this value.
199 Type *VoidPtr = Type::getInt8PtrTy(F.getContext());
200
201 // Truncate the ShadowStackDescriptor if some metadata is null.
202 unsigned NumMeta = 0;
203 SmallVector Metadata;
204 for (unsigned I = 0; I != Roots.size(); ++I) {
205 Constant *C = cast(Roots[I].first->getArgOperand(1));
206 if (!C->isNullValue())
207 NumMeta = I + 1;
208 Metadata.push_back(ConstantExpr::getBitCast(C, VoidPtr));
209 }
210 Metadata.resize(NumMeta);
211
212 Type *Int32Ty = Type::getInt32Ty(F.getContext());
213
214 Constant *BaseElts[] = {
215 ConstantInt::get(Int32Ty, Roots.size(), false),
216 ConstantInt::get(Int32Ty, NumMeta, false),
217 };
218
219 Constant *DescriptorElts[] = {
220 ConstantStruct::get(FrameMapTy, BaseElts),
221 ConstantArray::get(ArrayType::get(VoidPtr, NumMeta), Metadata)};
222
223 Type *EltTys[] = {DescriptorElts[0]->getType(), DescriptorElts[1]->getType()};
224 StructType *STy = StructType::create(EltTys, "gc_map." + utostr(NumMeta));
225
226 Constant *FrameMap = ConstantStruct::get(STy, DescriptorElts);
227
228 // FIXME: Is this actually dangerous as WritingAnLLVMPass.html claims? Seems
229 // that, short of multithreaded LLVM, it should be safe; all that is
230 // necessary is that a simple Module::iterator loop not be invalidated.
231 // Appending to the GlobalVariable list is safe in that sense.
232 //
233 // All of the output passes emit globals last. The ExecutionEngine
234 // explicitly supports adding globals to the module after
235 // initialization.
236 //
237 // Still, if it isn't deemed acceptable, then this transformation needs
238 // to be a ModulePass (which means it cannot be in the 'llc' pipeline
239 // (which uses a FunctionPassManager (which segfaults (not asserts) if
240 // provided a ModulePass))).
241 Constant *GV = new GlobalVariable(*F.getParent(), FrameMap->getType(), true,
242 GlobalVariable::InternalLinkage, FrameMap,
243 "__gc_" + F.getName());
244
245 Constant *GEPIndices[2] = {
246 ConstantInt::get(Type::getInt32Ty(F.getContext()), 0),
247 ConstantInt::get(Type::getInt32Ty(F.getContext()), 0)};
248 return ConstantExpr::getGetElementPtr(GV, GEPIndices);
249 }
250
251 Type *ShadowStackGC::GetConcreteStackEntryType(Function &F) {
252 // doInitialization creates the generic version of this type.
253 std::vector EltTys;
254 EltTys.push_back(StackEntryTy);
255 for (size_t I = 0; I != Roots.size(); I++)
256 EltTys.push_back(Roots[I].second->getAllocatedType());
257
258 return StructType::create(EltTys, "gc_stackentry." + F.getName().str());
259 }
260
261 /// doInitialization - If this module uses the GC intrinsics, find them now. If
262 /// not, exit fast.
263 bool ShadowStackGC::initializeCustomLowering(Module &M) {
264 // struct FrameMap {
265 // int32_t NumRoots; // Number of roots in stack frame.
266 // int32_t NumMeta; // Number of metadata descriptors. May be < NumRoots.
267 // void *Meta[]; // May be absent for roots without metadata.
268 // };
269 std::vector EltTys;
270 // 32 bits is ok up to a 32GB stack frame. :)
271 EltTys.push_back(Type::getInt32Ty(M.getContext()));
272 // Specifies length of variable length array.
273 EltTys.push_back(Type::getInt32Ty(M.getContext()));
274 FrameMapTy = StructType::create(EltTys, "gc_map");
275 PointerType *FrameMapPtrTy = PointerType::getUnqual(FrameMapTy);
276
277 // struct StackEntry {
278 // ShadowStackEntry *Next; // Caller's stack entry.
279 // FrameMap *Map; // Pointer to constant FrameMap.
280 // void *Roots[]; // Stack roots (in-place array, so we pretend).
281 // };
282
283 StackEntryTy = StructType::create(M.getContext(), "gc_stackentry");
284
285 EltTys.clear();
286 EltTys.push_back(PointerType::getUnqual(StackEntryTy));
287 EltTys.push_back(FrameMapPtrTy);
288 StackEntryTy->setBody(EltTys);
289 PointerType *StackEntryPtrTy = PointerType::getUnqual(StackEntryTy);
290
291 // Get the root chain if it already exists.
292 Head = M.getGlobalVariable("llvm_gc_root_chain");
293 if (!Head) {
294 // If the root chain does not exist, insert a new one with linkonce
295 // linkage!
296 Head = new GlobalVariable(
297 M, StackEntryPtrTy, false, GlobalValue::LinkOnceAnyLinkage,
298 Constant::getNullValue(StackEntryPtrTy), "llvm_gc_root_chain");
299 } else if (Head->hasExternalLinkage() && Head->isDeclaration()) {
300 Head->setInitializer(Constant::getNullValue(StackEntryPtrTy));
301 Head->setLinkage(GlobalValue::LinkOnceAnyLinkage);
302 }
303
304 return true;
305 }
306
307 bool ShadowStackGC::IsNullValue(Value *V) {
308 if (Constant *C = dyn_cast(V))
309 return C->isNullValue();
310 return false;
311 }
312
313 void ShadowStackGC::CollectRoots(Function &F) {
314 // FIXME: Account for original alignment. Could fragment the root array.
315 // Approach 1: Null initialize empty slots at runtime. Yuck.
316 // Approach 2: Emit a map of the array instead of just a count.
317
318 assert(Roots.empty() && "Not cleaned up?");
319
320 SmallVector, 16> MetaRoots;
321
322 for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)
323 for (BasicBlock::iterator II = BB->begin(), E = BB->end(); II != E;)
324 if (IntrinsicInst *CI = dyn_cast(II++))
325 if (Function *F = CI->getCalledFunction())
326 if (F->getIntrinsicID() == Intrinsic::gcroot) {
327 std::pair Pair = std::make_pair(
328 CI,
329 cast(CI->getArgOperand(0)->stripPointerCasts()));
330 if (IsNullValue(CI->getArgOperand(1)))
331 Roots.push_back(Pair);
332 else
333 MetaRoots.push_back(Pair);
334 }
335
336 // Number roots with metadata (usually empty) at the beginning, so that the
337 // FrameMap::Meta array can be elided.
338 Roots.insert(Roots.begin(), MetaRoots.begin(), MetaRoots.end());
339 }
340
341 GetElementPtrInst *ShadowStackGC::CreateGEP(LLVMContext &Context,
342 IRBuilder<> &B, Value *BasePtr,
343 int Idx, int Idx2,
344 const char *Name) {
345 Value *Indices[] = {ConstantInt::get(Type::getInt32Ty(Context), 0),
346 ConstantInt::get(Type::getInt32Ty(Context), Idx),
347 ConstantInt::get(Type::getInt32Ty(Context), Idx2)};
348 Value *Val = B.CreateGEP(BasePtr, Indices, Name);
349
350 assert(isa(Val) && "Unexpected folded constant");
351
352 return dyn_cast(Val);
353 }
354
355 GetElementPtrInst *ShadowStackGC::CreateGEP(LLVMContext &Context,
356 IRBuilder<> &B, Value *BasePtr,
357 int Idx, const char *Name) {
358 Value *Indices[] = {ConstantInt::get(Type::getInt32Ty(Context), 0),
359 ConstantInt::get(Type::getInt32Ty(Context), Idx)};
360 Value *Val = B.CreateGEP(BasePtr, Indices, Name);
361
362 assert(isa(Val) && "Unexpected folded constant");
363
364 return dyn_cast(Val);
365 }
366
367 /// runOnFunction - Insert code to maintain the shadow stack.
368 bool ShadowStackGC::performCustomLowering(Function &F) {
369 LLVMContext &Context = F.getContext();
370
371 // Find calls to llvm.gcroot.
372 CollectRoots(F);
373
374 // If there are no roots in this function, then there is no need to add a
375 // stack map entry for it.
376 if (Roots.empty())
377 return false;
378
379 // Build the constant map and figure the type of the shadow stack entry.
380 Value *FrameMap = GetFrameMap(F);
381 Type *ConcreteStackEntryTy = GetConcreteStackEntryType(F);
382
383 // Build the shadow stack entry at the very start of the function.
384 BasicBlock::iterator IP = F.getEntryBlock().begin();
385 IRBuilder<> AtEntry(IP->getParent(), IP);
386
387 Instruction *StackEntry =
388 AtEntry.CreateAlloca(ConcreteStackEntryTy, nullptr, "gc_frame");
389
390 while (isa(IP))
391 ++IP;
392 AtEntry.SetInsertPoint(IP->getParent(), IP);
393
394 // Initialize the map pointer and load the current head of the shadow stack.
395 Instruction *CurrentHead = AtEntry.CreateLoad(Head, "gc_currhead");
396 Instruction *EntryMapPtr =
397 CreateGEP(Context, AtEntry, StackEntry, 0, 1, "gc_frame.map");
398 AtEntry.CreateStore(FrameMap, EntryMapPtr);
399
400 // After all the allocas...
401 for (unsigned I = 0, E = Roots.size(); I != E; ++I) {
402 // For each root, find the corresponding slot in the aggregate...
403 Value *SlotPtr = CreateGEP(Context, AtEntry, StackEntry, 1 + I, "gc_root");
404
405 // And use it in lieu of the alloca.
406 AllocaInst *OriginalAlloca = Roots[I].second;
407 SlotPtr->takeName(OriginalAlloca);
408 OriginalAlloca->replaceAllUsesWith(SlotPtr);
409 }
410
411 // Move past the original stores inserted by GCStrategy::InitRoots. This isn't
412 // really necessary (the collector would never see the intermediate state at
413 // runtime), but it's nicer not to push the half-initialized entry onto the
414 // shadow stack.
415 while (isa(IP))
416 ++IP;
417 AtEntry.SetInsertPoint(IP->getParent(), IP);
418
419 // Push the entry onto the shadow stack.
420 Instruction *EntryNextPtr =
421 CreateGEP(Context, AtEntry, StackEntry, 0, 0, "gc_frame.next");
422 Instruction *NewHeadVal =
423 CreateGEP(Context, AtEntry, StackEntry, 0, "gc_newhead");
424 AtEntry.CreateStore(CurrentHead, EntryNextPtr);
425 AtEntry.CreateStore(NewHeadVal, Head);
426
427 // For each instruction that escapes...
428 EscapeEnumerator EE(F, "gc_cleanup");
429 while (IRBuilder<> *AtExit = EE.Next()) {
430 // Pop the entry from the shadow stack. Don't reuse CurrentHead from
431 // AtEntry, since that would make the value live for the entire function.
432 Instruction *EntryNextPtr2 =
433 CreateGEP(Context, *AtExit, StackEntry, 0, 0, "gc_frame.next");
434 Value *SavedHead = AtExit->CreateLoad(EntryNextPtr2, "gc_savedhead");
435 AtExit->CreateStore(SavedHead, Head);
436 }
437
438 // Delete the original allocas (which are no longer used) and the intrinsic
439 // calls (which are no longer valid). Doing this last avoids invalidating
440 // iterators.
441 for (unsigned I = 0, E = Roots.size(); I != E; ++I) {
442 Roots[I].first->eraseFromParent();
443 Roots[I].second->eraseFromParent();
444 }
445
446 Roots.clear();
447 return true;
448 }
0 //===-- ShadowStackGCLowering.cpp - Custom lowering for shadow-stack gc ---===//
1 //
2 // The LLVM Compiler Infrastructure
3 //
4 // This file is distributed under the University of Illinois Open Source
5 // License. See LICENSE.TXT for details.
6 //
7 //===----------------------------------------------------------------------===//
8 //
9 // This file contains the custom lowering code required by the shadow-stack GC
10 // strategy.
11 //
12 //===----------------------------------------------------------------------===//
13
14 #include "llvm/CodeGen/Passes.h"
15 #include "llvm/CodeGen/GCStrategy.h"
16 #include "llvm/ADT/StringExtras.h"
17 #include "llvm/IR/CallSite.h"
18 #include "llvm/IR/IRBuilder.h"
19 #include "llvm/IR/IntrinsicInst.h"
20 #include "llvm/IR/Module.h"
21
22 using namespace llvm;
23
24 #define DEBUG_TYPE "shadowstackgclowering"
25
26 namespace {
27
28 class ShadowStackGCLowering : public FunctionPass {
29 /// RootChain - This is the global linked-list that contains the chain of GC
30 /// roots.
31 GlobalVariable *Head;
32
33 /// StackEntryTy - Abstract type of a link in the shadow stack.
34 ///
35 StructType *StackEntryTy;
36 StructType *FrameMapTy;
37
38 /// Roots - GC roots in the current function. Each is a pair of the
39 /// intrinsic call and its corresponding alloca.
40 std::vector> Roots;
41
42 public:
43 static char ID;
44 ShadowStackGCLowering();
45
46 bool doInitialization(Module &M) override;
47 bool runOnFunction(Function &F) override;
48
49 private:
50 bool IsNullValue(Value *V);
51 Constant *GetFrameMap(Function &F);
52 Type *GetConcreteStackEntryType(Function &F);
53 void CollectRoots(Function &F);
54 static GetElementPtrInst *CreateGEP(LLVMContext &Context, IRBuilder<> &B,
55 Value *BasePtr, int Idx1,
56 const char *Name);
57 static GetElementPtrInst *CreateGEP(LLVMContext &Context, IRBuilder<> &B,
58 Value *BasePtr, int Idx1, int Idx2,
59 const char *Name);
60 };
61 }
62
63 INITIALIZE_PASS_BEGIN(ShadowStackGCLowering, "shadow-stack-gc-lowering",
64 "Shadow Stack GC Lowering", false, false)
65 INITIALIZE_PASS_DEPENDENCY(GCModuleInfo)
66 INITIALIZE_PASS_END(ShadowStackGCLowering, "shadow-stack-gc-lowering",
67 "Shadow Stack GC Lowering", false, false)
68
69 FunctionPass *llvm::createShadowStackGCLoweringPass() { return new ShadowStackGCLowering(); }
70
71 char ShadowStackGCLowering::ID = 0;
72
73 ShadowStackGCLowering::ShadowStackGCLowering()
74 : FunctionPass(ID), Head(nullptr), StackEntryTy(nullptr),
75 FrameMapTy(nullptr) {
76 initializeShadowStackGCLoweringPass(*PassRegistry::getPassRegistry());
77 }
78
79 namespace {
80 /// EscapeEnumerator - This is a little algorithm to find all escape points
81 /// from a function so that "finally"-style code can be inserted. In addition
82 /// to finding the existing return and unwind instructions, it also (if
83 /// necessary) transforms any call instructions into invokes and sends them to
84 /// a landing pad.
85 ///
86 /// It's wrapped up in a state machine using the same transform C# uses for
87 /// 'yield return' enumerators, This transform allows it to be non-allocating.
88 class EscapeEnumerator {
89 Function &F;
90 const char *CleanupBBName;
91
92 // State.
93 int State;
94 Function::iterator StateBB, StateE;
95 IRBuilder<> Builder;
96
97 public:
98 EscapeEnumerator(Function &F, const char *N = "cleanup")
99 : F(F), CleanupBBName(N), State(0), Builder(F.getContext()) {}
100
101 IRBuilder<> *Next() {
102 switch (State) {
103 default:
104 return nullptr;
105
106 case 0:
107 StateBB = F.begin();
108 StateE = F.end();
109 State = 1;
110
111 case 1:
112 // Find all 'return', 'resume', and 'unwind' instructions.
113 while (StateBB != StateE) {
114 BasicBlock *CurBB = StateBB++;
115
116 // Branches and invokes do not escape, only unwind, resume, and return
117 // do.
118 TerminatorInst *TI = CurBB->getTerminator();
119 if (!isa(TI) && !isa(TI))
120 continue;
121
122 Builder.SetInsertPoint(TI->getParent(), TI);
123 return &Builder;
124 }
125
126 State = 2;
127
128 // Find all 'call' instructions.
129 SmallVector Calls;
130 for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)
131 for (BasicBlock::iterator II = BB->begin(), EE = BB->end(); II != EE;
132 ++II)
133 if (CallInst *CI = dyn_cast(II))
134 if (!CI->getCalledFunction() ||
135 !CI->getCalledFunction()->getIntrinsicID())
136 Calls.push_back(CI);
137
138 if (Calls.empty())
139 return nullptr;
140
141 // Create a cleanup block.
142 LLVMContext &C = F.getContext();
143 BasicBlock *CleanupBB = BasicBlock::Create(C, CleanupBBName, &F);
144 Type *ExnTy =
145 StructType::get(Type::getInt8PtrTy(C), Type::getInt32Ty(C), nullptr);
146 Constant *PersFn = F.getParent()->getOrInsertFunction(
147 "__gcc_personality_v0", FunctionType::get(Type::getInt32Ty(C), true));
148 LandingPadInst *LPad =
149 LandingPadInst::Create(ExnTy, PersFn, 1, "cleanup.lpad", CleanupBB);
150 LPad->setCleanup(true);
151 ResumeInst *RI = ResumeInst::Create(LPad, CleanupBB);
152
153 // Transform the 'call' instructions into 'invoke's branching to the
154 // cleanup block. Go in reverse order to make prettier BB names.
155 SmallVector Args;
156 for (unsigned I = Calls.size(); I != 0;) {
157 CallInst *CI = cast(Calls[--I]);
158
159 // Split the basic block containing the function call.
160 BasicBlock *CallBB = CI->getParent();
161 BasicBlock *NewBB =
162 CallBB->splitBasicBlock(CI, CallBB->getName() + ".cont");
163
164 // Remove the unconditional branch inserted at the end of CallBB.
165 CallBB->getInstList().pop_back();
166 NewBB->getInstList().remove(CI);
167
168 // Create a new invoke instruction.
169 Args.clear();
170 CallSite CS(CI);
171 Args.append(CS.arg_begin(), CS.arg_end());
172
173 InvokeInst *II =
174 InvokeInst::Create(CI->getCalledValue(), NewBB, CleanupBB, Args,
175 CI->getName(), CallBB);
176 II->setCallingConv(CI->getCallingConv());
177 II->setAttributes(CI->getAttributes());
178 CI->replaceAllUsesWith(II);
179 delete CI;
180 }
181
182 Builder.SetInsertPoint(RI->getParent(), RI);
183 return &Builder;
184 }
185 }
186 };
187 }
188
189
190 Constant *ShadowStackGCLowering::GetFrameMap(Function &F) {
191 // doInitialization creates the abstract type of this value.
192 Type *VoidPtr = Type::getInt8PtrTy(F.getContext());
193
194 // Truncate the ShadowStackDescriptor if some metadata is null.
195 unsigned NumMeta = 0;
196 SmallVector Metadata;
197 for (unsigned I = 0; I != Roots.size(); ++I) {
198 Constant *C = cast(Roots[I].first->getArgOperand(1));
199 if (!C->isNullValue())
200 NumMeta = I + 1;
201 Metadata.push_back(ConstantExpr::getBitCast(C, VoidPtr));
202 }
203 Metadata.resize(NumMeta);
204
205 Type *Int32Ty = Type::getInt32Ty(F.getContext());
206
207 Constant *BaseElts[] = {
208 ConstantInt::get(Int32Ty, Roots.size(), false),
209 ConstantInt::get(Int32Ty, NumMeta, false),
210 };
211
212 Constant *DescriptorElts[] = {
213 ConstantStruct::get(FrameMapTy, BaseElts),
214 ConstantArray::get(ArrayType::get(VoidPtr, NumMeta), Metadata)};
215
216 Type *EltTys[] = {DescriptorElts[0]->getType(), DescriptorElts[1]->getType()};
217 StructType *STy = StructType::create(EltTys, "gc_map." + utostr(NumMeta));
218
219 Constant *FrameMap = ConstantStruct::get(STy, DescriptorElts);
220
221 // FIXME: Is this actually dangerous as WritingAnLLVMPass.html claims? Seems
222 // that, short of multithreaded LLVM, it should be safe; all that is
223 // necessary is that a simple Module::iterator loop not be invalidated.
224 // Appending to the GlobalVariable list is safe in that sense.
225 //
226 // All of the output passes emit globals last. The ExecutionEngine
227 // explicitly supports adding globals to the module after
228 // initialization.
229 //
230 // Still, if it isn't deemed acceptable, then this transformation needs
231 // to be a ModulePass (which means it cannot be in the 'llc' pipeline
232 // (which uses a FunctionPassManager (which segfaults (not asserts) if
233 // provided a ModulePass))).
234 Constant *GV = new GlobalVariable(*F.getParent(), FrameMap->getType(), true,
235 GlobalVariable::InternalLinkage, FrameMap,
236 "__gc_" + F.getName());
237
238 Constant *GEPIndices[2] = {
239 ConstantInt::get(Type::getInt32Ty(F.getContext()), 0),
240 ConstantInt::get(Type::getInt32Ty(F.getContext()), 0)};
241 return ConstantExpr::getGetElementPtr(GV, GEPIndices);
242 }
243
244 Type *ShadowStackGCLowering::GetConcreteStackEntryType(Function &F) {
245 // doInitialization creates the generic version of this type.
246 std::vector EltTys;
247 EltTys.push_back(StackEntryTy);
248 for (size_t I = 0; I != Roots.size(); I++)
249 EltTys.push_back(Roots[I].second->getAllocatedType());
250
251 return StructType::create(EltTys, "gc_stackentry." + F.getName().str());
252 }
253
254 /// doInitialization - If this module uses the GC intrinsics, find them now. If
255 /// not, exit fast.
256 bool ShadowStackGCLowering::doInitialization(Module &M) {
257 bool Active = false;
258 for (Function &F : M) {
259 if (F.hasGC() && F.getGC() == std::string("shadow-stack")) {
260 Active = true;
261 break;
262 }
263 }
264 if (!Active)
265 return false;
266
267 // struct FrameMap {
268 // int32_t NumRoots; // Number of roots in stack frame.
269 // int32_t NumMeta; // Number of metadata descriptors. May be < NumRoots.
270 // void *Meta[]; // May be absent for roots without metadata.
271 // };
272 std::vector EltTys;
273 // 32 bits is ok up to a 32GB stack frame. :)
274 EltTys.push_back(Type::getInt32Ty(M.getContext()));
275 // Specifies length of variable length array.
276 EltTys.push_back(Type::getInt32Ty(M.getContext()));
277 FrameMapTy = StructType::create(EltTys, "gc_map");
278 PointerType *FrameMapPtrTy = PointerType::getUnqual(FrameMapTy);
279
280 // struct StackEntry {
281 // ShadowStackEntry *Next; // Caller's stack entry.
282 // FrameMap *Map; // Pointer to constant FrameMap.
283 // void *Roots[]; // Stack roots (in-place array, so we pretend).
284 // };
285
286 StackEntryTy = StructType::create(M.getContext(), "gc_stackentry");
287
288 EltTys.clear();
289 EltTys.push_back(PointerType::getUnqual(StackEntryTy));
290 EltTys.push_back(FrameMapPtrTy);
291 StackEntryTy->setBody(EltTys);
292 PointerType *StackEntryPtrTy = PointerType::getUnqual(StackEntryTy);
293
294 // Get the root chain if it already exists.
295 Head = M.getGlobalVariable("llvm_gc_root_chain");
296 if (!Head) {
297 // If the root chain does not exist, insert a new one with linkonce
298 // linkage!
299 Head = new GlobalVariable(
300 M, StackEntryPtrTy, false, GlobalValue::LinkOnceAnyLinkage,
301 Constant::getNullValue(StackEntryPtrTy), "llvm_gc_root_chain");
302 } else if (Head->hasExternalLinkage() && Head->isDeclaration()) {
303 Head->setInitializer(Constant::getNullValue(StackEntryPtrTy));
304 Head->setLinkage(GlobalValue::LinkOnceAnyLinkage);
305 }
306
307 return true;
308 }
309
310 bool ShadowStackGCLowering::IsNullValue(Value *V) {
311 if (Constant *C = dyn_cast(V))
312 return C->isNullValue();
313 return false;
314 }
315
316 void ShadowStackGCLowering::CollectRoots(Function &F) {
317 // FIXME: Account for original alignment. Could fragment the root array.
318 // Approach 1: Null initialize empty slots at runtime. Yuck.
319 // Approach 2: Emit a map of the array instead of just a count.
320
321 assert(Roots.empty() && "Not cleaned up?");
322
323 SmallVector, 16> MetaRoots;
324
325 for (Function::iterator BB = F.begin(), E = F.end(); BB != E; ++BB)
326 for (BasicBlock::iterator II = BB->begin(), E = BB->end(); II != E;)
327 if (IntrinsicInst *CI = dyn_cast(II++))
328 if (Function *F = CI->getCalledFunction())
329 if (F->getIntrinsicID() == Intrinsic::gcroot) {
330 std::pair Pair = std::make_pair(
331 CI,
332 cast(CI->getArgOperand(0)->stripPointerCasts()));
333 if (IsNullValue(CI->getArgOperand(1)))
334 Roots.push_back(Pair);
335 else
336 MetaRoots.push_back(Pair);
337 }
338
339 // Number roots with metadata (usually empty) at the beginning, so that the
340 // FrameMap::Meta array can be elided.
341 Roots.insert(Roots.begin(), MetaRoots.begin(), MetaRoots.end());
342 }
343
344 GetElementPtrInst *ShadowStackGCLowering::CreateGEP(LLVMContext &Context,
345 IRBuilder<> &B, Value *BasePtr,
346 int Idx, int Idx2,
347 const char *Name) {
348 Value *Indices[] = {ConstantInt::get(Type::getInt32Ty(Context), 0),
349 ConstantInt::get(Type::getInt32Ty(Context), Idx),
350 ConstantInt::get(Type::getInt32Ty(Context), Idx2)};
351 Value *Val = B.CreateGEP(BasePtr, Indices, Name);
352
353 assert(isa(Val) && "Unexpected folded constant");
354
355 return dyn_cast(Val);
356 }
357
358 GetElementPtrInst *ShadowStackGCLowering::CreateGEP(LLVMContext &Context,
359 IRBuilder<> &B, Value *BasePtr,
360 int Idx, const char *Name) {
361 Value *Indices[] = {ConstantInt::get(Type::getInt32Ty(Context), 0),
362 ConstantInt::get(Type::getInt32Ty(Context), Idx)};
363 Value *Val = B.CreateGEP(BasePtr, Indices, Name);
364
365 assert(isa(Val) && "Unexpected folded constant");
366
367 return dyn_cast(Val);
368 }
369
370 /// runOnFunction - Insert code to maintain the shadow stack.
371 bool ShadowStackGCLowering::runOnFunction(Function &F) {
372 // Quick exit for functions that do not use the shadow stack GC.
373 if (!F.hasGC() ||
374 F.getGC() != std::string("shadow-stack"))
375 return false;
376
377 LLVMContext &Context = F.getContext();
378
379 // Find calls to llvm.gcroot.
380 CollectRoots(F);
381
382 // If there are no roots in this function, then there is no need to add a
383 // stack map entry for it.
384 if (Roots.empty())
385 return false;
386
387 // Build the constant map and figure the type of the shadow stack entry.
388 Value *FrameMap = GetFrameMap(F);
389 Type *ConcreteStackEntryTy = GetConcreteStackEntryType(F);
390
391 // Build the shadow stack entry at the very start of the function.
392 BasicBlock::iterator IP = F.getEntryBlock().begin();
393 IRBuilder<> AtEntry(IP->getParent(), IP);
394
395 Instruction *StackEntry =
396 AtEntry.CreateAlloca(ConcreteStackEntryTy, nullptr, "gc_frame");
397
398 while (isa(IP))
399 ++IP;
400 AtEntry.SetInsertPoint(IP->getParent(), IP);
401
402 // Initialize the map pointer and load the current head of the shadow stack.
403 Instruction *CurrentHead = AtEntry.CreateLoad(Head, "gc_currhead");
404 Instruction *EntryMapPtr =
405 CreateGEP(Context, AtEntry, StackEntry, 0, 1, "gc_frame.map");
406 AtEntry.CreateStore(FrameMap, EntryMapPtr);
407
408 // After all the allocas...
409 for (unsigned I = 0, E = Roots.size(); I != E; ++I) {
410 // For each root, find the corresponding slot in the aggregate...
411 Value *SlotPtr = CreateGEP(Context, AtEntry, StackEntry, 1 + I, "gc_root");
412
413 // And use it in lieu of the alloca.
414 AllocaInst *OriginalAlloca = Roots[I].second;
415 SlotPtr->takeName(OriginalAlloca);
416 OriginalAlloca->replaceAllUsesWith(SlotPtr);
417 }
418
419 // Move past the original stores inserted by GCStrategy::InitRoots. This isn't
420 // really necessary (the collector would never see the intermediate state at
421 // runtime), but it's nicer not to push the half-initialized entry onto the
422 // shadow stack.
423 while (isa(IP))
424 ++IP;
425 AtEntry.SetInsertPoint(IP->getParent(), IP);
426
427 // Push the entry onto the shadow stack.
428 Instruction *EntryNextPtr =
429 CreateGEP(Context, AtEntry, StackEntry, 0, 0, "gc_frame.next");
430 Instruction *NewHeadVal =
431 CreateGEP(Context, AtEntry, StackEntry, 0, "gc_newhead");
432 AtEntry.CreateStore(CurrentHead, EntryNextPtr);
433 AtEntry.CreateStore(NewHeadVal, Head);
434
435 // For each instruction that escapes...
436 EscapeEnumerator EE(F, "gc_cleanup");
437 while (IRBuilder<> *AtExit = EE.Next()) {
438 // Pop the entry from the shadow stack. Don't reuse CurrentHead from
439 // AtEntry, since that would make the value live for the entire function.
440 Instruction *EntryNextPtr2 =
441 CreateGEP(Context, *AtExit, StackEntry, 0, 0, "gc_frame.next");
442 Value *SavedHead = AtExit->CreateLoad(EntryNextPtr2, "gc_savedhead");
443 AtExit->CreateStore(SavedHead, Head);
444 }
445
446 // Delete the original allocas (which are no longer used) and the intrinsic
447 // calls (which are no longer valid). Doing this last avoids invalidating
448 // iterators.
449 for (unsigned I = 0, E = Roots.size(); I != E; ++I) {
450 Roots[I].first->eraseFromParent();
451 Roots[I].second->eraseFromParent();
452 }
453
454 Roots.clear();
455 return true;
456 }