llvm.org GIT mirror llvm / f91f5af
Start removing the use of an ad-hoc 'never inline' set and instead directly query the function information which this set was representing. This simplifies the interface of the inline cost analysis, and makes the always-inline pass significantly more efficient. Previously, always-inline would first make a single set of every function in the module *except* those marked with the always-inline attribute. It would then query this set at every call site to see if the function was a member of the set, and if so, refuse to inline it. This is quite wasteful. Instead, simply check the function attribute directly when looking at the callsite. The normal inliner also had similar redundancy. It added every function in the module with the noinline attribute to its set to ignore, even though inside the cost analysis function we *already tested* the noinline attribute and produced the same result. The only tricky part of removing this is that we have to be able to correctly remove only the functions inlined by the always-inline pass when finalizing, which requires a bit of a hack. Still, much less of a hack than the set of all non-always-inline functions was. While I was touching this function, I switched a heavy-weight set to a vector with sort+unique. The algorithm already had a two-phase insert and removal pattern, we were just needlessly paying the uniquing cost on every insert. This probably speeds up some compiles by a small amount (-O0 compiles with lots of always-inline, so potentially heavy libc++ users), but I've not tried to measure it. I believe there is no functional change here, but yell if you spot one. None are intended. Finally, the direction this is going in is to greatly simplify the inline cost query interface so that we can replace its implementation with a much more clever one. Along the way, all the APIs get simplified, so it seems incrementally good. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@152903 91177308-0d34-0410-b5e6-96231b3b80d8 Chandler Carruth 7 years ago
6 changed file(s) with 69 addition(s) and 56 deletion(s). Raw diff Collapse all Expand all
165165 /// getInlineCost - The heuristic used to determine if we should inline the
166166 /// function call or not.
167167 ///
168 InlineCost getInlineCost(CallSite CS,
169 SmallPtrSet &NeverInline);
168 InlineCost getInlineCost(CallSite CS);
170169 /// getCalledFunction - The heuristic used to determine if we should inline
171170 /// the function call or not. The callee is explicitly specified, to allow
172171 /// you to calculate the cost of inlining a function via a pointer. The
173172 /// result assumes that the inlined version will always be used. You should
174173 /// weight it yourself in cases where this callee will not always be called.
175 InlineCost getInlineCost(CallSite CS,
176 Function *Callee,
177 SmallPtrSet &NeverInline);
174 InlineCost getInlineCost(CallSite CS, Function *Callee);
178175
179176 /// getInlineFudgeFactor - Return a > 1.0 factor if the inliner should use a
180177 /// higher threshold to determine if the function call should be inlined.
7878 /// has been inlined.
7979 virtual void growCachedCostInfo(Function *Caller, Function *Callee) = 0;
8080
81 /// removeDeadFunctions - Remove dead functions that are not included in
82 /// DNR (Do Not Remove) list.
83 bool removeDeadFunctions(CallGraph &CG,
84 SmallPtrSet *DNR = NULL);
81 /// removeDeadFunctions - Remove dead functions.
82 ///
83 /// This also includes a hack in the form of the 'AlwaysInlineOnly' flag
84 /// which restricts it to deleting functions with an 'AlwaysInline'
85 /// attribute. This is useful for the InlineAlways pass that only wants to
86 /// deal with that subset of the functions.
87 bool removeDeadFunctions(CallGraph &CG, bool AlwaysInlineOnly = false);
88
8589 private:
8690 // InlineThreshold - Cache the value here for easy access.
8791 unsigned InlineThreshold;
519519 // getInlineCost - The heuristic used to determine if we should inline the
520520 // function call or not.
521521 //
522 InlineCost InlineCostAnalyzer::getInlineCost(CallSite CS,
523 SmallPtrSet &NeverInline) {
524 return getInlineCost(CS, CS.getCalledFunction(), NeverInline);
525 }
526
527 InlineCost InlineCostAnalyzer::getInlineCost(CallSite CS,
528 Function *Callee,
529 SmallPtrSet &NeverInline) {
522 InlineCost InlineCostAnalyzer::getInlineCost(CallSite CS) {
523 return getInlineCost(CS, CS.getCalledFunction());
524 }
525
526 InlineCost InlineCostAnalyzer::getInlineCost(CallSite CS, Function *Callee) {
530527 Instruction *TheCall = CS.getInstruction();
531528 Function *Caller = TheCall->getParent()->getParent();
532529
533530 // Don't inline functions which can be redefined at link-time to mean
534531 // something else. Don't inline functions marked noinline or call sites
535532 // marked noinline.
536 if (Callee->mayBeOverridden() ||
537 Callee->hasFnAttr(Attribute::NoInline) || NeverInline.count(Callee) ||
533 if (Callee->mayBeOverridden() || Callee->hasFnAttr(Attribute::NoInline) ||
538534 CS.isNoInline())
539535 return llvm::InlineCost::getNever();
540536
3131
3232 // AlwaysInliner only inlines functions that are mark as "always inline".
3333 class AlwaysInliner : public Inliner {
34 // Functions that are never inlined
35 SmallPtrSet NeverInline;
3634 InlineCostAnalyzer CA;
3735 public:
3836 // Use extremely low threshold.
4543 }
4644 static char ID; // Pass identification, replacement for typeid
4745 InlineCost getInlineCost(CallSite CS) {
48 return CA.getInlineCost(CS, NeverInline);
46 Function *Callee = CS.getCalledFunction();
47 // We assume indirect calls aren't calling an always-inline function.
48 if (!Callee) return InlineCost::getNever();
49
50 // We can't inline calls to external functions.
51 // FIXME: We shouldn't even get here.
52 if (Callee->isDeclaration()) return InlineCost::getNever();
53
54 // Return never for anything not marked as always inline.
55 if (!Callee->hasFnAttr(Attribute::AlwaysInline))
56 return InlineCost::getNever();
57
58 // We still have to check the inline cost in case there are reasons to
59 // not inline which trump the always-inline attribute such as setjmp and
60 // indirectbr.
61 return CA.getInlineCost(CS);
4962 }
5063 float getInlineFudgeFactor(CallSite CS) {
5164 return CA.getInlineFudgeFactor(CS);
5770 CA.growCachedCostInfo(Caller, Callee);
5871 }
5972 virtual bool doFinalization(CallGraph &CG) {
60 return removeDeadFunctions(CG, &NeverInline);
73 return removeDeadFunctions(CG, /*AlwaysInlineOnly=*/true);
6174 }
6275 virtual bool doInitialization(CallGraph &CG);
6376 void releaseMemory() {
8396 // been annotated with the "always inline" attribute.
8497 bool AlwaysInliner::doInitialization(CallGraph &CG) {
8598 CA.setTargetData(getAnalysisIfAvailable());
86
87 Module &M = CG.getModule();
88
89 for (Module::iterator I = M.begin(), E = M.end(); I != E; ++I)
90 if (!I->isDeclaration() && !I->hasFnAttr(Attribute::AlwaysInline))
91 NeverInline.insert(I);
92
9399 return false;
94100 }
4242 }
4343 static char ID; // Pass identification, replacement for typeid
4444 InlineCost getInlineCost(CallSite CS) {
45 return CA.getInlineCost(CS, NeverInline);
45 // Filter out functions which should never be inlined due to the global
46 // 'llvm.noinline'.
47 // FIXME: I'm 99% certain that this is an ancient bit of legacy that we
48 // no longer need to support, but I don't want to blindly nuke it just
49 // yet.
50 if (Function *Callee = CS.getCalledFunction())
51 if (NeverInline.count(Callee))
52 return InlineCost::getNever();
53
54 return CA.getInlineCost(CS);
4655 }
4756 float getInlineFudgeFactor(CallSite CS) {
4857 return CA.getInlineFudgeFactor(CS);
8089
8190 Module &M = CG.getModule();
8291
83 for (Module::iterator I = M.begin(), E = M.end();
84 I != E; ++I)
85 if (!I->isDeclaration() && I->hasFnAttr(Attribute::NoInline))
86 NeverInline.insert(I);
87
8892 // Get llvm.noinline
8993 GlobalVariable *GV = M.getNamedGlobal("llvm.noinline");
9094
555555
556556 /// removeDeadFunctions - Remove dead functions that are not included in
557557 /// DNR (Do Not Remove) list.
558 bool Inliner::removeDeadFunctions(CallGraph &CG,
559 SmallPtrSet *DNR) {
560 SmallPtrSet FunctionsToRemove;
558 bool Inliner::removeDeadFunctions(CallGraph &CG, bool AlwaysInlineOnly) {
559 SmallVector FunctionsToRemove;
561560
562561 // Scan for all of the functions, looking for ones that should now be removed
563562 // from the program. Insert the dead ones in the FunctionsToRemove set.
564563 for (CallGraph::iterator I = CG.begin(), E = CG.end(); I != E; ++I) {
565564 CallGraphNode *CGN = I->second;
566 if (CGN->getFunction() == 0)
565 Function *F = CGN->getFunction();
566 if (!F || F->isDeclaration())
567567 continue;
568
569 Function *F = CGN->getFunction();
570
568
569 // Handle the case when this function is called and we only want to care
570 // about always-inline functions. This is a bit of a hack to share code
571 // between here and the InlineAlways pass.
572 if (AlwaysInlineOnly && !F->hasFnAttr(Attribute::AlwaysInline))
573 continue;
574
571575 // If the only remaining users of the function are dead constants, remove
572576 // them.
573577 F->removeDeadConstantUsers();
574578
575 if (DNR && DNR->count(F))
576 continue;
577579 if (!F->isDefTriviallyDead())
578580 continue;
579581
586588 CG.getExternalCallingNode()->removeAnyCallEdgeTo(CGN);
587589
588590 // Removing the node for callee from the call graph and delete it.
589 FunctionsToRemove.insert(CGN);
590 }
591 FunctionsToRemove.push_back(CGN);
592 }
593 if (FunctionsToRemove.empty())
594 return false;
591595
592596 // Now that we know which functions to delete, do so. We didn't want to do
593597 // this inline, because that would invalidate our CallGraph::iterator
594598 // objects. :(
595599 //
596 // Note that it doesn't matter that we are iterating over a non-stable set
600 // Note that it doesn't matter that we are iterating over a non-stable order
597601 // here to do this, it doesn't matter which order the functions are deleted
598602 // in.
599 bool Changed = false;
600 for (SmallPtrSet::iterator I = FunctionsToRemove.begin(),
601 E = FunctionsToRemove.end(); I != E; ++I) {
603 std::sort(FunctionsToRemove.begin(), FunctionsToRemove.end());
604 FunctionsToRemove.erase(std::unique(FunctionsToRemove.begin(),
605 FunctionsToRemove.end()),
606 FunctionsToRemove.end());
607 for (SmallVectorImpl::iterator I = FunctionsToRemove.begin(),
608 E = FunctionsToRemove.end();
609 I != E; ++I) {
602610 resetCachedCostInfo((*I)->getFunction());
603611 delete CG.removeFunctionFromModule(*I);
604612 ++NumDeleted;
605 Changed = true;
606 }
607
608 return Changed;
609 }
613 }
614 return true;
615 }