llvm.org GIT mirror llvm / 6fbb41e
[PM] Teach the CGSCC's CG update utility to more carefully invalidate analyses when we're about to break apart an SCC. We can't wait until after breaking apart the SCC to invalidate things: 1) Which SCC do we then invalidate? All of them? 2) Even if we invalidate all of them, a newly created SCC may not have a proxy that will convey the invalidation to functions! Previously we only invalidated one of the SCCs and too late. This led to stale analyses remaining in the cache. And because the caching strategy actually works, they would get used and chaos would ensue. Doing invalidation early is somewhat pessimizing though if we *know* that the SCC structure won't change. So it turns out that the design to make the mutation API force the caller to know the *kind* of mutation in advance was indeed 100% correct and we didn't do enough of it. So this change also splits two cases of switching a call edge to a ref edge into two separate APIs so that callers can clearly test for this and take the easy path without invalidating when appropriate. This is particularly important in this case as we expect most inlines to be between functions in separate SCCs and so the common case is that we don't have to so aggressively invalidate analyses. The LCG API change in turn needed some basic cleanups and better testing in its unittest. No interesting functionality changed there other than more coverage of the returned sequence of SCCs. While this seems like an obvious improvement over the current state, I'd like to revisit the core concept of invalidating within the CG-update layer at all. I'm wondering if we would be better served forcing the callers to handle the invalidation beforehand in the cases that they can handle it. An interesting example is when we want to teach the inliner to *update and preserve* analyses. But we can cross that bridge when we get there. With this patch, the new pass manager an build all of the LLVM test suite at -O3 and everything passes. =D I haven't bootstrapped yet and I'm sure there are still plenty of bugs, but this gives a nice baseline so I'm going to increasingly focus on fleshing out the missing functionality, especially the bits that are just turned off right now in order to let us establish this baseline. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@290664 91177308-0d34-0410-b5e6-96231b3b80d8 Chandler Carruth 2 years ago
5 changed file(s) with 240 addition(s) and 69 deletion(s). Raw diff Collapse all Expand all
619619 SmallVector switchInternalEdgeToCall(Node &SourceN,
620620 Node &TargetN);
621621
622 /// Make an existing internal call edge into a ref edge.
623 ///
624 /// If SourceN and TargetN are part of a single SCC, it may be split up due
625 /// to breaking a cycle in the call edges that formed it. If that happens,
626 /// then this routine will insert new SCCs into the postorder list *before*
627 /// the SCC of TargetN (previously the SCC of both). This preserves
628 /// postorder as the TargetN can reach all of the other nodes by definition
629 /// of previously being in a single SCC formed by the cycle from SourceN to
630 /// TargetN.
622 /// Make an existing internal call edge between separate SCCs into a ref
623 /// edge.
624 ///
625 /// If SourceN and TargetN in separate SCCs within this RefSCC, changing
626 /// the call edge between them to a ref edge is a trivial operation that
627 /// does not require any structural changes to the call graph.
628 void switchTrivialInternalEdgeToRef(Node &SourceN, Node &TargetN);
629
630 /// Make an existing internal call edge within a single SCC into a ref
631 /// edge.
632 ///
633 /// Since SourceN and TargetN are part of a single SCC, this SCC may be
634 /// split up due to breaking a cycle in the call edges that formed it. If
635 /// that happens, then this routine will insert new SCCs into the postorder
636 /// list *before* the SCC of TargetN (previously the SCC of both). This
637 /// preserves postorder as the TargetN can reach all of the other nodes by
638 /// definition of previously being in a single SCC formed by the cycle from
639 /// SourceN to TargetN.
631640 ///
632641 /// The newly added SCCs are added *immediately* and contiguously
633642 /// prior to the TargetN SCC and return the range covering the new SCCs in
634643 /// the RefSCC's postorder sequence. You can directly iterate the returned
635644 /// range to observe all of the new SCCs in postorder.
645 ///
646 /// Note that if SourceN and TargetN are in separate SCCs, the simpler
647 /// routine `switchTrivialInternalEdgeToRef` should be used instead.
636648 iterator_range switchInternalEdgeToRef(Node &SourceN,
637649 Node &TargetN);
638650
228228 if (NewSCCRange.begin() == NewSCCRange.end())
229229 return C;
230230
231 // Invalidate the analyses of the current SCC and add it to the worklist since
232 // it has changed its shape.
233 AM.invalidate(*C, PreservedAnalyses::none());
231 // Add the current SCC to the worklist as its shape has changed.
234232 UR.CWorklist.insert(C);
235233 if (DebugLogging)
236234 dbgs() << "Enqueuing the existing SCC in the worklist:" << *C << "\n";
342340 dbgs() << "Deleting internal " << (IsCall ? "call" : "ref")
343341 << " edge from '" << N << "' to '" << TargetN << "'\n";
344342
345 if (IsCall)
346 C = incorporateNewSCCRange(RC->switchInternalEdgeToRef(N, TargetN), G, N,
347 C, AM, UR, DebugLogging);
343 if (IsCall) {
344 if (C != &TargetC) {
345 // For separate SCCs this is trivial.
346 RC->switchTrivialInternalEdgeToRef(N, TargetN);
347 } else {
348 // Otherwise we may end up re-structuring the call graph. First,
349 // invalidate any SCC analyses. We have to do this before we split
350 // functions into new SCCs and lose track of where their analyses are
351 // cached.
352 // FIXME: We should accept a more precise preserved set here. For
353 // example, it might be possible to preserve some function analyses
354 // even as the SCC structure is changed.
355 AM.invalidate(*C, PreservedAnalyses::none());
356 // Now update the call graph.
357 C = incorporateNewSCCRange(RC->switchInternalEdgeToRef(N, TargetN), G,
358 N, C, AM, UR, DebugLogging);
359 }
360 }
348361
349362 auto NewRefSCCs = RC->removeInternalRefEdge(N, TargetN);
350363 if (!NewRefSCCs.empty()) {
400413 continue;
401414 }
402415
403 // Otherwise we are switching an internal call edge to a ref edge. This
404 // may split up some SCCs.
405 C = incorporateNewSCCRange(RC->switchInternalEdgeToRef(N, TargetN), G, N, C,
406 AM, UR, DebugLogging);
416 // We are switching an internal call edge to a ref edge. This may split up
417 // some SCCs.
418 if (C != &TargetC) {
419 // For separate SCCs this is trivial.
420 RC->switchTrivialInternalEdgeToRef(N, TargetN);
421 continue;
422 }
423
424 // Otherwise we may end up re-structuring the call graph. First, invalidate
425 // any SCC analyses. We have to do this before we split functions into new
426 // SCCs and lose track of where their analyses are cached.
427 // FIXME: We should accept a more precise preserved set here. For example,
428 // it might be possible to preserve some function analyses even as the SCC
429 // structure is changed.
430 AM.invalidate(*C, PreservedAnalyses::none());
431 // Now update the call graph.
432 C = incorporateNewSCCRange(RC->switchInternalEdgeToRef(N, TargetN), G,
433 N, C, AM, UR, DebugLogging);
407434 }
408435
409436 // Now promote ref edges into call edges.
605605 return DeletedSCCs;
606606 }
607607
608 iterator_range
609 LazyCallGraph::RefSCC::switchInternalEdgeToRef(Node &SourceN, Node &TargetN) {
608 void LazyCallGraph::RefSCC::switchTrivialInternalEdgeToRef(Node &SourceN,
609 Node &TargetN) {
610610 assert(SourceN[TargetN].isCall() && "Must start with a call edge!");
611611
612612 #ifndef NDEBUG
616616 auto VerifyOnExit = make_scope_exit([&]() { verify(); });
617617 #endif
618618
619 SCC &SourceSCC = *G->lookupSCC(SourceN);
620 SCC &TargetSCC = *G->lookupSCC(TargetN);
621
622 assert(&SourceSCC.getOuterRefSCC() == this &&
619 assert(G->lookupRefSCC(SourceN) == this &&
623620 "Source must be in this RefSCC.");
624 assert(&TargetSCC.getOuterRefSCC() == this &&
621 assert(G->lookupRefSCC(TargetN) == this &&
625622 "Target must be in this RefSCC.");
623 assert(G->lookupSCC(SourceN) != G->lookupSCC(TargetN) &&
624 "Source and Target must be in separate SCCs for this to be trivial!");
626625
627626 // Set the edge kind.
628627 SourceN.setEdgeKind(TargetN.getFunction(), Edge::Ref);
629
630 // If this call edge is just connecting two separate SCCs within this RefSCC,
631 // there is nothing to do.
632 if (&SourceSCC != &TargetSCC)
633 return make_range(SCCs.end(), SCCs.end());
628 }
629
630 iterator_range
631 LazyCallGraph::RefSCC::switchInternalEdgeToRef(Node &SourceN, Node &TargetN) {
632 assert(SourceN[TargetN].isCall() && "Must start with a call edge!");
633
634 #ifndef NDEBUG
635 // In a debug build, verify the RefSCC is valid to start with and when this
636 // routine finishes.
637 verify();
638 auto VerifyOnExit = make_scope_exit([&]() { verify(); });
639 #endif
640
641 assert(G->lookupRefSCC(SourceN) == this &&
642 "Source must be in this RefSCC.");
643 assert(G->lookupRefSCC(TargetN) == this &&
644 "Target must be in this RefSCC.");
645
646 SCC &TargetSCC = *G->lookupSCC(TargetN);
647 assert(G->lookupSCC(SourceN) == &TargetSCC && "Source and Target must be in "
648 "the same SCC to require the "
649 "full CG update.");
650
651 // Set the edge kind.
652 SourceN.setEdgeKind(TargetN.getFunction(), Edge::Ref);
634653
635654 // Otherwise we are removing a call edge from a single SCC. This may break
636655 // the cycle. In order to compute the new set of SCCs, we need to do a small
0 ; This test tries to ensure that the inliner successfully invalidates function
1 ; analyses after inlining into the function body.
2 ;
3 ; The strategy for these tests is to compute domtree over all the functions,
4 ; then run the inliner, and then verify the domtree. Then we can arrange the
5 ; inline to disturb the domtree (easy) and detect any stale cached entries in
6 ; the verifier. We do the initial computation both *inside* the CGSCC walk and
7 ; in a pre-step to make sure both work.
8 ;
9 ; RUN: opt < %s -passes='function(require),cgscc(inline,function(verify))' -S | FileCheck %s
10 ; RUN: opt < %s -passes='cgscc(function(require),inline,function(verify))' -S | FileCheck %s
11
12 ; An external function used to control branches.
13 declare i1 @flag()
14 ; CHECK-LABEL: declare i1 @flag()
15
16 ; The utility function with interesting control flow that gets inlined below to
17 ; perturb the dominator tree.
18 define internal void @callee() {
19 ; CHECK-LABEL: @callee
20 entry:
21 %ptr = alloca i8
22 %flag = call i1 @flag()
23 br i1 %flag, label %then, label %else
24
25 then:
26 store volatile i8 42, i8* %ptr
27 br label %return
28
29 else:
30 store volatile i8 -42, i8* %ptr
31 br label %return
32
33 return:
34 ret void
35 }
36
37
38 ; The 'test1_' prefixed functions test the basic scenario of inlining
39 ; destroying dominator tree.
40
41 define void @test1_caller() {
42 ; CHECK-LABEL: define void @test1_caller()
43 entry:
44 call void @callee()
45 ; CHECK-NOT: @callee
46 ret void
47 ; CHECK: ret void
48 }
49
50
51 ; The 'test2_' prefixed functions test the scenario of not inlining preserving
52 ; dominators.
53
54 define void @test2_caller() {
55 ; CHECK-LABEL: define void @test2_caller()
56 entry:
57 call void @callee() noinline
58 ; CHECK: call void @callee
59 ret void
60 ; CHECK: ret void
61 }
62
63
64 ; The 'test3_' prefixed functions test the scenario of not inlining preserving
65 ; dominators after splitting an SCC into two smaller SCCs.
66
67 ; The first function gets visited first and we end up inlining everything we
68 ; can into this routine. That splits test3_g into a separate SCC that is enqued
69 ; for later processing.
70 define void @test3_f() {
71 ; CHECK-LABEL: define void @test3_f()
72 entry:
73 ; Create the first edge in the SCC cycle.
74 call void @test3_g()
75 ; CHECK-NOT: @test3_g()
76 ; CHECK: call void @test3_f()
77
78 ; Pull interesting CFG into this function.
79 call void @callee()
80 ; CHECK-NOT: call void @callee()
81
82 ret void
83 ; CHECK: ret void
84 }
85
86 ; This function ends up split into a separate SCC, which can cause its analyses
87 ; to become stale if the splitting doesn't properly invalidate things. Also, as
88 ; a consequence of being split out, test3_f is too large to inline by the time
89 ; we get here.
90 define void @test3_g() {
91 ; CHECK-LABEL: define void @test3_g()
92 entry:
93 ; Create the second edge in the SCC cycle.
94 call void @test3_f()
95 ; CHECK: call void @test3_f()
96
97 ; Pull interesting CFG into this function.
98 call void @callee()
99 ; CHECK-NOT: call void @callee()
100
101 ret void
102 ; CHECK: ret void
103 }
14951495 // Switch the call edge from 'b' to 'c' to a ref edge. This will break the
14961496 // call cycle and cause us to form more SCCs. The RefSCC will remain the same
14971497 // though.
1498 RC.switchInternalEdgeToRef(B, C);
1498 auto NewCs = RC.switchInternalEdgeToRef(B, C);
14991499 EXPECT_EQ(&RC, CG.lookupRefSCC(A));
15001500 EXPECT_EQ(&RC, CG.lookupRefSCC(B));
15011501 EXPECT_EQ(&RC, CG.lookupRefSCC(C));
15071507 EXPECT_EQ(&*J++, CG.lookupSCC(A));
15081508 EXPECT_EQ(&*J++, CG.lookupSCC(C));
15091509 EXPECT_EQ(RC.end(), J);
1510 // And the returned range must be the slice of this sequence containing new
1511 // SCCs.
1512 EXPECT_EQ(RC.begin(), NewCs.begin());
1513 EXPECT_EQ(std::prev(RC.end()), NewCs.end());
15101514
15111515 // Test turning the ref edge from A to C into a call edge. This will form an
15121516 // SCC out of A and C. Since we previously had a call edge from C to A, the
17091713 EXPECT_EQ(CG.postorder_ref_scc_end(), I);
17101714
17111715 EXPECT_EQ(1, RC.size());
1712 LazyCallGraph::SCC &CallC = *RC.begin();
1713
1714 LazyCallGraph::Node &A = *CG.lookup(lookupFunction(*M, "a"));
1715 LazyCallGraph::Node &B = *CG.lookup(lookupFunction(*M, "b"));
1716 LazyCallGraph::Node &C = *CG.lookup(lookupFunction(*M, "c"));
1717 EXPECT_EQ(&CallC, CG.lookupSCC(A));
1718 EXPECT_EQ(&CallC, CG.lookupSCC(B));
1719 EXPECT_EQ(&CallC, CG.lookupSCC(C));
1716 LazyCallGraph::SCC &AC = *RC.begin();
1717
1718 LazyCallGraph::Node &AN = *CG.lookup(lookupFunction(*M, "a"));
1719 LazyCallGraph::Node &BN = *CG.lookup(lookupFunction(*M, "b"));
1720 LazyCallGraph::Node &CN = *CG.lookup(lookupFunction(*M, "c"));
1721 EXPECT_EQ(&AC, CG.lookupSCC(AN));
1722 EXPECT_EQ(&AC, CG.lookupSCC(BN));
1723 EXPECT_EQ(&AC, CG.lookupSCC(CN));
17201724
17211725 // Remove the call edge from b -> a to a ref edge, which should leave the
17221726 // 3 functions still in a single connected component because of a -> b ->
17231727 // c -> a.
1724 RC.switchInternalEdgeToRef(B, A);
1728 auto NewCs = RC.switchInternalEdgeToRef(BN, AN);
1729 EXPECT_EQ(NewCs.begin(), NewCs.end());
17251730 EXPECT_EQ(1, RC.size());
1726 EXPECT_EQ(&CallC, CG.lookupSCC(A));
1727 EXPECT_EQ(&CallC, CG.lookupSCC(B));
1728 EXPECT_EQ(&CallC, CG.lookupSCC(C));
1731 EXPECT_EQ(&AC, CG.lookupSCC(AN));
1732 EXPECT_EQ(&AC, CG.lookupSCC(BN));
1733 EXPECT_EQ(&AC, CG.lookupSCC(CN));
17291734
17301735 // Remove the edge from c -> a, which should leave 'a' in the original SCC
17311736 // and form a new SCC for 'b' and 'c'.
1732 RC.switchInternalEdgeToRef(C, A);
1737 NewCs = RC.switchInternalEdgeToRef(CN, AN);
1738 EXPECT_EQ(1, std::distance(NewCs.begin(), NewCs.end()));
17331739 EXPECT_EQ(2, RC.size());
1734 EXPECT_EQ(&CallC, CG.lookupSCC(A));
1735 LazyCallGraph::SCC &BCallC = *CG.lookupSCC(B);
1736 EXPECT_NE(&BCallC, &CallC);
1737 EXPECT_EQ(&BCallC, CG.lookupSCC(C));
1738 auto J = RC.find(CallC);
1739 EXPECT_EQ(&CallC, &*J);
1740 EXPECT_EQ(&AC, CG.lookupSCC(AN));
1741 LazyCallGraph::SCC &BC = *CG.lookupSCC(BN);
1742 EXPECT_NE(&BC, &AC);
1743 EXPECT_EQ(&BC, CG.lookupSCC(CN));
1744 auto J = RC.find(AC);
1745 EXPECT_EQ(&AC, &*J);
17401746 --J;
1741 EXPECT_EQ(&BCallC, &*J);
1747 EXPECT_EQ(&BC, &*J);
17421748 EXPECT_EQ(RC.begin(), J);
1749 EXPECT_EQ(J, NewCs.begin());
17431750
17441751 // Remove the edge from c -> b, which should leave 'b' in the original SCC
17451752 // and form a new SCC for 'c'. It shouldn't change 'a's SCC.
1746 RC.switchInternalEdgeToRef(C, B);
1753 NewCs = RC.switchInternalEdgeToRef(CN, BN);
1754 EXPECT_EQ(1, std::distance(NewCs.begin(), NewCs.end()));
17471755 EXPECT_EQ(3, RC.size());
1748 EXPECT_EQ(&CallC, CG.lookupSCC(A));
1749 EXPECT_EQ(&BCallC, CG.lookupSCC(B));
1750 LazyCallGraph::SCC &CCallC = *CG.lookupSCC(C);
1751 EXPECT_NE(&CCallC, &CallC);
1752 EXPECT_NE(&CCallC, &BCallC);
1753 J = RC.find(CallC);
1754 EXPECT_EQ(&CallC, &*J);
1756 EXPECT_EQ(&AC, CG.lookupSCC(AN));
1757 EXPECT_EQ(&BC, CG.lookupSCC(BN));
1758 LazyCallGraph::SCC &CC = *CG.lookupSCC(CN);
1759 EXPECT_NE(&CC, &AC);
1760 EXPECT_NE(&CC, &BC);
1761 J = RC.find(AC);
1762 EXPECT_EQ(&AC, &*J);
17551763 --J;
1756 EXPECT_EQ(&BCallC, &*J);
1764 EXPECT_EQ(&BC, &*J);
17571765 --J;
1758 EXPECT_EQ(&CCallC, &*J);
1766 EXPECT_EQ(&CC, &*J);
17591767 EXPECT_EQ(RC.begin(), J);
1768 EXPECT_EQ(J, NewCs.begin());
17601769 }
17611770
17621771 TEST(LazyCallGraphTest, InternalRefEdgeToCall) {
19261935
19271936 // Several call edges are initially present to force a particual post-order.
19281937 // Remove them now, leaving an interleaved post-order pattern.
1929 RC.switchInternalEdgeToRef(B3, C3);
1930 RC.switchInternalEdgeToRef(C2, B3);
1931 RC.switchInternalEdgeToRef(B2, C2);
1932 RC.switchInternalEdgeToRef(C1, B2);
1933 RC.switchInternalEdgeToRef(B1, C1);
1938 RC.switchTrivialInternalEdgeToRef(B3, C3);
1939 RC.switchTrivialInternalEdgeToRef(C2, B3);
1940 RC.switchTrivialInternalEdgeToRef(B2, C2);
1941 RC.switchTrivialInternalEdgeToRef(C1, B2);
1942 RC.switchTrivialInternalEdgeToRef(B1, C1);
19341943
19351944 // Check the initial post-order. We ensure this order with the extra edges
19361945 // that are nuked above.
20532062 LazyCallGraph::SCC &GC = *CG.lookupSCC(G);
20542063
20552064 // Remove the extra edges that were used to force a particular post-order.
2056 RC.switchInternalEdgeToRef(C, D);
2057 RC.switchInternalEdgeToRef(D, E);
2065 RC.switchTrivialInternalEdgeToRef(C, D);
2066 RC.switchTrivialInternalEdgeToRef(D, E);
20582067
20592068 // Check the initial post-order. We ensure this order with the extra edges
20602069 // that are nuked above.