llvm.org GIT mirror llvm / fe40a5a
[PM] Fix a nasty bug in the new PM where we failed to properly invalidation of analyses when merging SCCs. While I've added a bunch of testing of this, it takes something much more like the inliner to really trigger this as you need to have partially-analyzed SCCs with updates at just the right time. So I've added a direct test for this using the inliner and verifying the domtree. Without the changes here, this test ends up finding a stale dominator tree. However, to handle this properly, we need to invalidate analyses *before* merging the SCCs. After talking to Philip and Sanjoy about this they convinced me this was the right approach. To do this, we need a callback mechanism when merging SCCs so we can observe the cycle that will be merged before the merge happens. This API update ended up being surprisingly easy. With this commit, the new PM passes the test-suite again. It hadn't since MemorySSA was enabled for EarlyCSE as that also will find this bug very quickly. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@307498 91177308-0d34-0410-b5e6-96231b3b80d8 Chandler Carruth 3 years ago
5 changed file(s) with 156 addition(s) and 43 deletion(s). Raw diff Collapse all Expand all
651651 /// Make an existing internal ref edge into a call edge.
652652 ///
653653 /// This may form a larger cycle and thus collapse SCCs into TargetN's SCC.
654 /// If that happens, the deleted SCC pointers are returned. These SCCs are
655 /// not in a valid state any longer but the pointers will remain valid
656 /// until destruction of the parent graph instance for the purpose of
657 /// clearing cached information.
654 /// If that happens, the optional callback \p MergedCB will be invoked (if
655 /// provided) on the SCCs being merged away prior to actually performing
656 /// the merge. Note that this will never include the target SCC as that
657 /// will be the SCC functions are merged into to resolve the cycle. Once
658 /// this function returns, these merged SCCs are not in a valid state but
659 /// the pointers will remain valid until destruction of the parent graph
660 /// instance for the purpose of clearing cached information. This function
661 /// also returns 'true' if a cycle was formed and some SCCs merged away as
662 /// a convenience.
658663 ///
659664 /// After this operation, both SourceN's SCC and TargetN's SCC may move
660665 /// position within this RefSCC's postorder list. Any SCCs merged are
661666 /// merged into the TargetN's SCC in order to preserve reachability analyses
662667 /// which took place on that SCC.
663 SmallVector switchInternalEdgeToCall(Node &SourceN,
664 Node &TargetN);
668 bool switchInternalEdgeToCall(
669 Node &SourceN, Node &TargetN,
670 function_ref MergedSCCs)> MergeCB = {});
665671
666672 /// Make an existing internal call edge between separate SCCs into a ref
667673 /// edge.
569569 // Otherwise we are switching an internal ref edge to a call edge. This
570570 // may merge away some SCCs, and we add those to the UpdateResult. We also
571571 // need to make sure to update the worklist in the event SCCs have moved
572 // before the current one in the post-order sequence.
572 // before the current one in the post-order sequence
573 bool HasFunctionAnalysisProxy = false;
573574 auto InitialSCCIndex = RC->find(*C) - RC->begin();
574 auto InvalidatedSCCs = RC->switchInternalEdgeToCall(N, *CallTarget);
575 if (!InvalidatedSCCs.empty()) {
575 bool FormedCycle = RC->switchInternalEdgeToCall(
576 N, *CallTarget, [&](ArrayRef MergedSCCs) {
577 for (SCC *MergedC : MergedSCCs) {
578 assert(MergedC != &TargetC && "Cannot merge away the target SCC!");
579
580 HasFunctionAnalysisProxy |=
581 AM.getCachedResult(
582 *MergedC) != nullptr;
583
584 // Mark that this SCC will no longer be valid.
585 UR.InvalidatedSCCs.insert(MergedC);
586
587 // FIXME: We should really do a 'clear' here to forcibly release
588 // memory, but we don't have a good way of doing that and
589 // preserving the function analyses.
590 auto PA = PreservedAnalyses::allInSet>();
591 PA.preserve();
592 AM.invalidate(*MergedC, PA);
593 }
594 });
595
596 // If we formed a cycle by creating this call, we need to update more data
597 // structures.
598 if (FormedCycle) {
576599 C = &TargetC;
577600 assert(G.lookupSCC(N) == C && "Failed to update current SCC!");
578601
602 // If one of the invalidated SCCs had a cached proxy to a function
603 // analysis manager, we need to create a proxy in the new current SCC as
604 // the invaliadted SCCs had their functions moved.
605 if (HasFunctionAnalysisProxy)
606 AM.getResult(*C, G);
607
579608 // Any analyses cached for this SCC are no longer precise as the shape
580 // has changed by introducing this cycle.
581 AM.invalidate(*C, PreservedAnalyses::none());
582
583 for (SCC *InvalidatedC : InvalidatedSCCs) {
584 assert(InvalidatedC != C && "Cannot invalidate the current SCC!");
585 UR.InvalidatedSCCs.insert(InvalidatedC);
586
587 // Also clear any cached analyses for the SCCs that are dead. This
588 // isn't really necessary for correctness but can release memory.
589 AM.clear(*InvalidatedC);
590 }
609 // has changed by introducing this cycle. However, we have taken care to
610 // update the proxies so it remains valide.
611 auto PA = PreservedAnalyses::allInSet>();
612 PA.preserve();
613 AM.invalidate(*C, PA);
591614 }
592615 auto NewSCCIndex = RC->find(*C) - RC->begin();
593616 if (InitialSCCIndex < NewSCCIndex) {
455455 return make_range(SCCs.begin() + SourceIdx, SCCs.begin() + TargetIdx);
456456 }
457457
458 SmallVector
459 LazyCallGraph::RefSCC::switchInternalEdgeToCall(Node &SourceN, Node &TargetN) {
458 bool
459 LazyCallGraph::RefSCC::switchInternalEdgeToCall(
460 Node &SourceN, Node &TargetN,
461 function_ref MergeSCCs)> MergeCB) {
460462 assert(!(*SourceN)[TargetN].isCall() && "Must start with a ref edge!");
461463 SmallVector DeletedSCCs;
462464
474476 // we've just added more connectivity.
475477 if (&SourceSCC == &TargetSCC) {
476478 SourceN->setEdgeKind(TargetN, Edge::Call);
477 return DeletedSCCs;
479 return false; // No new cycle.
478480 }
479481
480482 // At this point we leverage the postorder list of SCCs to detect when the
487489 int TargetIdx = SCCIndices[&TargetSCC];
488490 if (TargetIdx < SourceIdx) {
489491 SourceN->setEdgeKind(TargetN, Edge::Call);
490 return DeletedSCCs;
492 return false; // No new cycle.
491493 }
492494
493495 // Compute the SCCs which (transitively) reach the source.
554556 SourceSCC, TargetSCC, SCCs, SCCIndices, ComputeSourceConnectedSet,
555557 ComputeTargetConnectedSet);
556558
559 // Run the user's callback on the merged SCCs before we actually merge them.
560 if (MergeCB)
561 MergeCB(makeArrayRef(MergeRange.begin(), MergeRange.end()));
562
557563 // If the merge range is empty, then adding the edge didn't actually form any
558564 // new cycles. We're done.
559565 if (MergeRange.begin() == MergeRange.end()) {
560566 // Now that the SCC structure is finalized, flip the kind to call.
561567 SourceN->setEdgeKind(TargetN, Edge::Call);
562 return DeletedSCCs;
568 return false; // No new cycle.
563569 }
564570
565571 #ifndef NDEBUG
595601 // Now that the SCC structure is finalized, flip the kind to call.
596602 SourceN->setEdgeKind(TargetN, Edge::Call);
597603
598 // And we're done!
599 return DeletedSCCs;
604 // And we're done, but we did form a new cycle.
605 return true;
600606 }
601607
602608 void LazyCallGraph::RefSCC::switchTrivialInternalEdgeToRef(Node &SourceN,
126126 ret void
127127 ; CHECK: ret void
128128 }
129
130 ; The 'test2_' prefixed code works to carefully trigger forming an SCC with
131 ; a dominator tree for one of the functions but not the other and without even
132 ; a function analysis manager proxy for the SCC that things get merged into.
133 ; Without proper handling when updating the call graph this will find a stale
134 ; dominator tree.
135
136 @test2_global = external global i32, align 4
137
138 define void @test2_hoge(i1 (i32*)* %arg) {
139 ; CHECK-LABEL: define void @test2_hoge(
140 bb:
141 %tmp2 = call zeroext i1 %arg(i32* @test2_global)
142 ; CHECK: call zeroext i1 %arg(
143 br label %bb3
144
145 bb3:
146 %tmp5 = call zeroext i1 %arg(i32* @test2_global)
147 ; CHECK: call zeroext i1 %arg(
148 br i1 %tmp5, label %bb3, label %bb6
149
150 bb6:
151 ret void
152 }
153
154 define zeroext i1 @test2_widget(i32* %arg) {
155 ; CHECK-LABEL: define zeroext i1 @test2_widget(
156 bb:
157 %tmp1 = alloca i8, align 1
158 %tmp2 = alloca i32, align 4
159 call void @test2_quux()
160 ; CHECK-NOT: call
161 ;
162 ; CHECK: call zeroext i1 @test2_widget(i32* @test2_global)
163 ; CHECK-NEXT: br label %[[NEW_BB:.*]]
164 ;
165 ; CHECK: [[NEW_BB]]:
166 ; CHECK-NEXT: call zeroext i1 @test2_widget(i32* @test2_global)
167 ;
168 ; CHECK: {{.*}}:
169
170 call void @test2_hoge.1(i32* %arg)
171 ; CHECK-NEXT: call void @test2_hoge.1(
172
173 %tmp4 = call zeroext i1 @test2_barney(i32* %tmp2)
174 %tmp5 = zext i1 %tmp4 to i32
175 store i32 %tmp5, i32* %tmp2, align 4
176 %tmp6 = call zeroext i1 @test2_barney(i32* null)
177 call void @test2_ham(i8* %tmp1)
178 ; CHECK: call void @test2_ham(
179
180 call void @test2_quux()
181 ; CHECK-NOT: call
182 ;
183 ; CHECK: call zeroext i1 @test2_widget(i32* @test2_global)
184 ; CHECK-NEXT: br label %[[NEW_BB:.*]]
185 ;
186 ; CHECK: [[NEW_BB]]:
187 ; CHECK-NEXT: call zeroext i1 @test2_widget(i32* @test2_global)
188 ;
189 ; CHECK: {{.*}}:
190 ret i1 true
191 ; CHECK-NEXT: ret i1 true
192 }
193
194 define internal void @test2_quux() {
195 ; CHECK-NOT: @test2_quux
196 bb:
197 call void @test2_hoge(i1 (i32*)* @test2_widget)
198 ret void
199 }
200
201 declare void @test2_hoge.1(i32*)
202
203 declare zeroext i1 @test2_barney(i32*)
204
205 declare void @test2_ham(i8*)
12761276 // be invalidated.
12771277 LazyCallGraph::SCC &AC = *CG.lookupSCC(A);
12781278 LazyCallGraph::SCC &CC = *CG.lookupSCC(C);
1279 auto InvalidatedSCCs = RC.switchInternalEdgeToCall(A, C);
1280 ASSERT_EQ(1u, InvalidatedSCCs.size());
1281 EXPECT_EQ(&AC, InvalidatedSCCs[0]);
1279 EXPECT_TRUE(RC.switchInternalEdgeToCall(A, C, [&](ArrayRef MergedCs) {
1280 ASSERT_EQ(1u, MergedCs.size());
1281 EXPECT_EQ(&AC, MergedCs[0]);
1282 }));
12821283 EXPECT_EQ(2, CC.size());
12831284 EXPECT_EQ(&CC, CG.lookupSCC(A));
12841285 EXPECT_EQ(&CC, CG.lookupSCC(C));
15851586
15861587 // Switch the ref edge from A -> D to a call edge. This should have no
15871588 // effect as it is already in postorder and no new cycles are formed.
1588 auto MergedCs = RC.switchInternalEdgeToCall(A, D);
1589 EXPECT_EQ(0u, MergedCs.size());
1589 EXPECT_FALSE(RC.switchInternalEdgeToCall(A, D));
15901590 ASSERT_EQ(4, RC.size());
15911591 EXPECT_EQ(&DC, &RC[0]);
15921592 EXPECT_EQ(&BC, &RC[1]);
15951595
15961596 // Switch B -> C to a call edge. This doesn't form any new cycles but does
15971597 // require reordering the SCCs.
1598 MergedCs = RC.switchInternalEdgeToCall(B, C);
1599 EXPECT_EQ(0u, MergedCs.size());
1598 EXPECT_FALSE(RC.switchInternalEdgeToCall(B, C));
16001599 ASSERT_EQ(4, RC.size());
16011600 EXPECT_EQ(&DC, &RC[0]);
16021601 EXPECT_EQ(&CC, &RC[1]);
16041603 EXPECT_EQ(&AC, &RC[3]);
16051604
16061605 // Switch C -> B to a call edge. This forms a cycle and forces merging SCCs.
1607 MergedCs = RC.switchInternalEdgeToCall(C, B);
1608 ASSERT_EQ(1u, MergedCs.size());
1609 EXPECT_EQ(&CC, MergedCs[0]);
1606 EXPECT_TRUE(RC.switchInternalEdgeToCall(C, B, [&](ArrayRef MergedCs) {
1607 ASSERT_EQ(1u, MergedCs.size());
1608 EXPECT_EQ(&CC, MergedCs[0]);
1609 }));
16101610 ASSERT_EQ(3, RC.size());
16111611 EXPECT_EQ(&DC, &RC[0]);
16121612 EXPECT_EQ(&BC, &RC[1]);
17191719 // Switch C3 -> B1 to a call edge. This doesn't form any new cycles but does
17201720 // require reordering the SCCs in the face of tricky internal node
17211721 // structures.
1722 auto MergedCs = RC.switchInternalEdgeToCall(C3, B1);
1723 EXPECT_EQ(0u, MergedCs.size());
1722 EXPECT_FALSE(RC.switchInternalEdgeToCall(C3, B1));
17241723 ASSERT_EQ(8, RC.size());
17251724 EXPECT_EQ(&DC, &RC[0]);
17261725 EXPECT_EQ(&B3C, &RC[1]);
18511850 // C F C | |
18521851 // \ / \ / |
18531852 // G G |
1854 auto MergedCs = RC.switchInternalEdgeToCall(F, B);
1855 ASSERT_EQ(2u, MergedCs.size());
1856 EXPECT_EQ(&FC, MergedCs[0]);
1857 EXPECT_EQ(&DC, MergedCs[1]);
1853 EXPECT_TRUE(RC.switchInternalEdgeToCall(
1854 F, B, [&](ArrayRef MergedCs) {
1855 ASSERT_EQ(2u, MergedCs.size());
1856 EXPECT_EQ(&FC, MergedCs[0]);
1857 EXPECT_EQ(&DC, MergedCs[1]);
1858 }));
18581859 EXPECT_EQ(3, BC.size());
18591860
18601861 // And make sure the postorder was updated.