llvm.org GIT mirror
[Loop Predication] Teach LP about reverse loops Summary: Currently, we only support predication for forward loops with step of 1. This patch enables loop predication for reverse or countdownLoops, which satisfy the following conditions: 1. The step of the IV is -1. 2. The loop has a singe latch as B(X) = X <pred> latchLimit with pred as s> or u> 3. The IV of the guard is the decrement IV of the latch condition (Guard is: G(X) = X-1 u< guardLimit). This patch was downstream for a while and is the last series of patches that's from our LP implementation downstream. Reviewers: apilipenko, mkazantsev, sanjoy Subscribers: llvm-commits Differential Revision: https://reviews.llvm.org/D40353 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@319659 91177308-0d34-0410-b5e6-96231b3b80d8 Anna Thomas 1 year, 9 months ago
2 changed file(s) with 277 addition(s) and 60 deletion(s).
 97 97 // Note that we can use anything stronger than M, i.e. any condition which 98 98 // implies M. 99 99 // 100 // For now the transformation is limited to the following case:⏎ 100 // When S = 1 (i.e. forward iterating loop), the transformation is supported⏎ 101 // when: 101 102 // * The loop has a single latch with the condition of the form: 102 103 // B(X) = latchStart + X latchLimit, 103 104 // where is u<, u<=, s<, or s<=. 104 // * The step of the IV used in the latch condition is 1. 105 105 // * The guard condition is of the form 106 106 // G(X) = guardStart + X u< guardLimit 107 107 // 108 // For the ult latch comparison case M is: 109 // forall X . guardStart + X u< guardLimit && latchStart + X 110 // guardStart + X + 1 u< guardLimit 111 // 112 // The only way the antecedent can be true and the consequent can be false is 113 // if 114 // X == guardLimit - 1 - guardStart 115 // (and guardLimit is non-zero, but we won't use this latter fact). 116 // If X == guardLimit - 1 - guardStart then the second half of the antecedent is 117 // latchStart + guardLimit - 1 - guardStart u< latchLimit 118 // and its negation is 119 // latchStart + guardLimit - 1 - guardStart u>= latchLimit 120 // 121 // In other words, if 122 // latchLimit u<= latchStart + guardLimit - 1 - guardStart 123 // then: 124 // (the ranges below are written in ConstantRange notation, where [A, B) is the 125 // set for (I = A; I != B; I++ /*maywrap*/) yield(I);) 126 // 127 // forall X . guardStart + X u< guardLimit && 128 // latchStart + X u< latchLimit => 129 // guardStart + X + 1 u< guardLimit 130 // == forall X . guardStart + X u< guardLimit && 131 // latchStart + X u< latchStart + guardLimit - 1 - guardStart => 132 // guardStart + X + 1 u< guardLimit 133 // == forall X . (guardStart + X) in [0, guardLimit) && 134 // (latchStart + X) in [0, latchStart + guardLimit - 1 - guardStart) => 135 // (guardStart + X + 1) in [0, guardLimit) 136 // == forall X . X in [-guardStart, guardLimit - guardStart) && 137 // X in [-latchStart, guardLimit - 1 - guardStart) => 138 // X in [-guardStart - 1, guardLimit - guardStart - 1) 139 // == true 140 // 141 // So the widened condition is: 142 // guardStart u< guardLimit && 143 // latchStart + guardLimit - 1 - guardStart u>= latchLimit 144 // Similarly for ule condition the widened condition is: 145 // guardStart u< guardLimit && 146 // latchStart + guardLimit - 1 - guardStart u> latchLimit 147 // For slt condition the widened condition is: 148 // guardStart u< guardLimit && 149 // latchStart + guardLimit - 1 - guardStart s>= latchLimit 150 // For sle condition the widened condition is: 151 // guardStart u< guardLimit && 152 // latchStart + guardLimit - 1 - guardStart s> latchLimit 153 //⏎ 108 // For the ult latch comparison case M is:⏎ 109 // forall X . guardStart + X u< guardLimit && latchStart + X 110 // guardStart + X + 1 u< guardLimit 111 // 112 // The only way the antecedent can be true and the consequent can be false is 113 // if 114 // X == guardLimit - 1 - guardStart 115 // (and guardLimit is non-zero, but we won't use this latter fact). 116 // If X == guardLimit - 1 - guardStart then the second half of the antecedent is 117 // latchStart + guardLimit - 1 - guardStart u< latchLimit 118 // and its negation is 119 // latchStart + guardLimit - 1 - guardStart u>= latchLimit 120 // 121 // In other words, if 122 // latchLimit u<= latchStart + guardLimit - 1 - guardStart 123 // then: 124 // (the ranges below are written in ConstantRange notation, where [A, B) is the 125 // set for (I = A; I != B; I++ /*maywrap*/) yield(I);) 126 // 127 // forall X . guardStart + X u< guardLimit && 128 // latchStart + X u< latchLimit => 129 // guardStart + X + 1 u< guardLimit 130 // == forall X . guardStart + X u< guardLimit && 131 // latchStart + X u< latchStart + guardLimit - 1 - guardStart => 132 // guardStart + X + 1 u< guardLimit 133 // == forall X . (guardStart + X) in [0, guardLimit) && 134 // (latchStart + X) in [0, latchStart + guardLimit - 1 - guardStart) => 135 // (guardStart + X + 1) in [0, guardLimit) 136 // == forall X . X in [-guardStart, guardLimit - guardStart) && 137 // X in [-latchStart, guardLimit - 1 - guardStart) => 138 // X in [-guardStart - 1, guardLimit - guardStart - 1) 139 // == true 140 // 141 // So the widened condition is: 142 // guardStart u< guardLimit && 143 // latchStart + guardLimit - 1 - guardStart u>= latchLimit 144 // Similarly for ule condition the widened condition is: 145 // guardStart u< guardLimit && 146 // latchStart + guardLimit - 1 - guardStart u> latchLimit 147 // For slt condition the widened condition is: 148 // guardStart u< guardLimit && 149 // latchStart + guardLimit - 1 - guardStart s>= latchLimit 150 // For sle condition the widened condition is: 151 // guardStart u< guardLimit && 152 // latchStart + guardLimit - 1 - guardStart s> latchLimit 153 // 154 // When S = -1 (i.e. reverse iterating loop), the transformation is supported 155 // when: 156 // * The loop has a single latch with the condition of the form: 157 // B(X) = X latchLimit, where is u> or s>. 158 // * The guard condition is of the form 159 // G(X) = X - 1 u< guardLimit 160 // 161 // For the ugt latch comparison case M is: 162 // forall X. X-1 u< guardLimit and X u> latchLimit => X-2 u< guardLimit 163 // 164 // The only way the antecedent can be true and the consequent can be false is if 165 // X == 1. 166 // If X == 1 then the second half of the antecedent is 167 // 1 u> latchLimit, and its negation is latchLimit u>= 1. 168 // 169 // So the widened condition is: 170 // guardStart u< guardLimit && latchLimit u>= 1. 171 // Similarly for sgt condition the widened condition is: 172 // guardStart u< guardLimit && latchLimit s>= 1. 154 173 //===----------------------------------------------------------------------===// 155 174 156 175 #include "llvm/Transforms/Scalar/LoopPredication.h" 176 195 static cl::opt EnableIVTruncation("loop-predication-enable-iv-truncation", 177 196 cl::Hidden, cl::init(true)); 178 197 198 static cl::opt EnableCountDownLoop("loop-predication-enable-count-down-loop", 199 cl::Hidden, cl::init(true)); 179 200 namespace { 180 201 class LoopPredication { 181 202 /// Represents an induction variable check: 222 243 LoopICmp RangeCheck, 223 244 SCEVExpander &Expander, 224 245 IRBuilder<> &Builder); 225 ⏎ 246 Optional widenICmpRangeCheckDecrementingLoop(LoopICmp LatchCheck,⏎ 247 LoopICmp RangeCheck, 248 SCEVExpander &Expander, 249 IRBuilder<> &Builder); 226 250 bool widenGuardConditions(IntrinsicInst *II, SCEVExpander &Expander); 227 251 228 252 // When the IV type is wider than the range operand type, we can still do loop 359 383 } 360 384 361 385 bool LoopPredication::isSupportedStep(const SCEV* Step) { 362 return Step->isOne();⏎ 386 return Step->isOne() || (Step->isAllOnesValue() && EnableCountDownLoop);⏎ 363 387 } 364 388 365 389 bool LoopPredication::CanExpand(const SCEV* S) { 419 443 GuardStart, GuardLimit, InsertAt); 420 444 return Builder.CreateAnd(FirstIterationCheck, LimitCheck); 421 445 } 446 447 Optional LoopPredication::widenICmpRangeCheckDecrementingLoop( 448 LoopPredication::LoopICmp LatchCheck, LoopPredication::LoopICmp RangeCheck, 449 SCEVExpander &Expander, IRBuilder<> &Builder) { 450 auto *Ty = RangeCheck.IV->getType(); 451 const SCEV *GuardStart = RangeCheck.IV->getStart(); 452 const SCEV *GuardLimit = RangeCheck.Limit; 453 const SCEV *LatchLimit = LatchCheck.Limit; 454 if (!CanExpand(GuardStart) || !CanExpand(GuardLimit) || 455 !CanExpand(LatchLimit)) { 456 DEBUG(dbgs() << "Can't expand limit check!\n"); 457 return None; 458 } 459 // The decrement of the latch check IV should be the same as the 460 // rangeCheckIV. 461 auto *PostDecLatchCheckIV = LatchCheck.IV->getPostIncExpr(*SE); 462 if (RangeCheck.IV != PostDecLatchCheckIV) { 463 DEBUG(dbgs() << "Not the same. PostDecLatchCheckIV: " 464 << *PostDecLatchCheckIV 465 << " and RangeCheckIV: " << *RangeCheck.IV << "\n"); 466 return None; 467 } 468 469 // Generate the widened condition for CountDownLoop: 470 // guardStart u< guardLimit && 471 // latchLimit 1. 472 // See the header comment for reasoning of the checks. 473 Instruction *InsertAt = Preheader->getTerminator(); 474 auto LimitCheckPred = ICmpInst::isSigned(LatchCheck.Pred) 475 ? ICmpInst::ICMP_SGE 476 : ICmpInst::ICMP_UGE; 477 auto *FirstIterationCheck = expandCheck(Expander, Builder, ICmpInst::ICMP_ULT, 478 GuardStart, GuardLimit, InsertAt); 479 auto *LimitCheck = expandCheck(Expander, Builder, LimitCheckPred, LatchLimit, 480 SE->getOne(Ty), InsertAt); 481 return Builder.CreateAnd(FirstIterationCheck, LimitCheck); 482 } 483 422 484 /// If ICI can be widened to a loop invariant condition emits the loop 423 485 /// invariant condition in the loop preheader and return it, otherwise 424 486 /// returns None. 466 528 } 467 529 468 530 LoopICmp CurrLatchCheck = *CurrLatchCheckOpt; 469 // At this point the range check step and latch step should have the same 470 // value and type. 471 assert(Step == CurrLatchCheck.IV->getStepRecurrence(*SE) && 472 "Range and latch should have same step recurrence!"); 473 474 return widenICmpRangeCheckIncrementingLoop(CurrLatchCheck, *RangeCheck, 475 Expander, Builder);⏎ 531 // At this point, the range and latch step should have the same type, but need⏎ 532 // not have the same value (we support both 1 and -1 steps). 533 assert(Step->getType() == 534 CurrLatchCheck.IV->getStepRecurrence(*SE)->getType() && 535 "Range and latch steps should be of same type!"); 536 if (Step != CurrLatchCheck.IV->getStepRecurrence(*SE)) { 537 DEBUG(dbgs() << "Range and latch have different step values!\n"); 538 return None; 539 } 540 541 if (Step->isOne()) 542 return widenICmpRangeCheckIncrementingLoop(CurrLatchCheck, *RangeCheck, 543 Expander, Builder); 544 else { 545 assert(Step->isAllOnesValue() && "Step should be -1!"); 546 return widenICmpRangeCheckDecrementingLoop(CurrLatchCheck, *RangeCheck, 547 Expander, Builder); 548 } 476 549 } 477 550 478 551 bool LoopPredication::widenGuardConditions(IntrinsicInst *Guard, 579 652 } 580 653 581 654 auto IsUnsupportedPredicate = [](const SCEV *Step, ICmpInst::Predicate Pred) { 582 assert(Step->isOne() && "expected Step to be one!"); 583 return Pred != ICmpInst::ICMP_ULT && Pred != ICmpInst::ICMP_SLT && 584 Pred != ICmpInst::ICMP_ULE && Pred != ICmpInst::ICMP_SLE;⏎ 655 if (Step->isOne()) {⏎ 656 return Pred != ICmpInst::ICMP_ULT && Pred != ICmpInst::ICMP_SLT && 657 Pred != ICmpInst::ICMP_ULE && Pred != ICmpInst::ICMP_SLE; 658 } else { 659 assert(Step->isAllOnesValue() && "Step should be -1!"); 660 return Pred != ICmpInst::ICMP_UGT && Pred != ICmpInst::ICMP_SGT; 661 } 585 662 }; 586 663 587 664 if (IsUnsupportedPredicate(Step, Result->Pred)) {