llvm.org GIT mirror
BranchProb: modify the definition of an edge in BranchProbabilityInfo to handle the case of multiple edges from one block to another. A simple example is a switch statement with multiple values to the same destination. The definition of an edge is modified from a pair of blocks to a pair of PredBlock and an index into the successors. Also set the weight correctly when building SelectionDAG from LLVM IR, especially when converting a Switch. IntegersSubsetMapping is updated to calculate the weight for each cluster. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@162572 91177308-0d34-0410-b5e6-96231b3b80d8 Manman Ren 7 years ago
8 changed file(s) with 250 addition(s) and 113 deletion(s).
 27 27 /// 28 28 /// This is a function analysis pass which provides information on the relative 29 29 /// probabilities of each "edge" in the function's CFG where such an edge is 30 /// defined by a pair of basic blocks. The probability for a given block and 31 /// a successor block are always relative to the probabilities of the other 32 /// successor blocks. Another way of looking at it is that the probabilities 33 /// for a given block B and each of its successors should sum to exactly 34 /// one (100%).⏎ 30 /// defined by a pair (PredBlock and an index in the successors). The⏎ 31 /// probability of an edge from one block is always relative to the 32 /// probabilities of other edges from the block. The probabilites of all edges 33 /// from a block sum to exactly one (100%). 34 /// We use a pair (PredBlock and an index in the successors) to uniquely 35 /// identify an edge, since we can have multiple edges from Src to Dst. 36 /// As an example, we can have a switch which jumps to Dst with value 0 and 37 /// value 10. 35 38 class BranchProbabilityInfo : public FunctionPass { 36 39 public: 37 40 static char ID; 50 53 /// (0%) and one (100%) of this edge executing, relative to other edges 51 54 /// leaving the 'Src' block. The returned probability is never zero, and can 52 55 /// only be one if the source block has only one successor. 56 BranchProbability getEdgeProbability(const BasicBlock *Src, 57 unsigned IndexInSuccessors) const; 58 59 /// \brief Get the probability of going from Src to Dst. 60 /// 61 /// It returns the sum of all probabilities for edges from Src to Dst. 53 62 BranchProbability getEdgeProbability(const BasicBlock *Src, 54 63 const BasicBlock *Dst) const; 55 64 73 82 raw_ostream &printEdgeProbability(raw_ostream &OS, const BasicBlock *Src, 74 83 const BasicBlock *Dst) const; 75 84 76 /// \brief Get the raw edge weight calculated for the block pair.⏎ 85 /// \brief Get the raw edge weight calculated for the edge.⏎ 77 86 /// 78 87 /// This returns the raw edge weight. It is guaranteed to fall between 1 and 79 88 /// UINT32_MAX. Note that the raw edge weight is not meaningful in isolation. 80 89 /// This interface should be very carefully, and primarily by routines that 81 90 /// are updating the analysis by later calling setEdgeWeight. 91 uint32_t getEdgeWeight(const BasicBlock *Src, 92 unsigned IndexInSuccessors) const; 93 94 /// \brief Get the raw edge weight calculated for the block pair. 95 /// 96 /// This returns the sum of all raw edge weights from Src to Dst. 97 /// It is guaranteed to fall between 1 and UINT32_MAX. 82 98 uint32_t getEdgeWeight(const BasicBlock *Src, const BasicBlock *Dst) const; 83 99 84 /// \brief Set the raw edge weight for the block pair.⏎ 100 /// \brief Set the raw edge weight for a given edge.⏎ 85 101 /// 86 /// This allows a pass to explicitly set the edge weight for a block. It can⏎ 102 /// This allows a pass to explicitly set the edge weight for an edge. It can⏎ 87 103 /// be used when updating the CFG to update and preserve the branch 88 104 /// probability information. Read the implementation of how these edge 89 105 /// weights are calculated carefully before using! 90 void setEdgeWeight(const BasicBlock *Src, const BasicBlock *Dst,⏎ 106 void setEdgeWeight(const BasicBlock *Src, unsigned IndexInSuccessors,⏎ 91 107 uint32_t Weight); 92 108 93 109 private: 94 typedef std::pair Edge;⏎ 110 // Since we allow duplicate edges from one basic block to another, we use⏎ 111 // a pair (PredBlock and an index in the successors) to specify an edge. 112 typedef std::pair Edge; 95 113 96 114 // Default weight value. Used when we don't have information about the edge. 97 115 // TODO: DEFAULT_WEIGHT makes sense during static predication, when none of
 41 41 struct RangeEx : public RangeTy { 42 42 RangeEx() : Weight(1) {} 43 43 RangeEx(const RangeTy &R) : RangeTy(R), Weight(1) {} 44 RangeEx(const RangeTy &R, unsigned W) : RangeTy(R), Weight(W) {} 44 45 RangeEx(const IntTy &C) : RangeTy(C), Weight(1) {} 45 46 RangeEx(const IntTy &L, const IntTy &H) : RangeTy(L, H), Weight(1) {} 46 47 RangeEx(const IntTy &L, const IntTy &H, unsigned W) : 315 316 Items.clear(); 316 317 const IntTy *Low = &OldItems.begin()->first.getLow(); 317 318 const IntTy *High = &OldItems.begin()->first.getHigh(); 318 unsigned Weight = 1;⏎ 319 unsigned Weight = OldItems.begin()->first.Weight;⏎ 319 320 SuccessorClass *Successor = OldItems.begin()->second; 320 321 for (CaseItemIt j = OldItems.begin(), i = j++, e = OldItems.end(); 321 322 j != e; i = j++) { 322 323 if (isJoinable(i, j)) { 323 324 const IntTy *CurHigh = &j->first.getHigh(); 324 ++Weight;⏎ 325 Weight += j->first.Weight;⏎ 325 326 if (*CurHigh > *High) 326 327 High = CurHigh; 327 328 } else { 329 330 add(R, Successor); 330 331 Low = &j->first.getLow(); 331 332 High = &j->first.getHigh(); 332 Weight = 1;⏎ 333 Weight = j->first.Weight;⏎ 333 334 Successor = j->second; 334 335 } 335 336 } 361 362 362 363 /// Adds all ranges and values from given ranges set to the current 363 364 /// mapping. 364 void add(const IntegersSubsetTy &CRS, SuccessorClass *S = 0) {⏎ 365 void add(const IntegersSubsetTy &CRS, SuccessorClass *S = 0,⏎ 366 unsigned Weight = 0) { 367 unsigned ItemWeight = 1; 368 if (Weight) 369 // Weight is associated with CRS, for now we perform a division to 370 // get the weight for each item. 371 ItemWeight = Weight / CRS.getNumItems(); 365 372 for (unsigned i = 0, e = CRS.getNumItems(); i < e; ++i) { 366 373 RangeTy R = CRS.getItem(i); 367 add(R, S);⏎ 374 RangeEx REx(R, ItemWeight);⏎ 375 add(REx, S); 368 376 } 369 377 } 370 378
 114 114 return false; 115 115 } 116 116 117 SmallPtrSet UnreachableEdges; 118 SmallPtrSet ReachableEdges;⏎ 117 SmallVector UnreachableEdges;⏎ 118 SmallVector ReachableEdges; 119 119 120 120 for (succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) { 121 121 if (PostDominatedByUnreachable.count(*I)) 122 UnreachableEdges.insert(*I);⏎ 122 UnreachableEdges.push_back(I.getSuccessorIndex());⏎ 123 123 else 124 ReachableEdges.insert(*I);⏎ 124 ReachableEdges.push_back(I.getSuccessorIndex());⏎ 125 125 } 126 126 127 127 // If all successors are in the set of blocks post-dominated by unreachable, 135 135 return false; 136 136 137 137 uint32_t UnreachableWeight = 138 std::max(UR_TAKEN_WEIGHT / UnreachableEdges.size(), MIN_WEIGHT); 139 for (SmallPtrSet::iterator I = UnreachableEdges.begin(), 140 E = UnreachableEdges.end();⏎ 138 std::max(UR_TAKEN_WEIGHT / (unsigned)UnreachableEdges.size(), MIN_WEIGHT);⏎ 139 for (SmallVector::iterator I = UnreachableEdges.begin(), 140 E = UnreachableEdges.end(); 141 141 I != E; ++I) 142 142 setEdgeWeight(BB, *I, UnreachableWeight); 143 143 144 144 if (ReachableEdges.empty()) 145 145 return true; 146 146 uint32_t ReachableWeight = 147 std::max(UR_NONTAKEN_WEIGHT / ReachableEdges.size(), NORMAL_WEIGHT); 148 for (SmallPtrSet::iterator I = ReachableEdges.begin(), 149 E = ReachableEdges.end();⏎ 147 std::max(UR_NONTAKEN_WEIGHT / (unsigned)ReachableEdges.size(),⏎ 148 NORMAL_WEIGHT); 149 for (SmallVector::iterator I = ReachableEdges.begin(), 150 E = ReachableEdges.end(); 150 151 I != E; ++I) 151 152 setEdgeWeight(BB, *I, ReachableWeight); 152 153 186 187 } 187 188 assert(Weights.size() == TI->getNumSuccessors() && "Checked above"); 188 189 for (unsigned i = 0, e = TI->getNumSuccessors(); i != e; ++i) 189 setEdgeWeight(BB, TI->getSuccessor(i), Weights[i]);⏎ 190 setEdgeWeight(BB, i, Weights[i]);⏎ 190 191 191 192 return true; 192 193 } 210 211 211 212 assert(CI->getOperand(1)->getType()->isPointerTy()); 212 213 213 BasicBlock *Taken = BI->getSuccessor(0); 214 BasicBlock *NonTaken = BI->getSuccessor(1); 215 216 214 // p != 0 -> isProb = true 217 215 // p == 0 -> isProb = false 218 216 // p != q -> isProb = true 219 217 // p == q -> isProb = false; 218 unsigned TakenIdx = 0, NonTakenIdx = 1; 220 219 bool isProb = CI->getPredicate() == ICmpInst::ICMP_NE; 221 220 if (!isProb) 222 std::swap(Taken, NonTaken); 223 224 setEdgeWeight(BB, Taken, PH_TAKEN_WEIGHT); 225 setEdgeWeight(BB, NonTaken, PH_NONTAKEN_WEIGHT);⏎ 221 std::swap(TakenIdx, NonTakenIdx);⏎ 222 223 setEdgeWeight(BB, TakenIdx, PH_TAKEN_WEIGHT); 224 setEdgeWeight(BB, NonTakenIdx, PH_NONTAKEN_WEIGHT); 226 225 return true; 227 226 } 228 227 233 232 if (!L) 234 233 return false; 235 234 236 SmallPtrSet BackEdges; 237 SmallPtrSet ExitingEdges; 238 SmallPtrSet InEdges; // Edges from header to the loop.⏎ 235 SmallVector BackEdges;⏎ 236 SmallVector ExitingEdges; 237 SmallVector InEdges; // Edges from header to the loop. 239 238 240 239 for (succ_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) { 241 240 if (!L->contains(*I)) 242 ExitingEdges.insert(*I);⏎ 241 ExitingEdges.push_back(I.getSuccessorIndex());⏎ 243 242 else if (L->getHeader() == *I) 244 BackEdges.insert(*I);⏎ 243 BackEdges.push_back(I.getSuccessorIndex());⏎ 245 244 else 246 InEdges.insert(*I);⏎ 245 InEdges.push_back(I.getSuccessorIndex());⏎ 247 246 } 248 247 249 248 if (uint32_t numBackEdges = BackEdges.size()) { 251 250 if (backWeight < NORMAL_WEIGHT) 252 251 backWeight = NORMAL_WEIGHT; 253 252 254 for (SmallPtrSet, 8>::iterator EI = BackEdges.begin(),⏎ 253 for (SmallVector, 8>::iterator EI = BackEdges.begin(),⏎ 255 254 EE = BackEdges.end(); EI != EE; ++EI) { 256 BasicBlock *Back = *EI; 257 setEdgeWeight(BB, Back, backWeight);⏎ 255 setEdgeWeight(BB, *EI, backWeight);⏎ 258 256 } 259 257 } 260 258 263 261 if (inWeight < NORMAL_WEIGHT) 264 262 inWeight = NORMAL_WEIGHT; 265 263 266 for (SmallPtrSet, 8>::iterator EI = InEdges.begin(),⏎ 264 for (SmallVector, 8>::iterator EI = InEdges.begin(),⏎ 267 265 EE = InEdges.end(); EI != EE; ++EI) { 268 BasicBlock *Back = *EI; 269 setEdgeWeight(BB, Back, inWeight);⏎ 266 setEdgeWeight(BB, *EI, inWeight);⏎ 270 267 } 271 268 } 272 269 275 272 if (exitWeight < MIN_WEIGHT) 276 273 exitWeight = MIN_WEIGHT; 277 274 278 for (SmallPtrSet, 8>::iterator EI = ExitingEdges.begin(),⏎ 275 for (SmallVector, 8>::iterator EI = ExitingEdges.begin(),⏎ 279 276 EE = ExitingEdges.end(); EI != EE; ++EI) { 280 BasicBlock *Exiting = *EI; 281 setEdgeWeight(BB, Exiting, exitWeight);⏎ 277 setEdgeWeight(BB, *EI, exitWeight);⏎ 282 278 } 283 279 } 284 280 334 330 return false; 335 331 } 336 332 337 BasicBlock *Taken = BI->getSuccessor(0); 338 BasicBlock *NonTaken = BI->getSuccessor(1);⏎ 333 unsigned TakenIdx = 0, NonTakenIdx = 1;⏎ 339 334 340 335 if (!isProb) 341 std::swap(Taken, NonTaken); 342 343 setEdgeWeight(BB, Taken, ZH_TAKEN_WEIGHT); 344 setEdgeWeight(BB, NonTaken, ZH_NONTAKEN_WEIGHT);⏎ 336 std::swap(TakenIdx, NonTakenIdx);⏎ 337 338 setEdgeWeight(BB, TakenIdx, ZH_TAKEN_WEIGHT); 339 setEdgeWeight(BB, NonTakenIdx, ZH_NONTAKEN_WEIGHT); 345 340 346 341 return true; 347 342 } 371 366 return false; 372 367 } 373 368 374 BasicBlock *Taken = BI->getSuccessor(0); 375 BasicBlock *NonTaken = BI->getSuccessor(1);⏎ 369 unsigned TakenIdx = 0, NonTakenIdx = 1;⏎ 376 370 377 371 if (!isProb) 378 std::swap(Taken, NonTaken); 379 380 setEdgeWeight(BB, Taken, FPH_TAKEN_WEIGHT); 381 setEdgeWeight(BB, NonTaken, FPH_NONTAKEN_WEIGHT);⏎ 372 std::swap(TakenIdx, NonTakenIdx);⏎ 373 374 setEdgeWeight(BB, TakenIdx, FPH_TAKEN_WEIGHT); 375 setEdgeWeight(BB, NonTakenIdx, FPH_NONTAKEN_WEIGHT); 382 376 383 377 return true; 384 378 } 388 382 if (!II) 389 383 return false; 390 384 391 BasicBlock *Normal = II->getNormalDest(); 392 BasicBlock *Unwind = II->getUnwindDest(); 393 394 setEdgeWeight(BB, Normal, IH_TAKEN_WEIGHT); 395 setEdgeWeight(BB, Unwind, IH_NONTAKEN_WEIGHT);⏎ 385 setEdgeWeight(BB, 0/*Index for Normal*/, IH_TAKEN_WEIGHT);⏎ 386 setEdgeWeight(BB, 1/*Index for Unwind*/, IH_NONTAKEN_WEIGHT); 396 387 return true; 397 388 } 398 389 449 440 uint32_t Sum = 0; 450 441 451 442 for (succ_const_iterator I = succ_begin(BB), E = succ_end(BB); I != E; ++I) { 452 const BasicBlock *Succ = *I; 453 uint32_t Weight = getEdgeWeight(BB, Succ);⏎ 443 uint32_t Weight = getEdgeWeight(BB, I.getSuccessorIndex());⏎ 454 444 uint32_t PrevSum = Sum; 455 445 456 446 Sum += Weight; 493 483 return 0; 494 484 } 495 485 496 // Return edge's weight. If can't find it, return DEFAULT_WEIGHT value.⏎ 486 /// Get the raw edge weight for the edge. If can't find it, return⏎ 487 /// DEFAULT_WEIGHT value. Here an edge is specified using PredBlock and an index 488 /// to the successors. 489 uint32_t BranchProbabilityInfo:: 490 getEdgeWeight(const BasicBlock *Src, unsigned IndexInSuccessors) const { 491 DenseMap::const_iterator I = 492 Weights.find(std::make_pair(Src, IndexInSuccessors)); 493 494 if (I != Weights.end()) 495 return I->second; 496 497 return DEFAULT_WEIGHT; 498 } 499 500 /// Get the raw edge weight calculated for the block pair. This returns the sum 501 /// of all raw edge weights from Src to Dst. 497 502 uint32_t BranchProbabilityInfo:: 498 503 getEdgeWeight(const BasicBlock *Src, const BasicBlock *Dst) const { 499 Edge E(Src, Dst); 500 DenseMap::const_iterator I = Weights.find(E); 501 502 if (I != Weights.end()) 503 return I->second; 504 505 return DEFAULT_WEIGHT; 506 } 507 ⏎ 504 uint32_t Weight = 0;⏎ 505 DenseMap::const_iterator MapI; 506 for (succ_const_iterator I = succ_begin(Src), E = succ_end(Src); I != E; ++I) 507 if (*I == Dst) { 508 MapI = Weights.find(std::make_pair(Src, I.getSuccessorIndex())); 509 if (MapI != Weights.end()) 510 Weight += MapI->second; 511 } 512 return (Weight == 0) ? DEFAULT_WEIGHT : Weight; 513 } 514 515 /// Set the edge weight for a given edge specified by PredBlock and an index 516 /// to the successors. 508 517 void BranchProbabilityInfo:: 509 setEdgeWeight(const BasicBlock *Src, const BasicBlock *Dst, uint32_t Weight) { 510 Weights[std::make_pair(Src, Dst)] = Weight;⏎ 518 setEdgeWeight(const BasicBlock *Src, unsigned IndexInSuccessors,⏎ 519 uint32_t Weight) { 520 Weights[std::make_pair(Src, IndexInSuccessors)] = Weight; 511 521 DEBUG(dbgs() << "set edge " << Src->getName() << " -> " 512 << Dst->getName() << " weight to " << Weight 513 << (isEdgeHot(Src, Dst) ? " [is HOT now]\n" : "\n")); 514 } 515 516 ⏎ 522 << IndexInSuccessors << " successor weight to "⏎ 523 << Weight << "\n"); 524 } 525 526 /// Get an edge's probability, relative to other out-edges from Src. 527 BranchProbability BranchProbabilityInfo:: 528 getEdgeProbability(const BasicBlock *Src, unsigned IndexInSuccessors) const { 529 uint32_t N = getEdgeWeight(Src, IndexInSuccessors); 530 uint32_t D = getSumForBlock(Src); 531 532 return BranchProbability(N, D); 533 } 534 535 /// Get the probability of going from Src to Dst. It returns the sum of all 536 /// probabilities for edges from Src to Dst. 517 537 BranchProbability BranchProbabilityInfo:: 518 538 getEdgeProbability(const BasicBlock *Src, const BasicBlock *Dst) const { 519 539