llvm.org GIT mirror llvm / ff03048
LangRef and basic memory-representation/reading/writing for 'cmpxchg' and 'atomicrmw' instructions, which allow representing all the current atomic rmw intrinsics. The allowed operands for these instructions are heavily restricted at the moment; we can probably loosen it a bit, but supporting general first-class types (where it makes sense) might get a bit complicated, given how SelectionDAG works. As an initial cut, these operations do not support specifying an alignment, but it would be possible to add if we think it's useful. Specifying an alignment lower than the natural alignment would be essentially impossible to support on anything other than x86, but specifying a greater alignment would be possible. I can't think of any useful optimizations which would use that information, but maybe someone else has ideas. Optimizer/codegen support coming soon. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@136404 91177308-0d34-0410-b5e6-96231b3b80d8 Eli Friedman 9 years ago
18 changed file(s) with 946 addition(s) and 43 deletion(s). Raw diff Collapse all Expand all
5353
  • Pointer Aliasing Rules
  • 5454
  • Volatile Memory Accesses
  • 5555
  • Memory Model for Concurrent Operations
  • 56
  • Atomic Memory Ordering Constraints
  • 5657
    5758
    5859
  • Type System
  • 167168
    168169
  • Memory Access and Addressing Operations
  • 169170
    170
  • 'alloca' Instruction
  • 171
  • 'load' Instruction
  • 172
  • 'store' Instruction
  • 173
  • 'fence' Instruction
  • 171
  • 'alloca' Instruction
  • 172
  • 'load' Instruction
  • 173
  • 'store' Instruction
  • 174
  • 'fence' Instruction
  • 175
  • 'cmpxchg' Instruction
  • 176
  • 'atomicrmw' Instruction
  • 174177
  • 'getelementptr' Instruction
  • 175178
    176179
    14991502
  • When a synchronizes-with b, includes an edge from
  • 15001503 a to b. Synchronizes-with pairs are introduced
    15011504 by platform-specific techniques, like pthread locks, thread
    1502 creation, thread joining, etc., and by the atomic operations described
    1503 in the Atomic intrinsics section.
    1505 creation, thread joining, etc., and by atomic instructions.
    1506 (See also Atomic Memory Ordering Constraints).
    1507
    15041508
    15051509
    15061510

    Note that program order does not introduce happens-before edges

    15351539 write.
    15361540
  • Otherwise, if R is atomic, and all the writes
  • 15371541 Rbyte may see are atomic, it chooses one of the
    1538 values written. See the Atomic intrinsics
    1539 section for additional guarantees on how the choice is made.
    1542 values written. See the Atomic Memory Ordering
    1543 Constraints section for additional constraints on how the choice
    1544 is made.
    15401545
  • Otherwise Rbyte returns undef.
  • 15411546
    15421547
    15651570 non-atomic write with an alignment or width less than the minimum width
    15661571 as if it writes to the relevant surrounding bytes.
    15671572 -->
    1573
    1574
    1575
    1576
    1577
    1578 Atomic Memory Ordering Constraints
    1579
    1580
    1581
    1582
    1583

    Atomic instructions (cmpxchg,

    1584 atomicrmw, and
    1585 fence) take an ordering parameter
    1586 that determines which other atomic instructions on the same address they
    1587 synchronize with. These semantics are borrowed from Java and C++0x,
    1588 but are somewhat more colloquial. If these descriptions aren't precise enough,
    1589 check those specs. fence instructions
    1590 treat these orderings somewhat differently since they don't take an address.
    1591 See that instruction's documentation for details.

    1592
    1593
    1594
    1595
    1596
    1598
    unordered
    1599
    The set of values that can be read is governed by the happens-before
    1600 partial order. A value cannot be read unless some operation wrote it.
    1601 This is intended to provide a guarantee strong enough to model Java's
    1602 non-volatile shared variables. This ordering cannot be specified for
    1603 read-modify-write operations; it is not strong enough to make them atomic
    1604 in any interesting way.
    1605
    monotonic
    1606
    In addition to the guarantees of unordered, there is a single
    1607 total order for modifications by monotonic operations on each
    1608 address. All modification orders must be compatible with the happens-before
    1609 order. There is no guarantee that the modification orders can be combined to
    1610 a global total order for the whole program (and this often will not be
    1611 possible). The read in an atomic read-modify-write operation
    1612 (cmpxchg and
    1613 atomicrmw)
    1614 reads the value in the modification order immediately before the value it
    1615 writes. If one atomic read happens before another atomic read of the same
    1616 address, the later read must see the same value or a later value in the
    1617 address's modification order. This disallows reordering of
    1618 monotonic (or stronger) operations on the same address. If an
    1619 address is written monotonically by one thread, and other threads
    1620 monotonically read that address repeatedly, the other threads must
    1621 eventually see the write. This is intended to model C++'s relaxed atomic
    1622 variables.
    1623
    acquire
    1624
    In addition to the guarantees of monotonic, if this operation
    1625 reads a value written by a release atomic operation, it
    1626 synchronizes-with that operation.
    1627
    release
    1628
    In addition to the guarantees of monotonic,
    1629 a synchronizes-with edge may be formed by an acquire
    1630 operation.
    1631
    acq_rel (acquire+release)
    Acts as both an
    1632 acquire and release operation on its address.
    1633
    seq_cst (sequentially consistent)
    1634
    In addition to the guarantees of acq_rel
    1635 (acquire for an operation which only reads, release
    1636 for an operation which only writes), there is a global total order on all
    1637 sequentially-consistent operations on all addresses, which is consistent with
    1638 the happens-before partial order and with the modification orders of
    1639 all the affected addresses. Each sequentially-consistent read sees the last
    1640 preceding write to the same address in this global order. This is intended
    1641 to model C++'s sequentially-consistent atomic variables and Java's volatile
    1642 shared variables.
    1643
    1644
    1645

    If an atomic operation is marked singlethread,

    1646 it only synchronizes with or participates in modification and seq_cst
    1647 total orderings with other operations running in the same thread (for example,
    1648 in signal handlers).

    15681649
    15691650
    15701651
    46414722
    46424723
    46434724
    4725
    4726 Instruction
    4727
    4728
    4729
    4730
    Syntax:
    4731
    
                      
                    
    4732 [volatile] cmpxchg <ty>* <pointer>, <ty> <cmp>, <ty> <new> [singlethread] <ordering> ; yields {ty}
    4733
    4734
    4735
    Overview:
    4736

    The 'cmpxchg' instruction is used to atomically modify memory.

    4737 It loads a value in memory and compares it to a given value. If they are
    4738 equal, it stores a new value into the memory.

    4739
    4740
    Arguments:
    4741

    There are three arguments to the 'cmpxchg' instruction: an

    4742 address to operate on, a value to compare to the value currently be at that
    4743 address, and a new value to place at that address if the compared values are
    4744 equal. The type of '<cmp>' must be an integer type whose
    4745 bit width is a power of two greater than or equal to eight and less than
    4746 or equal to a target-specific size limit. '<cmp>' and
    4747 '<new>' must have the same type, and the type of
    4748 '<pointer>' must be a pointer to that type. If the
    4749 cmpxchg is marked as volatile, then the
    4750 optimizer is not allowed to modify the number or order of execution
    4751 of this cmpxchg with other volatile
    4752 operations.

    4753
    4754
    4755
    4756

    The ordering argument specifies how this

    4757 cmpxchg synchronizes with other atomic operations.

    4758
    4759

    The optional "singlethread" argument declares that the

    4760 cmpxchg is only atomic with respect to code (usually signal
    4761 handlers) running in the same thread as the cmpxchg. Otherwise the
    4762 cmpxchg is atomic with respect to all other code in the system.

    4763
    4764

    The pointer passed into cmpxchg must have alignment greater than or equal to

    4765 the size in memory of the operand.
    4766
    4767
    Semantics:
    4768

    The contents of memory at the location specified by the

    4769 '<pointer>' operand is read and compared to
    4770 '<cmp>'; if the read value is the equal,
    4771 '<new>' is written. The original value at the location
    4772 is returned.
    4773
    4774

    A successful cmpxchg is a read-modify-write instruction for the

    4775 purpose of identifying release sequences. A
    4776 failed cmpxchg is equivalent to an atomic load with an ordering
    4777 parameter determined by dropping any release part of the
    4778 cmpxchg's ordering.

    4779
    4780
    4786
    4787
    Example:
    4788
    
                      
                    
    4789 entry:
    4790 %orig = atomic load i32* %ptr unordered ; yields {i32}
    4791 br label %loop
    4792
    4793 loop:
    4794 %cmp = phi i32 [ %orig, %entry ], [%old, %loop]
    4795 %squared = mul i32 %cmp, %cmp
    4796 %old = cmpxchg i32* %ptr, i32 %cmp, i32 %squared ; yields {i32}
    4797 %success = icmp eq i32 %cmp, %old
    4798 br i1 %success, label %done, label %loop
    4799
    4800 done:
    4801 ...
    4802
    4803
    4804
    4805
    4806
    4807
    4808 Instruction
    4809
    4810
    4811
    4812
    Syntax:
    4813
    
                      
                    
    4814 [volatile] atomicrmw <operation> <ty>* <pointer>, <ty> <value> [singlethread] <ordering> ; yields {ty}
    4815
    4816
    4817
    Overview:
    4818

    The 'atomicrmw' instruction is used to atomically modify memory.

    4819
    4820
    Arguments:
    4821

    There are three arguments to the 'atomicrmw' instruction: an

    4822 operation to apply, an address whose value to modify, an argument to the
    4823 operation. The operation must be one of the following keywords:

    4824
    4825
  • xchg
  • 4826
  • add
  • 4827
  • sub
  • 4828
  • and
  • 4829
  • nand
  • 4830
  • or
  • 4831
  • xor
  • 4832
  • max
  • 4833
  • min
  • 4834
  • umax
  • 4835
  • umin
  • 4836
    4837
    4838

    The type of '<value>' must be an integer type whose

    4839 bit width is a power of two greater than or equal to eight and less than
    4840 or equal to a target-specific size limit. The type of the
    4841 '<pointer>' operand must be a pointer to that type.
    4842 If the atomicrmw is marked as volatile, then the
    4843 optimizer is not allowed to modify the number or order of execution of this
    4844 atomicrmw with other volatile
    4845 operations.

    4846
    4847
    4848
    4849
    Semantics:
    4850

    The contents of memory at the location specified by the

    4851 '<pointer>' operand are atomically read, modified, and written
    4852 back. The original value at the location is returned. The modification is
    4853 specified by the operation argument:

    4854
    4855
    4856
  • xchg: *ptr = val
  • 4857
  • add: *ptr = *ptr + val
  • 4858
  • sub: *ptr = *ptr - val
  • 4859
  • and: *ptr = *ptr & val
  • 4860
  • nand: *ptr = ~(*ptr & val)
  • 4861
  • or: *ptr = *ptr | val
  • 4862
  • xor: *ptr = *ptr ^ val
  • 4863
  • max: *ptr = *ptr > val ? *ptr : val (using a signed comparison)
  • 4864
  • min: *ptr = *ptr < val ? *ptr : val (using a signed comparison)
  • 4865
  • umax: *ptr = *ptr > val ? *ptr : val (using an unsigned comparison)
  • 4866
  • umin: *ptr = *ptr < val ? *ptr : val (using an unsigned comparison)
  • 4867
    4868
    4869
    Example:
    4870
    
                      
                    
    4871 %old = atomicrmw add i32* %ptr, i32 1 acquire ; yields {i32}
    4872
    4873
    4874
    4875
    4876
    46444877

    46454878 'getelementptr' Instruction
    46464879
    204204 BINOP_XOR = 12
    205205 };
    206206
    207 /// These are values used in the bitcode files to encode AtomicRMW operations.
    208 /// The values of these enums have no fixed relation to the LLVM IR enum
    209 /// values. Changing these will break compatibility with old files.
    210 enum RMWOperations {
    211 RMW_XCHG = 0,
    212 RMW_ADD = 1,
    213 RMW_SUB = 2,
    214 RMW_AND = 3,
    215 RMW_NAND = 4,
    216 RMW_OR = 5,
    217 RMW_XOR = 6,
    218 RMW_MAX = 7,
    219 RMW_MIN = 8,
    220 RMW_UMAX = 9,
    221 RMW_UMIN = 10
    222 };
    223
    207224 /// OverflowingBinaryOperatorOptionalFlags - Flags for serializing
    208225 /// OverflowingBinaryOperator's SubclassOptionalData contents.
    209226 enum OverflowingBinaryOperatorOptionalFlags {
    284301
    285302 FUNC_CODE_DEBUG_LOC = 35, // DEBUG_LOC: [Line,Col,ScopeVal, IAVal]
    286303 FUNC_CODE_INST_FENCE = 36, // FENCE: [ordering, synchscope]
    287 FUNC_CODE_INST_LANDINGPAD = 37 // LANDINGPAD: [ty,val,val,num,id0,val0...]
    304 FUNC_CODE_INST_LANDINGPAD = 37, // LANDINGPAD: [ty,val,val,num,id0,val0...]
    305 FUNC_CODE_INST_CMPXCHG = 38, // CMPXCHG: [ptrty,ptr,cmp,new, align, vol,
    306 // ordering, synchscope]
    307 FUNC_CODE_INST_ATOMICRMW = 39 // ATOMICRMW: [ptrty,ptr,val, operation,
    308 // align, vol,
    309 // ordering, synchscope]
    310
    288311 };
    289312 } // End bitc namespace
    290313 } // End llvm namespace
    134134 HANDLE_MEMORY_INST(29, Store , StoreInst )
    135135 HANDLE_MEMORY_INST(30, GetElementPtr, GetElementPtrInst)
    136136 HANDLE_MEMORY_INST(31, Fence , FenceInst )
    137 LAST_MEMORY_INST(31)
    137 HANDLE_MEMORY_INST(32, AtomicCmpXchg , AtomicCmpXchgInst )
    138 HANDLE_MEMORY_INST(33, AtomicRMW , AtomicRMWInst )
    139 LAST_MEMORY_INST(33)
    138140
    139141 // Cast operators ...
    140142 // NOTE: The order matters here because CastInst::isEliminableCastPair
    141143 // NOTE: (see Instructions.cpp) encodes a table based on this ordering.
    142 FIRST_CAST_INST(33)
    143 HANDLE_CAST_INST(33, Trunc , TruncInst ) // Truncate integers
    144 HANDLE_CAST_INST(34, ZExt , ZExtInst ) // Zero extend integers
    145 HANDLE_CAST_INST(35, SExt , SExtInst ) // Sign extend integers
    146 HANDLE_CAST_INST(36, FPToUI , FPToUIInst ) // floating point -> UInt
    147 HANDLE_CAST_INST(37, FPToSI , FPToSIInst ) // floating point -> SInt
    148 HANDLE_CAST_INST(38, UIToFP , UIToFPInst ) // UInt -> floating point
    149 HANDLE_CAST_INST(39, SIToFP , SIToFPInst ) // SInt -> floating point
    150 HANDLE_CAST_INST(40, FPTrunc , FPTruncInst ) // Truncate floating point
    151 HANDLE_CAST_INST(41, FPExt , FPExtInst ) // Extend floating point
    152 HANDLE_CAST_INST(42, PtrToInt, PtrToIntInst) // Pointer -> Integer
    153 HANDLE_CAST_INST(43, IntToPtr, IntToPtrInst) // Integer -> Pointer
    154 HANDLE_CAST_INST(44, BitCast , BitCastInst ) // Type cast
    155 LAST_CAST_INST(44)
    144 FIRST_CAST_INST(34)
    145 HANDLE_CAST_INST(34, Trunc , TruncInst ) // Truncate integers
    146 HANDLE_CAST_INST(35, ZExt , ZExtInst ) // Zero extend integers
    147 HANDLE_CAST_INST(36, SExt , SExtInst ) // Sign extend integers
    148 HANDLE_CAST_INST(37, FPToUI , FPToUIInst ) // floating point -> UInt
    149 HANDLE_CAST_INST(38, FPToSI , FPToSIInst ) // floating point -> SInt
    150 HANDLE_CAST_INST(39, UIToFP , UIToFPInst ) // UInt -> floating point
    151 HANDLE_CAST_INST(40, SIToFP , SIToFPInst ) // SInt -> floating point
    152 HANDLE_CAST_INST(41, FPTrunc , FPTruncInst ) // Truncate floating point
    153 HANDLE_CAST_INST(42, FPExt , FPExtInst ) // Extend floating point
    154 HANDLE_CAST_INST(43, PtrToInt, PtrToIntInst) // Pointer -> Integer
    155 HANDLE_CAST_INST(44, IntToPtr, IntToPtrInst) // Integer -> Pointer
    156 HANDLE_CAST_INST(45, BitCast , BitCastInst ) // Type cast
    157 LAST_CAST_INST(45)
    156158
    157159 // Other operators...
    158 FIRST_OTHER_INST(45)
    159 HANDLE_OTHER_INST(45, ICmp , ICmpInst ) // Integer comparison instruction
    160 HANDLE_OTHER_INST(46, FCmp , FCmpInst ) // Floating point comparison instr.
    161 HANDLE_OTHER_INST(47, PHI , PHINode ) // PHI node instruction
    162 HANDLE_OTHER_INST(48, Call , CallInst ) // Call a function
    163 HANDLE_OTHER_INST(49, Select , SelectInst ) // select instruction
    164 HANDLE_OTHER_INST(50, UserOp1, Instruction) // May be used internally in a pass
    165 HANDLE_OTHER_INST(51, UserOp2, Instruction) // Internal to passes only
    166 HANDLE_OTHER_INST(52, VAArg , VAArgInst ) // vaarg instruction
    167 HANDLE_OTHER_INST(53, ExtractElement, ExtractElementInst)// extract from vector
    168 HANDLE_OTHER_INST(54, InsertElement, InsertElementInst) // insert into vector
    169 HANDLE_OTHER_INST(55, ShuffleVector, ShuffleVectorInst) // shuffle two vectors.
    170 HANDLE_OTHER_INST(56, ExtractValue, ExtractValueInst)// extract from aggregate
    171 HANDLE_OTHER_INST(57, InsertValue, InsertValueInst) // insert into aggregate
    172 HANDLE_OTHER_INST(58, LandingPad, LandingPadInst) // Landing pad instruction.
    173 LAST_OTHER_INST(58)
    160 FIRST_OTHER_INST(46)
    161 HANDLE_OTHER_INST(46, ICmp , ICmpInst ) // Integer comparison instruction
    162 HANDLE_OTHER_INST(47, FCmp , FCmpInst ) // Floating point comparison instr.
    163 HANDLE_OTHER_INST(48, PHI , PHINode ) // PHI node instruction
    164 HANDLE_OTHER_INST(49, Call , CallInst ) // Call a function
    165 HANDLE_OTHER_INST(50, Select , SelectInst ) // select instruction
    166 HANDLE_OTHER_INST(51, UserOp1, Instruction) // May be used internally in a pass
    167 HANDLE_OTHER_INST(52, UserOp2, Instruction) // Internal to passes only
    168 HANDLE_OTHER_INST(53, VAArg , VAArgInst ) // vaarg instruction
    169 HANDLE_OTHER_INST(54, ExtractElement, ExtractElementInst)// extract from vector
    170 HANDLE_OTHER_INST(55, InsertElement, InsertElementInst) // insert into vector
    171 HANDLE_OTHER_INST(56, ShuffleVector, ShuffleVectorInst) // shuffle two vectors.
    172 HANDLE_OTHER_INST(57, ExtractValue, ExtractValueInst)// extract from aggregate
    173 HANDLE_OTHER_INST(58, InsertValue, InsertValueInst) // insert into aggregate
    174 HANDLE_OTHER_INST(59, LandingPad, LandingPadInst) // Landing pad instruction.
    175 LAST_OTHER_INST(59)
    174176
    175177 #undef FIRST_TERM_INST
    176178 #undef HANDLE_TERM_INST
    359359 Instruction::setInstructionSubclassData(D);
    360360 }
    361361 };
    362
    363 //===----------------------------------------------------------------------===//
    364 // AtomicCmpXchgInst Class
    365 //===----------------------------------------------------------------------===//
    366
    367 /// AtomicCmpXchgInst - an instruction that atomically checks whether a
    368 /// specified value is in a memory location, and, if it is, stores a new value
    369 /// there. Returns the value that was loaded.
    370 ///
    371 class AtomicCmpXchgInst : public Instruction {
    372 void *operator new(size_t, unsigned); // DO NOT IMPLEMENT
    373 void Init(Value *Ptr, Value *Cmp, Value *NewVal,
    374 AtomicOrdering Ordering, SynchronizationScope SynchScope);
    375 protected:
    376 virtual AtomicCmpXchgInst *clone_impl() const;
    377 public:
    378 // allocate space for exactly three operands
    379 void *operator new(size_t s) {
    380 return User::operator new(s, 3);
    381 }
    382 AtomicCmpXchgInst(Value *Ptr, Value *Cmp, Value *NewVal,
    383 AtomicOrdering Ordering, SynchronizationScope SynchScope,
    384 Instruction *InsertBefore = 0);
    385 AtomicCmpXchgInst(Value *Ptr, Value *Cmp, Value *NewVal,
    386 AtomicOrdering Ordering, SynchronizationScope SynchScope,
    387 BasicBlock *InsertAtEnd);
    388
    389 /// isVolatile - Return true if this is a cmpxchg from a volatile memory
    390 /// location.
    391 ///
    392 bool isVolatile() const {
    393 return getSubclassDataFromInstruction() & 1;
    394 }
    395
    396 /// setVolatile - Specify whether this is a volatile cmpxchg.
    397 ///
    398 void setVolatile(bool V) {
    399 setInstructionSubclassData((getSubclassDataFromInstruction() & ~1) |
    400 (unsigned)V);
    401 }
    402
    403 /// Transparently provide more efficient getOperand methods.
    404 DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
    405
    406 /// Set the ordering constraint on this cmpxchg.
    407 void setOrdering(AtomicOrdering Ordering) {
    408 assert(Ordering != NotAtomic &&
    409 "CmpXchg instructions can only be atomic.");
    410 setInstructionSubclassData((getSubclassDataFromInstruction() & 3) |
    411 (Ordering << 2));
    412 }
    413
    414 /// Specify whether this cmpxchg is atomic and orders other operations with
    415 /// respect to all concurrently executing threads, or only with respect to
    416 /// signal handlers executing in the same thread.
    417 void setSynchScope(SynchronizationScope SynchScope) {
    418 setInstructionSubclassData((getSubclassDataFromInstruction() & ~2) |
    419 (SynchScope << 1));
    420 }
    421
    422 /// Returns the ordering constraint on this cmpxchg.
    423 AtomicOrdering getOrdering() const {
    424 return AtomicOrdering(getSubclassDataFromInstruction() >> 2);
    425 }
    426
    427 /// Returns whether this cmpxchg is atomic between threads or only within a
    428 /// single thread.
    429 SynchronizationScope getSynchScope() const {
    430 return SynchronizationScope((getSubclassDataFromInstruction() & 2) >> 1);
    431 }
    432
    433 Value *getPointerOperand() { return getOperand(0); }
    434 const Value *getPointerOperand() const { return getOperand(0); }
    435 static unsigned getPointerOperandIndex() { return 0U; }
    436
    437 Value *getCompareOperand() { return getOperand(1); }
    438 const Value *getCompareOperand() const { return getOperand(1); }
    439
    440 Value *getNewValOperand() { return getOperand(2); }
    441 const Value *getNewValOperand() const { return getOperand(2); }
    442
    443 unsigned getPointerAddressSpace() const {
    444 return cast(getPointerOperand()->getType())->getAddressSpace();
    445 }
    446
    447 // Methods for support type inquiry through isa, cast, and dyn_cast:
    448 static inline bool classof(const AtomicCmpXchgInst *) { return true; }
    449 static inline bool classof(const Instruction *I) {
    450 return I->getOpcode() == Instruction::AtomicCmpXchg;
    451 }
    452 static inline bool classof(const Value *V) {
    453 return isa(V) && classof(cast(V));
    454 }
    455 private:
    456 // Shadow Instruction::setInstructionSubclassData with a private forwarding
    457 // method so that subclasses cannot accidentally use it.
    458 void setInstructionSubclassData(unsigned short D) {
    459 Instruction::setInstructionSubclassData(D);
    460 }
    461 };
    462
    463 template <>
    464 struct OperandTraits :
    465 public FixedNumOperandTraits {
    466 };
    467
    468 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(AtomicCmpXchgInst, Value)
    469
    470 //===----------------------------------------------------------------------===//
    471 // AtomicRMWInst Class
    472 //===----------------------------------------------------------------------===//
    473
    474 /// AtomicRMWInst - an instruction that atomically reads a memory location,
    475 /// combines it with another value, and then stores the result back. Returns
    476 /// the old value.
    477 ///
    478 class AtomicRMWInst : public Instruction {
    479 void *operator new(size_t, unsigned); // DO NOT IMPLEMENT
    480 protected:
    481 virtual AtomicRMWInst *clone_impl() const;
    482 public:
    483 /// This enumeration lists the possible modifications atomicrmw can make. In
    484 /// the descriptions, 'p' is the pointer to the instruction's memory location,
    485 /// 'old' is the initial value of *p, and 'v' is the other value passed to the
    486 /// instruction. These instructions always return 'old'.
    487 enum BinOp {
    488 /// *p = v
    489 Xchg,
    490 /// *p = old + v
    491 Add,
    492 /// *p = old - v
    493 Sub,
    494 /// *p = old & v
    495 And,
    496 /// *p = ~old & v
    497 Nand,
    498 /// *p = old | v
    499 Or,
    500 /// *p = old ^ v
    501 Xor,
    502 /// *p = old >signed v ? old : v
    503 Max,
    504 /// *p = old
    505 Min,
    506 /// *p = old >unsigned v ? old : v
    507 UMax,
    508 /// *p = old
    509 UMin,
    510
    511 FIRST_BINOP = Xchg,
    512 LAST_BINOP = UMin,
    513 BAD_BINOP
    514 };
    515
    516 // allocate space for exactly two operands
    517 void *operator new(size_t s) {
    518 return User::operator new(s, 2);
    519 }
    520 AtomicRMWInst(BinOp Operation, Value *Ptr, Value *Val,
    521 AtomicOrdering Ordering, SynchronizationScope SynchScope,
    522 Instruction *InsertBefore = 0);
    523 AtomicRMWInst(BinOp Operation, Value *Ptr, Value *Val,
    524 AtomicOrdering Ordering, SynchronizationScope SynchScope,
    525 BasicBlock *InsertAtEnd);
    526
    527 BinOp getOperation() const {
    528 return static_cast(getSubclassDataFromInstruction() >> 5);
    529 }
    530
    531 void setOperation(BinOp Operation) {
    532 unsigned short SubclassData = getSubclassDataFromInstruction();
    533 setInstructionSubclassData((SubclassData & 31) |
    534 (Operation << 5));
    535 }
    536
    537 /// isVolatile - Return true if this is a RMW on a volatile memory location.
    538 ///
    539 bool isVolatile() const {
    540 return getSubclassDataFromInstruction() & 1;
    541 }
    542
    543 /// setVolatile - Specify whether this is a volatile RMW or not.
    544 ///
    545 void setVolatile(bool V) {
    546 setInstructionSubclassData((getSubclassDataFromInstruction() & ~1) |
    547 (unsigned)V);
    548 }
    549
    550 /// Transparently provide more efficient getOperand methods.
    551 DECLARE_TRANSPARENT_OPERAND_ACCESSORS(Value);
    552
    553 /// Set the ordering constraint on this RMW.
    554 void setOrdering(AtomicOrdering Ordering) {
    555 assert(Ordering != NotAtomic &&
    556 "atomicrmw instructions can only be atomic.");
    557 setInstructionSubclassData((getSubclassDataFromInstruction() & ~28) |
    558 (Ordering << 2));
    559 }
    560
    561 /// Specify whether this RMW orders other operations with respect to all
    562 /// concurrently executing threads, or only with respect to signal handlers
    563 /// executing in the same thread.
    564 void setSynchScope(SynchronizationScope SynchScope) {
    565 setInstructionSubclassData((getSubclassDataFromInstruction() & ~2) |
    566 (SynchScope << 1));
    567 }
    568
    569 /// Returns the ordering constraint on this RMW.
    570 AtomicOrdering getOrdering() const {
    571 return AtomicOrdering((getSubclassDataFromInstruction() & 28) >> 2);
    572 }
    573
    574 /// Returns whether this RMW is atomic between threads or only within a
    575 /// single thread.
    576 SynchronizationScope getSynchScope() const {
    577 return SynchronizationScope((getSubclassDataFromInstruction() & 2) >> 1);
    578 }
    579
    580 Value *getPointerOperand() { return getOperand(0); }
    581 const Value *getPointerOperand() const { return getOperand(0); }
    582 static unsigned getPointerOperandIndex() { return 0U; }
    583
    584 Value *getValOperand() { return getOperand(1); }
    585 const Value *getValOperand() const { return getOperand(1); }
    586
    587 unsigned getPointerAddressSpace() const {
    588 return cast(getPointerOperand()->getType())->getAddressSpace();
    589 }
    590
    591 // Methods for support type inquiry through isa, cast, and dyn_cast:
    592 static inline bool classof(const AtomicRMWInst *) { return true; }
    593 static inline bool classof(const Instruction *I) {
    594 return I->getOpcode() == Instruction::AtomicRMW;
    595 }
    596 static inline bool classof(const Value *V) {
    597 return isa(V) && classof(cast(V));
    598 }
    599 private:
    600 void Init(BinOp Operation, Value *Ptr, Value *Val,
    601 AtomicOrdering Ordering, SynchronizationScope SynchScope);
    602 // Shadow Instruction::setInstructionSubclassData with a private forwarding
    603 // method so that subclasses cannot accidentally use it.
    604 void setInstructionSubclassData(unsigned short D) {
    605 Instruction::setInstructionSubclassData(D);
    606 }
    607 };
    608
    609 template <>
    610 struct OperandTraits
    611 : public FixedNumOperandTraits {
    612 };
    613
    614 DEFINE_TRANSPARENT_OPERAND_ACCESSORS(AtomicRMWInst, Value)
    362615
    363616 //===----------------------------------------------------------------------===//
    364617 // GetElementPtrInst Class
    169169 RetTy visitAllocaInst(AllocaInst &I) { DELEGATE(Instruction); }
    170170 RetTy visitLoadInst(LoadInst &I) { DELEGATE(Instruction); }
    171171 RetTy visitStoreInst(StoreInst &I) { DELEGATE(Instruction); }
    172 RetTy visitAtomicCmpXchgInst(AtomicCmpXchgInst &I){ DELEGATE(Instruction); }
    173 RetTy visitAtomicRMWInst(AtomicRMWInst &I) { DELEGATE(Instruction); }
    172174 RetTy visitFenceInst(FenceInst &I) { DELEGATE(Instruction); }
    173175 RetTy visitGetElementPtrInst(GetElementPtrInst &I){ DELEGATE(Instruction); }
    174176 RetTy visitPHINode(PHINode &I) { DELEGATE(Instruction); }
    186186
    187187 /* Atomic operators */
    188188 LLVMFence = 55,
    189 LLVMAtomicCmpXchg = 56,
    190 LLVMAtomicRMW = 57,
    189191
    190192 /* Exception Handling Operators */
    191 LLVMLandingPad = 56,
    192 LLVMResume = 57
    193 LLVMLandingPad = 58,
    194 LLVMResume = 59
    193195
    194196 } LLVMOpcode;
    195197
    578578 KEYWORD(oeq); KEYWORD(one); KEYWORD(olt); KEYWORD(ogt); KEYWORD(ole);
    579579 KEYWORD(oge); KEYWORD(ord); KEYWORD(uno); KEYWORD(ueq); KEYWORD(une);
    580580
    581 KEYWORD(xchg); KEYWORD(nand); KEYWORD(max); KEYWORD(min); KEYWORD(umax);
    582 KEYWORD(umin);
    583
    581584 KEYWORD(x);
    582585 KEYWORD(blockaddress);
    583586
    644647 INSTKEYWORD(alloca, Alloca);
    645648 INSTKEYWORD(load, Load);
    646649 INSTKEYWORD(store, Store);
    650 INSTKEYWORD(cmpxchg, AtomicCmpXchg);
    651 INSTKEYWORD(atomicrmw, AtomicRMW);
    647652 INSTKEYWORD(fence, Fence);
    648653 INSTKEYWORD(getelementptr, GetElementPtr);
    649654
    29512951 case lltok::kw_alloca: return ParseAlloc(Inst, PFS);
    29522952 case lltok::kw_load: return ParseLoad(Inst, PFS, false);
    29532953 case lltok::kw_store: return ParseStore(Inst, PFS, false);
    2954 case lltok::kw_cmpxchg: return ParseCmpXchg(Inst, PFS, false);
    2955 case lltok::kw_atomicrmw: return ParseAtomicRMW(Inst, PFS, false);
    29542956 case lltok::kw_fence: return ParseFence(Inst, PFS);
    29552957 case lltok::kw_volatile:
    29562958 if (EatIfPresent(lltok::kw_load))
    29572959 return ParseLoad(Inst, PFS, true);
    29582960 else if (EatIfPresent(lltok::kw_store))
    29592961 return ParseStore(Inst, PFS, true);
    2962 else if (EatIfPresent(lltok::kw_cmpxchg))
    2963 return ParseCmpXchg(Inst, PFS, true);
    2964 else if (EatIfPresent(lltok::kw_atomicrmw))
    2965 return ParseAtomicRMW(Inst, PFS, true);
    29602966 else
    29612967 return TokError("expected 'load' or 'store'");
    29622968 case lltok::kw_getelementptr: return ParseGetElementPtr(Inst, PFS);
    37243730 return AteExtraComma ? InstExtraComma : InstNormal;
    37253731 }
    37263732
    3733 /// ParseCmpXchg
    3734 /// ::= 'volatile'? 'cmpxchg' TypeAndValue ',' TypeAndValue ',' TypeAndValue
    3735 /// 'singlethread'? AtomicOrdering
    3736 int LLParser::ParseCmpXchg(Instruction *&Inst, PerFunctionState &PFS,
    3737 bool isVolatile) {
    3738 Value *Ptr, *Cmp, *New; LocTy PtrLoc, CmpLoc, NewLoc;
    3739 bool AteExtraComma = false;
    3740 AtomicOrdering Ordering = NotAtomic;
    3741 SynchronizationScope Scope = CrossThread;
    3742 if (ParseTypeAndValue(Ptr, PtrLoc, PFS) ||
    3743 ParseToken(lltok::comma, "expected ',' after cmpxchg address") ||
    3744 ParseTypeAndValue(Cmp, CmpLoc, PFS) ||
    3745 ParseToken(lltok::comma, "expected ',' after cmpxchg cmp operand") ||
    3746 ParseTypeAndValue(New, NewLoc, PFS) ||
    3747 ParseScopeAndOrdering(true /*Always atomic*/, Scope, Ordering))
    3748 return true;
    3749
    3750 if (Ordering == Unordered)
    3751 return TokError("cmpxchg cannot be unordered");
    3752 if (!Ptr->getType()->isPointerTy())
    3753 return Error(PtrLoc, "cmpxchg operand must be a pointer");
    3754 if (cast(Ptr->getType())->getElementType() != Cmp->getType())
    3755 return Error(CmpLoc, "compare value and pointer type do not match");
    3756 if (cast(Ptr->getType())->getElementType() != New->getType())
    3757 return Error(NewLoc, "new value and pointer type do not match");
    3758 if (!New->getType()->isIntegerTy())
    3759 return Error(NewLoc, "cmpxchg operand must be an integer");
    3760 unsigned Size = New->getType()->getPrimitiveSizeInBits();
    3761 if (Size < 8 || (Size & (Size - 1)))
    3762 return Error(NewLoc, "cmpxchg operand must be power-of-two byte-sized"
    3763 " integer");
    3764
    3765 AtomicCmpXchgInst *CXI =
    3766 new AtomicCmpXchgInst(Ptr, Cmp, New, Ordering, Scope);
    3767 CXI->setVolatile(isVolatile);
    3768 Inst = CXI;
    3769 return AteExtraComma ? InstExtraComma : InstNormal;
    3770 }
    3771
    3772 /// ParseAtomicRMW
    3773 /// ::= 'volatile'? 'atomicrmw' BinOp TypeAndValue ',' TypeAndValue
    3774 /// 'singlethread'? AtomicOrdering
    3775 int LLParser::ParseAtomicRMW(Instruction *&Inst, PerFunctionState &PFS,
    3776 bool isVolatile) {
    3777 Value *Ptr, *Val; LocTy PtrLoc, ValLoc;
    3778 bool AteExtraComma = false;
    3779 AtomicOrdering Ordering = NotAtomic;
    3780 SynchronizationScope Scope = CrossThread;
    3781 AtomicRMWInst::BinOp Operation;
    3782 switch (Lex.getKind()) {
    3783 default: return TokError("expected binary operation in atomicrmw");
    3784 case lltok::kw_xchg: Operation = AtomicRMWInst::Xchg; break;
    3785 case lltok::kw_add: Operation = AtomicRMWInst::Add; break;
    3786 case lltok::kw_sub: Operation = AtomicRMWInst::Sub; break;
    3787 case lltok::kw_and: Operation = AtomicRMWInst::And; break;
    3788 case lltok::kw_nand: Operation = AtomicRMWInst::Nand; break;
    3789 case lltok::kw_or: Operation = AtomicRMWInst::Or; break;
    3790 case lltok::kw_xor: Operation = AtomicRMWInst::Xor; break;
    3791 case lltok::kw_max: Operation = AtomicRMWInst::Max; break;
    3792 case lltok::kw_min: Operation = AtomicRMWInst::Min; break;
    3793 case lltok::kw_umax: Operation = AtomicRMWInst::UMax; break;
    3794 case lltok::kw_umin: Operation = AtomicRMWInst::UMin; break;
    3795 }
    3796 Lex.Lex(); // Eat the operation.
    3797
    3798 if (ParseTypeAndValue(Ptr, PtrLoc, PFS) ||
    3799 ParseToken(lltok::comma, "expected ',' after atomicrmw address") ||
    3800 ParseTypeAndValue(Val, ValLoc, PFS) ||
    3801 ParseScopeAndOrdering(true /*Always atomic*/, Scope, Ordering))
    3802 return true;
    3803
    3804 if (Ordering == Unordered)
    3805 return TokError("atomicrmw cannot be unordered");
    3806 if (!Ptr->getType()->isPointerTy())
    3807 return Error(PtrLoc, "atomicrmw operand must be a pointer");
    3808 if (cast(Ptr->getType())->getElementType() != Val->getType())
    3809 return Error(ValLoc, "atomicrmw value and pointer type do not match");
    3810 if (!Val->getType()->isIntegerTy())
    3811 return Error(ValLoc, "atomicrmw operand must be an integer");
    3812 unsigned Size = Val->getType()->getPrimitiveSizeInBits();
    3813 if (Size < 8 || (Size & (Size - 1)))
    3814 return Error(ValLoc, "atomicrmw operand must be power-of-two byte-sized"
    3815 " integer");
    3816
    3817 AtomicRMWInst *RMWI =
    3818 new AtomicRMWInst(Operation, Ptr, Val, Ordering, Scope);
    3819 RMWI->setVolatile(isVolatile);
    3820 Inst = RMWI;
    3821 return AteExtraComma ? InstExtraComma : InstNormal;
    3822 }
    3823
    37273824 /// ParseFence
    37283825 /// ::= 'fence' 'singlethread'? AtomicOrdering
    37293826 int LLParser::ParseFence(Instruction *&Inst, PerFunctionState &PFS) {
    364364 int ParseAlloc(Instruction *&I, PerFunctionState &PFS);
    365365 int ParseLoad(Instruction *&I, PerFunctionState &PFS, bool isVolatile);
    366366 int ParseStore(Instruction *&I, PerFunctionState &PFS, bool isVolatile);
    367 int ParseCmpXchg(Instruction *&I, PerFunctionState &PFS, bool isVolatile);
    368 int ParseAtomicRMW(Instruction *&I, PerFunctionState &PFS, bool isVolatile);
    367369 int ParseFence(Instruction *&I, PerFunctionState &PFS);
    368370 int ParseGetElementPtr(Instruction *&I, PerFunctionState &PFS);
    369371 int ParseExtractValue(Instruction *&I, PerFunctionState &PFS);
    109109 kw_uge, kw_oeq, kw_one, kw_olt, kw_ogt, kw_ole, kw_oge, kw_ord, kw_uno,
    110110 kw_ueq, kw_une,
    111111
    112 // atomicrmw operations that aren't also instruction keywords.
    113 kw_xchg, kw_nand, kw_max, kw_min, kw_umax, kw_umin,
    114
    112115 // Instruction Opcodes (Opcode in UIntVal).
    113116 kw_add, kw_fadd, kw_sub, kw_fsub, kw_mul, kw_fmul,
    114117 kw_udiv, kw_sdiv, kw_fdiv,
    125128 kw_ret, kw_br, kw_switch, kw_indirectbr, kw_invoke, kw_unwind, kw_resume,
    126129 kw_unreachable,
    127130
    128 kw_alloca, kw_load, kw_store, kw_fence, kw_getelementptr,
    131 kw_alloca, kw_load, kw_store, kw_fence, kw_cmpxchg, kw_atomicrmw,
    132 kw_getelementptr,
    129133
    130134 kw_extractelement, kw_insertelement, kw_shufflevector,
    131135 kw_extractvalue, kw_insertvalue, kw_blockaddress,
    127127 case bitc::BINOP_AND: return Instruction::And;
    128128 case bitc::BINOP_OR: return Instruction::Or;
    129129 case bitc::BINOP_XOR: return Instruction::Xor;
    130 }
    131 }
    132
    133 static AtomicRMWInst::BinOp GetDecodedRMWOperation(unsigned Val) {
    134 switch (Val) {
    135 default: return AtomicRMWInst::BAD_BINOP;
    136 case bitc::RMW_XCHG: return AtomicRMWInst::Xchg;
    137 case bitc::RMW_ADD: return AtomicRMWInst::Add;
    138 case bitc::RMW_SUB: return AtomicRMWInst::Sub;
    139 case bitc::RMW_AND: return AtomicRMWInst::And;
    140 case bitc::RMW_NAND: return AtomicRMWInst::Nand;
    141 case bitc::RMW_OR: return AtomicRMWInst::Or;
    142 case bitc::RMW_XOR: return AtomicRMWInst::Xor;
    143 case bitc::RMW_MAX: return AtomicRMWInst::Max;
    144 case bitc::RMW_MIN: return AtomicRMWInst::Min;
    145 case bitc::RMW_UMAX: return AtomicRMWInst::UMax;
    146 case bitc::RMW_UMIN: return AtomicRMWInst::UMin;
    130147 }
    131148 }
    132149
    25942611 InstructionList.push_back(I);
    25952612 break;
    25962613 }
    2614 case bitc::FUNC_CODE_INST_CMPXCHG: {
    2615 // CMPXCHG:[ptrty, ptr, cmp, new, vol, ordering, synchscope]
    2616 unsigned OpNum = 0;
    2617 Value *Ptr, *Cmp, *New;
    2618 if (getValueTypePair(Record, OpNum, NextValueNo, Ptr) ||
    2619 getValue(Record, OpNum,
    2620 cast(Ptr->getType())->getElementType(), Cmp) ||
    2621 getValue(Record, OpNum,
    2622 cast(Ptr->getType())->getElementType(), New) ||
    2623 OpNum+3 != Record.size())
    2624 return Error("Invalid CMPXCHG record");
    2625 AtomicOrdering Ordering = GetDecodedOrdering(Record[OpNum+1]);
    2626 if (Ordering == NotAtomic)
    2627 return Error("Invalid CMPXCHG record");
    2628 SynchronizationScope SynchScope = GetDecodedSynchScope(Record[OpNum+2]);
    2629 I = new AtomicCmpXchgInst(Ptr, Cmp, New, Ordering, SynchScope);
    2630 cast(I)->setVolatile(Record[OpNum]);
    2631 InstructionList.push_back(I);
    2632 break;
    2633 }
    2634 case bitc::FUNC_CODE_INST_ATOMICRMW: {
    2635 // ATOMICRMW:[ptrty, ptr, val, op, vol, ordering, synchscope]
    2636 unsigned OpNum = 0;
    2637 Value *Ptr, *Val;
    2638 if (getValueTypePair(Record, OpNum, NextValueNo, Ptr) ||
    2639 getValue(Record, OpNum,
    2640 cast(Ptr->getType())->getElementType(), Val) ||
    2641 OpNum+4 != Record.size())
    2642 return Error("Invalid ATOMICRMW record");
    2643 AtomicRMWInst::BinOp Operation = GetDecodedRMWOperation(Record[OpNum]);
    2644 if (Operation < AtomicRMWInst::FIRST_BINOP ||
    2645 Operation > AtomicRMWInst::LAST_BINOP)
    2646 return Error("Invalid ATOMICRMW record");
    2647 AtomicOrdering Ordering = GetDecodedOrdering(Record[OpNum+2]);
    2648 if (Ordering == NotAtomic)
    2649 return Error("Invalid ATOMICRMW record");
    2650 SynchronizationScope SynchScope = GetDecodedSynchScope(Record[OpNum+3]);
    2651 I = new AtomicRMWInst(Operation, Ptr, Val, Ordering, SynchScope);
    2652 cast(I)->setVolatile(Record[OpNum+1]);
    2653 InstructionList.push_back(I);
    2654 break;
    2655 }
    25972656 case bitc::FUNC_CODE_INST_FENCE: { // FENCE:[ordering, synchscope]
    25982657 if (2 != Record.size())
    25992658 return Error("Invalid FENCE record");
    9797 case Instruction::And: return bitc::BINOP_AND;
    9898 case Instruction::Or: return bitc::BINOP_OR;
    9999 case Instruction::Xor: return bitc::BINOP_XOR;
    100 }
    101 }
    102
    103 static unsigned GetEncodedRMWOperation(AtomicRMWInst::BinOp Op) {
    104 switch (Op) {
    105 default: llvm_unreachable("Unknown RMW operation!");
    106 case AtomicRMWInst::Xchg: return bitc::RMW_XCHG;
    107 case AtomicRMWInst::Add: return bitc::RMW_ADD;
    108 case AtomicRMWInst::Sub: return bitc::RMW_SUB;
    109 case AtomicRMWInst::And: return bitc::RMW_AND;
    110 case AtomicRMWInst::Nand: return bitc::RMW_NAND;
    111 case AtomicRMWInst::Or: return bitc::RMW_OR;
    112 case AtomicRMWInst::Xor: return bitc::RMW_XOR;
    113 case AtomicRMWInst::Max: return bitc::RMW_MAX;
    114 case AtomicRMWInst::Min: return bitc::RMW_MIN;
    115 case AtomicRMWInst::UMax: return bitc::RMW_UMAX;
    116 case AtomicRMWInst::UMin: return bitc::RMW_UMIN;
    100117 }
    101118 }
    102119
    11851202 Vals.push_back(Log2_32(cast(I).getAlignment())+1);
    11861203 Vals.push_back(cast(I).isVolatile());
    11871204 break;
    1205 case Instruction::AtomicCmpXchg:
    1206 Code = bitc::FUNC_CODE_INST_CMPXCHG;
    1207 PushValueAndType(I.getOperand(0), InstID, Vals, VE); // ptrty + ptr
    1208 Vals.push_back(VE.getValueID(I.getOperand(1))); // cmp.
    1209 Vals.push_back(VE.getValueID(I.getOperand(2))); // newval.
    1210 Vals.push_back(cast(I).isVolatile());
    1211 Vals.push_back(GetEncodedOrdering(
    1212 cast(I).getOrdering()));
    1213 Vals.push_back(GetEncodedSynchScope(
    1214 cast(I).getSynchScope()));
    1215 break;
    1216 case Instruction::AtomicRMW:
    1217 Code = bitc::FUNC_CODE_INST_ATOMICRMW;
    1218 PushValueAndType(I.getOperand(0), InstID, Vals, VE); // ptrty + ptr
    1219 Vals.push_back(VE.getValueID(I.getOperand(1))); // val.
    1220 Vals.push_back(GetEncodedRMWOperation(
    1221 cast(I).getOperation()));
    1222 Vals.push_back(cast(I).isVolatile());
    1223 Vals.push_back(GetEncodedOrdering(cast(I).getOrdering()));
    1224 Vals.push_back(GetEncodedSynchScope(
    1225 cast(I).getSynchScope()));
    1226 break;
    11881227 case Instruction::Fence:
    11891228 Code = bitc::FUNC_CODE_INST_FENCE;
    11901229 Vals.push_back(GetEncodedOrdering(cast(I).getOrdering()));
    32193219 ++SDNodeOrder;
    32203220 AssignOrderingToNode(StoreNode.getNode());
    32213221 DAG.setRoot(StoreNode);
    3222 }
    3223
    3224 void SelectionDAGBuilder::visitAtomicCmpXchg(const AtomicCmpXchgInst &I) {
    3225 }
    3226
    3227 void SelectionDAGBuilder::visitAtomicRMW(const AtomicRMWInst &I) {
    32223228 }
    32233229
    32243230 void SelectionDAGBuilder::visitFence(const FenceInst &I) {
    505505 void visitAlloca(const AllocaInst &I);
    506506 void visitLoad(const LoadInst &I);
    507507 void visitStore(const StoreInst &I);
    508 void visitAtomicCmpXchg(const AtomicCmpXchgInst &I);
    509 void visitAtomicRMW(const AtomicRMWInst &I);
    508510 void visitFence(const FenceInst &I);
    509511 void visitPHI(const PHINode &I);
    510512 void visitCall(const CallInst &I);
    657657 return pred;
    658658 }
    659659
    660 static void writeAtomicRMWOperation(raw_ostream &Out,
    661 AtomicRMWInst::BinOp Op) {
    662 switch (Op) {
    663 default: Out << " "; break;
    664 case AtomicRMWInst::Xchg: Out << " xchg"; break;
    665 case AtomicRMWInst::Add: Out << " add"; break;
    666 case AtomicRMWInst::Sub: Out << " sub"; break;
    667 case AtomicRMWInst::And: Out << " and"; break;
    668 case AtomicRMWInst::Nand: Out << " nand"; break;
    669 case AtomicRMWInst::Or: Out << " or"; break;
    670 case AtomicRMWInst::Xor: Out << " xor"; break;
    671 case AtomicRMWInst::Max: Out << " max"; break;
    672 case AtomicRMWInst::Min: Out << " min"; break;
    673 case AtomicRMWInst::UMax: Out << " umax"; break;
    674 case AtomicRMWInst::UMin: Out << " umin"; break;
    675 }
    676 }
    660677
    661678 static void WriteOptimizationInfo(raw_ostream &Out, const User *U) {
    662679 if (const OverflowingBinaryOperator *OBO =
    16691686 if (const CmpInst *CI = dyn_cast(&I))
    16701687 Out << ' ' << getPredicateText(CI->getPredicate());
    16711688
    1689 // Print out the atomicrmw operation
    1690 if (const AtomicRMWInst *RMWI = dyn_cast(&I))
    1691 writeAtomicRMWOperation(Out, RMWI->getOperation());
    1692
    16721693 // Print out the type of the operands...
    16731694 const Value *Operand = I.getNumOperands() ? I.getOperand(0) : 0;
    16741695
    19351956 Out << ", align " << cast(I).getAlignment();
    19361957 } else if (isa(I) && cast(I).getAlignment()) {
    19371958 Out << ", align " << cast(I).getAlignment();
    1959 } else if (const AtomicCmpXchgInst *CXI = dyn_cast(&I)) {
    1960 writeAtomic(CXI->getOrdering(), CXI->getSynchScope());
    1961 } else if (const AtomicRMWInst *RMWI = dyn_cast(&I)) {
    1962 writeAtomic(RMWI->getOrdering(), RMWI->getSynchScope());
    19381963 } else if (const FenceInst *FI = dyn_cast(&I)) {
    19391964 writeAtomic(FI->getOrdering(), FI->getSynchScope());
    19401965 }
    127127 case Alloca: return "alloca";
    128128 case Load: return "load";
    129129 case Store: return "store";
    130 case AtomicCmpXchg: return "cmpxchg";
    131 case AtomicRMW: return "atomicrmw";
    130132 case Fence: return "fence";
    131133 case GetElementPtr: return "getelementptr";
    132134
    11021102 setInstructionSubclassData((getSubclassDataFromInstruction() & 1) |
    11031103 ((Log2_32(Align)+1) << 1));
    11041104 assert(getAlignment() == Align && "Alignment representation error!");
    1105 }
    1106
    1107 //===----------------------------------------------------------------------===//
    1108 // AtomicCmpXchgInst Implementation
    1109 //===----------------------------------------------------------------------===//
    1110
    1111 void AtomicCmpXchgInst::Init(Value *Ptr, Value *Cmp, Value *NewVal,
    1112 AtomicOrdering Ordering,
    1113 SynchronizationScope SynchScope) {
    1114 Op<0>() = Ptr;
    1115 Op<1>() = Cmp;
    1116 Op<2>() = NewVal;
    1117 setOrdering(Ordering);
    1118 setSynchScope(SynchScope);
    1119
    1120 assert(getOperand(0) && getOperand(1) && getOperand(2) &&
    1121 "All operands must be non-null!");
    1122 assert(getOperand(0)->getType()->isPointerTy() &&
    1123 "Ptr must have pointer type!");
    1124 assert(getOperand(1)->getType() ==
    1125 cast(getOperand(0)->getType())->getElementType()
    1126 && "Ptr must be a pointer to Cmp type!");
    1127 assert(getOperand(2)->getType() ==
    1128 cast(getOperand(0)->getType())->getElementType()
    1129 && "Ptr must be a pointer to NewVal type!");
    1130 assert(Ordering != NotAtomic &&
    1131 "AtomicCmpXchg instructions must be atomic!");
    1132 }
    1133
    1134 AtomicCmpXchgInst::AtomicCmpXchgInst(Value *Ptr, Value *Cmp, Value *NewVal,
    1135 AtomicOrdering Ordering,
    1136 SynchronizationScope SynchScope,
    1137 Instruction *InsertBefore)
    1138 : Instruction(Cmp->getType(), AtomicCmpXchg,
    1139 OperandTraits::op_begin(this),
    1140 OperandTraits::operands(this),
    1141 InsertBefore) {
    1142 Init(Ptr, Cmp, NewVal, Ordering, SynchScope);
    1143 }
    1144
    1145 AtomicCmpXchgInst::AtomicCmpXchgInst(Value *Ptr, Value *Cmp, Value *NewVal,
    1146 AtomicOrdering Ordering,
    1147 SynchronizationScope SynchScope,
    1148 BasicBlock *InsertAtEnd)
    1149 : Instruction(Cmp->getType(), AtomicCmpXchg,
    1150 OperandTraits::op_begin(this),
    1151 OperandTraits::operands(this),
    1152 InsertAtEnd) {
    1153 Init(Ptr, Cmp, NewVal, Ordering, SynchScope);
    1154 }
    1155
    1156 //===----------------------------------------------------------------------===//
    1157 // AtomicRMWInst Implementation
    1158 //===----------------------------------------------------------------------===//
    1159
    1160 void AtomicRMWInst::Init(BinOp Operation, Value *Ptr, Value *Val,
    1161 AtomicOrdering Ordering,
    1162 SynchronizationScope SynchScope) {
    1163 Op<0>() = Ptr;
    1164 Op<1>() = Val;
    1165 setOperation(Operation);
    1166 setOrdering(Ordering);
    1167 setSynchScope(SynchScope);
    1168
    1169 assert(getOperand(0) && getOperand(1) &&
    1170 "All operands must be non-null!");
    1171 assert(getOperand(0)->getType()->isPointerTy() &&
    1172 "Ptr must have pointer type!");
    1173 assert(getOperand(1)->getType() ==
    1174 cast(getOperand(0)->getType())->getElementType()
    1175 && "Ptr must be a pointer to Val type!");
    1176 assert(Ordering != NotAtomic &&
    1177 "AtomicRMW instructions must be atomic!");
    1178 }
    1179
    1180 AtomicRMWInst::AtomicRMWInst(BinOp Operation, Value *Ptr, Value *Val,
    1181 AtomicOrdering Ordering,
    1182 SynchronizationScope SynchScope,
    1183 Instruction *InsertBefore)
    1184 : Instruction(Val->getType(), AtomicRMW,
    1185 OperandTraits::op_begin(this),
    1186 OperandTraits::operands(this),
    1187 InsertBefore) {
    1188 Init(Operation, Ptr, Val, Ordering, SynchScope);
    1189 }
    1190
    1191 AtomicRMWInst::AtomicRMWInst(BinOp Operation, Value *Ptr, Value *Val,
    1192 AtomicOrdering Ordering,
    1193 SynchronizationScope SynchScope,
    1194 BasicBlock *InsertAtEnd)
    1195 : Instruction(Val->getType(), AtomicRMW,
    1196 OperandTraits::op_begin(this),
    1197 OperandTraits::operands(this),
    1198 InsertAtEnd) {
    1199 Init(Operation, Ptr, Val, Ordering, SynchScope);
    11051200 }
    11061201
    11071202 //===----------------------------------------------------------------------===//
    31473242 isVolatile(), getAlignment());
    31483243 }
    31493244
    3245 AtomicCmpXchgInst *AtomicCmpXchgInst::clone_impl() const {
    3246 AtomicCmpXchgInst *Result =
    3247 new AtomicCmpXchgInst(getOperand(0), getOperand(1), getOperand(2),
    3248 getOrdering(), getSynchScope());
    3249 Result->setVolatile(isVolatile());
    3250 return Result;
    3251 }
    3252
    3253 AtomicRMWInst *AtomicRMWInst::clone_impl() const {
    3254 AtomicRMWInst *Result =
    3255 new AtomicRMWInst(getOperation(),getOperand(0), getOperand(1),
    3256 getOrdering(), getSynchScope());
    3257 Result->setVolatile(isVolatile());
    3258 return Result;
    3259 }
    3260
    31503261 FenceInst *FenceInst::clone_impl() const {
    31513262 return new FenceInst(getContext(), getOrdering(), getSynchScope());
    31523263 }
    287287 void visitUserOp1(Instruction &I);
    288288 void visitUserOp2(Instruction &I) { visitUserOp1(I); }
    289289 void visitIntrinsicFunctionCall(Intrinsic::ID ID, CallInst &CI);
    290 void visitAtomicCmpXchgInst(AtomicCmpXchgInst &CXI);
    291 void visitAtomicRMWInst(AtomicRMWInst &RMWI);
    290292 void visitFenceInst(FenceInst &FI);
    291293 void visitAllocaInst(AllocaInst &AI);
    292294 void visitExtractValueInst(ExtractValueInst &EVI);
    13261328 visitInstruction(AI);
    13271329 }
    13281330
    1331 void Verifier::visitAtomicCmpXchgInst(AtomicCmpXchgInst &CXI) {
    1332 Assert1(CXI.getOrdering() != NotAtomic,
    1333 "cmpxchg instructions must be atomic.", &CXI);
    1334 Assert1(CXI.getOrdering() != Unordered,
    1335 "cmpxchg instructions cannot be unordered.", &CXI);
    1336 PointerType *PTy = dyn_cast(CXI.getOperand(0)->getType());
    1337 Assert1(PTy, "First cmpxchg operand must be a pointer.", &CXI);
    1338 Type *ElTy = PTy->getElementType();
    1339 Assert2(ElTy == CXI.getOperand(1)->getType(),
    1340 "Expected value type does not match pointer operand type!",
    1341 &CXI, ElTy);
    1342 Assert2(ElTy == CXI.getOperand(2)->getType(),
    1343 "Stored value type does not match pointer operand type!",
    1344 &CXI, ElTy);
    1345 visitInstruction(CXI);
    1346 }
    1347
    1348 void Verifier::visitAtomicRMWInst(AtomicRMWInst &RMWI) {
    1349 Assert1(RMWI.getOrdering() != NotAtomic,
    1350 "atomicrmw instructions must be atomic.", &RMWI);
    1351 Assert1(RMWI.getOrdering() != Unordered,
    1352 "atomicrmw instructions cannot be unordered.", &RMWI);
    1353 PointerType *PTy = dyn_cast(RMWI.getOperand(0)->getType());
    1354 Assert1(PTy, "First atomicrmw operand must be a pointer.", &RMWI);
    1355 Type *ElTy = PTy->getElementType();
    1356 Assert2(ElTy == RMWI.getOperand(1)->getType(),
    1357 "Argument value type does not match pointer operand type!",
    1358 &RMWI, ElTy);
    1359 Assert1(AtomicRMWInst::FIRST_BINOP <= RMWI.getOperation() &&
    1360 RMWI.getOperation() <= AtomicRMWInst::LAST_BINOP,
    1361 "Invalid binary operation!", &RMWI);
    1362 visitInstruction(RMWI);
    1363 }
    1364
    13291365 void Verifier::visitFenceInst(FenceInst &FI) {
    13301366 const AtomicOrdering Ordering = FI.getOrdering();
    13311367 Assert1(Ordering == Acquire || Ordering == Release ||