llvm.org GIT mirror llvm / ac99eed
docs: Sphinxify TestingGuide git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@167979 91177308-0d34-0410-b5e6-96231b3b80d8 Sean Silva 6 years ago
7 changed file(s) with 723 addition(s) and 930 deletion(s). Raw diff Collapse all Expand all
179179 features added. Some tips for getting your testcase approved:
180180
181181 * All feature and regression test cases are added to the ``llvm/test``
182 directory. The appropriate sub-directory should be selected (see the `Testing
183 Guide `_ for details).
182 directory. The appropriate sub-directory should be selected (see the
183 :doc:`Testing Guide ` for details).
184184
185185 * Test cases should be written in `LLVM assembly language `_
186186 unless the feature or regression being tested requires another language
10721072 module contains a comprehensive correctness, performance, and benchmarking test
10731073 suite for LLVM. It is a separate Subversion module because not every LLVM user
10741074 is interested in downloading or building such a comprehensive test suite. For
1075 further details on this test suite, please see the `Testing
1076 Guide `_ document.
1075 further details on this test suite, please see the :doc:`Testing Guide
1076 ` document.
10771077
10781078 .. _tools:
10791079
338338 usage for this is to assist in running specific suites of tests. If
339339 ``TESTSUITE`` is not set, the implementation of ``check-local`` should run all
340340 normal tests. It is up to the project to define what different values for
341 ``TESTSUTE`` will do. See the `Testing Guide `_ for further
341 ``TESTSUTE`` will do. See the :doc:`Testing Guide ` for further
342342 details.
343343
344344 ``check-local``
155155 * LLVM provides a ``tcl`` procedure that is used by ``Dejagnu`` to run tests.
156156 It can be found in ``llvm/lib/llvm-dg.exp``. This test procedure uses ``RUN``
157157 lines in the actual test case to determine how to run the test. See the
158 `TestingGuide `_ for more details. You can easily write
159 Makefile support similar to the Makefiles in ``llvm/test`` to use ``Dejagnu``
160 to run your project's tests.
158 :doc:`TestingGuide` for more details. You can easily write Makefile
159 support similar to the Makefiles in ``llvm/test`` to use ``Dejagnu`` to
160 run your project's tests.
161161
162162 * LLVM contains an optional package called ``llvm-test``, which provides
163163 benchmarks and programs that are known to compile with the Clang front
+0
-921
docs/TestingGuide.html less more
None
1 "http://www.w3.org/TR/html4/strict.dtd">
2
3
4
5 LLVM Testing Infrastructure Guide
6
7
8
9
10

11 LLVM Testing Infrastructure Guide
12
13
14
15
  • Overview
  • 16
  • Requirements
  • 17
  • LLVM testing infrastructure organization
  • 18
    19
  • Regression tests
  • 20
  • test-suite
  • 21
  • Debugging Information tests
  • 22
    23
    24
  • Quick start
  • 25
    26
  • Regression tests
  • 27
  • Debugging Information tests
  • 28
    29
    30
  • Regression test structure
  • 31
    32
  • Writing new regression tests
  • 33
  • The FileCheck utility
  • 34
  • Variables and substitutions
  • 35
  • Other features
  • 36
    37
    38
  • test-suite Overview
  • 39
    40
  • test-suite Quickstart
  • 41
  • test-suite Makefiles
  • 42
    43
    44
    45
    46
    47

    Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner

    48
    49
    50
    51

    Overview

    52
    53
    54
    55
    56

    This document is the reference manual for the LLVM testing infrastructure. It

    57 documents the structure of the LLVM testing infrastructure, the tools needed to
    58 use it, and how to add and run tests.

    59
    60
    61
    62
    63

    Requirements

    64
    65
    66
    67
    68

    In order to use the LLVM testing infrastructure, you will need all of the

    69 software required to build LLVM, as well
    70 as Python 2.4 or later.

    71
    72
    73
    74
    75

    LLVM testing infrastructure organization

    76
    77
    78
    79
    80

    The LLVM testing infrastructure contains two major categories of tests:

    81 regression tests and whole programs. The regression tests are contained inside
    82 the LLVM repository itself under llvm/test and are expected to always
    83 pass -- they should be run before every commit.

    84
    85

    The whole programs tests are referred to as the "LLVM test suite" (or

    86 "test-suite") and are in the test-suite module in subversion. For
    87 historical reasons, these tests are also referred to as the "nightly tests" in
    88 places, which is less ambiguous than "test-suite" and remains in use although we
    89 run them much more often than nightly.

    90
    91
    92

    Regression tests

    93
    94
    95
    96
    97

    The regression tests are small pieces of code that test a specific feature of

    98 LLVM or trigger a specific bug in LLVM. They are usually written in LLVM
    99 assembly language, but can be written in other languages if the test targets a
    100 particular language front end (and the appropriate --with-llvmgcc
    101 options were used at configure time of the llvm module). These
    102 tests are driven by the 'lit' testing tool, which is part of LLVM.

    103
    104

    These code fragments are not complete programs. The code generated

    105 from them is never executed to determine correct behavior.

    106
    107

    These code fragment tests are located in the llvm/test

    108 directory.

    109
    110

    Typically when a bug is found in LLVM, a regression test containing

    111 just enough code to reproduce the problem should be written and placed
    112 somewhere underneath this directory. In most cases, this will be a small
    113 piece of LLVM assembly language code, often distilled from an actual
    114 application or benchmark.

    115
    116
    117
    118
    119

    test-suite

    120
    121
    122
    123
    124

    The test suite contains whole programs, which are pieces of code which can be

    125 compiled and linked into a stand-alone program that can be executed. These
    126 programs are generally written in high level languages such as C or C++.

    127
    128

    These programs are compiled using a user specified compiler and set of flags,

    129 and then executed to capture the program output and timing information. The
    130 output of these programs is compared to a reference output to ensure that the
    131 program is being compiled correctly.

    132
    133

    In addition to compiling and executing programs, whole program tests serve as

    134 a way of benchmarking LLVM performance, both in terms of the efficiency of the
    135 programs generated as well as the speed with which LLVM compiles, optimizes, and
    136 generates code.

    137
    138

    The test-suite is located in the test-suite Subversion module.

    139
    140
    141
    142
    143

    Debugging Information tests

    144
    145
    146
    147
    148

    The test suite contains tests to check quality of debugging information.

    149 The test are written in C based languages or in LLVM assembly language.

    150
    151

    These tests are compiled and run under a debugger. The debugger output

    152 is checked to validate of debugging information. See README.txt in the
    153 test suite for more information . This test suite is located in the
    154 debuginfo-tests Subversion module.

    155
    156
    157
    158
    159
    160
    161

    Quick start

    162
    163
    164
    165
    166

    The tests are located in two separate Subversion modules. The regressions

    167 tests are in the main "llvm" module under the directory
    168 llvm/test (so you get these tests for free with the main llvm
    169 tree). Use "make check-all" to run the regression tests after building
    170 LLVM.

    171
    172

    The more comprehensive test suite that includes whole programs in C and C++

    173 is in the test-suite
    174 module. See test-suite Quickstart
    175 for more information on running these tests.

    176
    177
    178

    Regression tests

    179
    180
    181

    To run all of the LLVM regression tests, use master Makefile in

    182 the llvm/test directory:

    183
    184
    185
    
                      
                    
    186 % gmake -C llvm/test
    187
    188
    189
    190

    or

    191
    192
    193
    
                      
                    
    194 % gmake check
    195
    196
    197
    198

    If you have Clang checked out and built,

    199 you can run the LLVM and Clang tests simultaneously using:

    200
    201

    or

    202
    203
    204
    
                      
                    
    205 % gmake check-all
    206
    207
    208
    209

    To run the tests with Valgrind (Memcheck by default), just append

    210 VG=1 to the commands above, e.g.:

    211
    212
    213
    
                      
                    
    214 % gmake check VG=1
    215
    216
    217
    218

    To run individual tests or subsets of tests, you can use the 'llvm-lit'

    219 script which is built as part of LLVM. For example, to run the
    220 'Integer/BitPacked.ll' test by itself you can run:

    221
    222
    223
    
                      
                    
    224 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
    225
    226
    227
    228

    or to run all of the ARM CodeGen tests:

    229
    230
    231
    
                      
                    
    232 % llvm-lit ~/llvm/test/CodeGen/ARM
    233
    234
    235
    236

    For more information on using the 'lit' tool, see 'llvm-lit --help' or the

    237 'lit' man page.

    238
    239
    240
    241
    242

    Debugging Information tests

    243
    244
    245
    246
    247

    To run debugging information tests simply checkout the tests inside

    248 clang/test directory.

    249
    250
    251
    
                      
                    
    252 %cd clang/test
    253 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
    254
    255
    256
    257

    These tests are already set up to run as part of clang regression tests.

    258
    259
    260
    261
    262
    263
    264
    265
    266

    Regression test structure

    267
    268
    269

    The LLVM regression tests are driven by 'lit' and are located in

    270 the llvm/test directory.
    271
    272

    This directory contains a large array of small tests

    273 that exercise various features of LLVM and to ensure that regressions do not
    274 occur. The directory is broken into several sub-directories, each focused on
    275 a particular area of LLVM. A few of the important ones are:

    276
    277
    278
  • Analysis: checks Analysis passes.
  • 279
  • Archive: checks the Archive library.
  • 280
  • Assembler: checks Assembly reader/writer functionality.
  • 281
  • Bitcode: checks Bitcode reader/writer functionality.
  • 282
  • CodeGen: checks code generation and each target.
  • 283
  • Features: checks various features of the LLVM language.
  • 284
  • Linker: tests bitcode linking.
  • 285
  • Transforms: tests each of the scalar, IPO, and utility
  • 286 transforms to ensure they make the right transformations.
    287
  • Verifier: tests the IR verifier.
  • 288
    289
    290
    291

    Writing new regression tests

    292
    293
    294

    The regression test structure is very simple, but does require some

    295 information to be set. This information is gathered via configure and
    296 is written to a file, lit.site.cfg
    297 in llvm/test. The llvm/test Makefile does this work for
    298 you.

    299
    300

    In order for the regression tests to work, each directory of tests must

    301 have a lit.local.cfg file. Lit looks for this file to determine how
    302 to run the tests. This file is just Python code and thus is very flexible,
    303 but we've standardized it for the LLVM regression tests. If you're adding a
    304 directory of tests, just copy lit.local.cfg from another directory to
    305 get running. The standard lit.local.cfg simply specifies which files
    306 to look in for tests. Any directory that contains only directories does not
    307 need the lit.local.cfg file. Read the
    308 Lit documentation for more
    309 information.

    310
    311

    The llvm-runtests function looks at each file that is passed to

    312 it and gathers any lines together that match "RUN:". These are the "RUN" lines
    313 that specify how the test is to be run. So, each test script must contain
    314 RUN lines if it is to do anything. If there are no RUN lines, the
    315 llvm-runtests function will issue an error and the test will
    316 fail.

    317
    318

    RUN lines are specified in the comments of the test program using the

    319 keyword RUN followed by a colon, and lastly the command (pipeline)
    320 to execute. Together, these lines form the "script" that
    321 llvm-runtests executes to run the test case. The syntax of the
    322 RUN lines is similar to a shell's syntax for pipelines including I/O
    323 redirection and variable substitution. However, even though these lines
    324 may look like a shell script, they are not. RUN lines are interpreted
    325 directly by the Tcl exec command. They are never executed by a
    326 shell. Consequently the syntax differs from normal shell script syntax in a
    327 few ways. You can specify as many RUN lines as needed.

    328
    329

    lit performs substitution on each RUN line to replace LLVM tool

    330 names with the full paths to the executable built for each tool (in
    331 $(LLVM_OBJ_ROOT)/$(BuildMode)/bin). This ensures that lit does not
    332 invoke any stray LLVM tools in the user's path during testing.

    333
    334

    Each RUN line is executed on its own, distinct from other lines unless

    335 its last character is \. This continuation character causes the RUN
    336 line to be concatenated with the next one. In this way you can build up long
    337 pipelines of commands without making huge line lengths. The lines ending in
    338 \ are concatenated until a RUN line that doesn't end in \ is
    339 found. This concatenated set of RUN lines then constitutes one execution.
    340 Tcl will substitute variables and arrange for the pipeline to be executed. If
    341 any process in the pipeline fails, the entire line (and test case) fails too.
    342

    343
    344

    Below is an example of legal RUN lines in a .ll file:

    345
    346
    347
    
                      
                    
    348 ; RUN: llvm-as < %s | llvm-dis > %t1
    349 ; RUN: llvm-dis < %s.bc-13 > %t2
    350 ; RUN: diff %t1 %t2
    351
    352
    353
    354

    As with a Unix shell, the RUN: lines permit pipelines and I/O redirection

    355 to be used. However, the usage is slightly different than for Bash. To check
    356 what's legal, see the documentation for the
    357 Tcl exec
    358 command and the
    359 tutorial.
    360 The major differences are:

    361
    362
  • You can't do 2>&1. That will cause Tcl to write to a
  • 363 file named &1. Usually this is done to get stderr to go through
    364 a pipe. You can do that in tcl with |& so replace this idiom:
    365 ... 2>&1 | grep with ... |& grep
    366
  • You can only redirect to a file, not to another descriptor and not from
  • 367 a here document.
    368
  • tcl supports redirecting to open files with the @ syntax but you
  • 369 shouldn't use that here.
    370
    371
    372

    There are some quoting rules that you must pay attention to when writing

    373 your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any
    374 quote characters so they will get passed to the invoked program. For
    375 example:

    376
    377
    378
    
                      
                    
    379 ... | grep 'find this string'
    380
    381
    382
    383

    This will fail because the ' characters are passed to grep. This would

    384 instruction grep to look for 'find in the files this and
    385 string'. To avoid this use curly braces to tell Tcl that it should
    386 treat everything enclosed as one value. So our example would become:

    387
    388
    389
    
                      
                    
    390 ... | grep {find this string}
    391
    392
    393
    394

    Additionally, the characters [ and ] are treated

    395 specially by Tcl. They tell Tcl to interpret the content as a command to
    396 execute. Since these characters are often used in regular expressions this can
    397 have disastrous results and cause the entire test run in a directory to fail.
    398 For example, a common idiom is to look for some basicblock number:

    399
    400
    401
    
                      
                    
    402 ... | grep bb[2-8]
    403
    404
    405
    406

    This, however, will cause Tcl to fail because its going to try to execute

    407 a program named "2-8". Instead, what you want is this:

    408
    409
    410
    
                      
                    
    411 ... | grep {bb\[2-8\]}
    412
    413
    414
    415

    Finally, if you need to pass the \ character down to a program,

    416 then it must be doubled. This is another Tcl special character. So, suppose
    417 you had:
    418
    419
    420
    
                      
                    
    421 ... | grep 'i32\*'
    422
    423
    424
    425

    This will fail to match what you want (a pointer to i32). First, the

    426 ' do not get stripped off. Second, the \ gets stripped off
    427 by Tcl so what grep sees is: 'i32*'. That's not likely to match
    428 anything. To resolve this you must use \\ and the {}, like
    429 this:

    430
    431
    432
    
                      
                    
    433 ... | grep {i32\\*}
    434
    435
    436
    437

    If your system includes GNU grep, make sure

    438 that GREP_OPTIONS is not set in your environment. Otherwise,
    439 you may get invalid results (both false positives and false
    440 negatives).

    441
    442
    443
    444
    445

    The FileCheck utility

    446
    447
    448
    449
    450

    A powerful feature of the RUN: lines is that it allows any arbitrary commands

    451 to be executed as part of the test harness. While standard (portable) unix
    452 tools like 'grep' work fine on run lines, as you see above, there are a lot
    453 of caveats due to interaction with Tcl syntax, and we want to make sure the
    454 run lines are portable to a wide range of systems. Another major problem is
    455 that grep is not very good at checking to verify that the output of a tools
    456 contains a series of different output in a specific order. The FileCheck
    457 tool was designed to help with these problems.

    458
    459

    FileCheck (whose basic command line arguments are described in

    460 href="http://llvm.org/cmds/FileCheck.html">the FileCheck man page is
    461 designed to read a file to check from standard input, and the set of things
    462 to verify from a file specified as a command line argument. A simple example
    463 of using FileCheck from a RUN line looks like this:

    464
    465
    466
    
                      
                    
    467 ; RUN: llvm-as < %s | llc -march=x86-64 | FileCheck %s
    468
    469
    470
    471

    This syntax says to pipe the current file ("%s") into llvm-as, pipe that into

    472 llc, then pipe the output of llc into FileCheck. This means that FileCheck will
    473 be verifying its standard input (the llc output) against the filename argument
    474 specified (the original .ll file specified by "%s"). To see how this works,
    475 let's look at the rest of the .ll file (after the RUN line):

    476
    477
    478
    
                      
                    
    479 define void @sub1(i32* %p, i32 %v) {
    480 entry:
    481 ; CHECK: sub1:
    482 ; CHECK: subl
    483 %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
    484 ret void
    485 }
    486
    487 define void @inc4(i64* %p) {
    488 entry:
    489 ; CHECK: inc4:
    490 ; CHECK: incq
    491 %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
    492 ret void
    493 }
    494
    495
    496
    497

    Here you can see some "CHECK:" lines specified in comments. Now you can see

    498 how the file is piped into llvm-as, then llc, and the machine code output is
    499 what we are verifying. FileCheck checks the machine code output to verify that
    500 it matches what the "CHECK:" lines specify.

    501
    502

    The syntax of the CHECK: lines is very simple: they are fixed strings that

    503 must occur in order. FileCheck defaults to ignoring horizontal whitespace
    504 differences (e.g. a space is allowed to match a tab) but otherwise, the contents
    505 of the CHECK: line is required to match some thing in the test file exactly.

    506
    507

    One nice thing about FileCheck (compared to grep) is that it allows merging

    508 test cases together into logical groups. For example, because the test above
    509 is checking for the "sub1:" and "inc4:" labels, it will not match unless there
    510 is a "subl" in between those labels. If it existed somewhere else in the file,
    511 that would not count: "grep subl" matches if subl exists anywhere in the
    512 file.

    513
    514
    515

    516 The FileCheck -check-prefix option
    517
    518
    519
    520
    521

    The FileCheck -check-prefix option allows multiple test configurations to be

    522 driven from one .ll file. This is useful in many circumstances, for example,
    523 testing different architectural variants with llc. Here's a simple example:

    524
    525
    526
    
                      
                    
    527 ; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
    528 ; RUN: | FileCheck %s -check-prefix=X32
    529 ; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
    530 ; RUN: | FileCheck %s -check-prefix=X64
    531
    532 define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
    533 %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
    534 ret <4 x i32> %tmp1
    535 ; X32: pinsrd_1:
    536 ; X32: pinsrd $1, 4(%esp), %xmm0
    537
    538 ; X64: pinsrd_1:
    539 ; X64: pinsrd $1, %edi, %xmm0
    540 }
    541
    542
    543
    544

    In this case, we're testing that we get the expected code generation with

    545 both 32-bit and 64-bit code generation.

    546
    547
    548
    549
    550

    551 The "CHECK-NEXT:" directive
    552
    553
    554
    555
    556

    Sometimes you want to match lines and would like to verify that matches

    557 happen on exactly consecutive lines with no other lines in between them. In
    558 this case, you can use CHECK: and CHECK-NEXT: directives to specify this. If
    559 you specified a custom check prefix, just use "<PREFIX>-NEXT:". For
    560 example, something like this works as you'd expect:

    561
    562
    563
    
                      
                    
    564 define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) {
    565 %tmp3 = load <2 x double>* %A, align 16
    566 %tmp7 = insertelement <2 x double> undef, double %B, i32 0
    567 %tmp9 = shufflevector <2 x double> %tmp3,
    568 <2 x double> %tmp7,
    569 <2 x i32> < i32 0, i32 2 >
    570 store <2 x double> %tmp9, <2 x double>* %r, align 16
    571 ret void
    572
    573 ; CHECK: t2:
    574 ; CHECK: movl 8(%esp), %eax
    575 ; CHECK-NEXT: movapd (%eax), %xmm0
    576 ; CHECK-NEXT: movhpd 12(%esp), %xmm0
    577 ; CHECK-NEXT: movl 4(%esp), %eax
    578 ; CHECK-NEXT: movapd %xmm0, (%eax)
    579 ; CHECK-NEXT: ret
    580 }
    581
    582
    583
    584

    CHECK-NEXT: directives reject the input unless there is exactly one newline

    585 between it an the previous directive. A CHECK-NEXT cannot be the first
    586 directive in a file.

    587
    588
    589
    590
    591

    592 The "CHECK-NOT:" directive
    593
    594
    595
    596
    597

    The CHECK-NOT: directive is used to verify that a string doesn't occur

    598 between two matches (or the first match and the beginning of the file). For
    599 example, to verify that a load is removed by a transformation, a test like this
    600 can be used:

    601
    602
    603
    
                      
                    
    604 define i8 @coerce_offset0(i32 %V, i32* %P) {
    605 store i32 %V, i32* %P
    606
    607 %P2 = bitcast i32* %P to i8*
    608 %P3 = getelementptr i8* %P2, i32 2
    609
    610 %A = load i8* %P3
    611 ret i8 %A
    612 ; CHECK: @coerce_offset0
    613 ; CHECK-NOT: load
    614 ; CHECK: ret i8
    615 }
    616
    617
    618
    619
    620
    621
    622

    623 FileCheck Pattern Matching Syntax
    624
    625
    626
    627
    628
    629
    630

    The CHECK: and CHECK-NOT: directives both take a pattern to match. For most

    631 uses of FileCheck, fixed string matching is perfectly sufficient. For some
    632 things, a more flexible form of matching is desired. To support this, FileCheck
    633 allows you to specify regular expressions in matching strings, surrounded by
    634 double braces: {{yourregex}}. Because we want to use fixed string
    635 matching for a majority of what we do, FileCheck has been designed to support
    636 mixing and matching fixed string matching with regular expressions. This allows
    637 you to write things like this:

    638
    639
    640
    
                      
                    
    641 ; CHECK: movhpd {{[0-9]+}}(%esp), {{%xmm[0-7]}}
    642
    643
    644
    645

    In this case, any offset from the ESP register will be allowed, and any xmm

    646 register will be allowed.

    647
    648

    Because regular expressions are enclosed with double braces, they are

    649 visually distinct, and you don't need to use escape characters within the double
    650 braces like you would in C. In the rare case that you want to match double
    651 braces explicitly from the input, you can use something ugly like
    652 {{[{][{]}} as your pattern.

    653
    654
    655
    656
    657
    658
    659

    660 FileCheck Variables
    661
    662
    663
    664
    665
    666
    667
    668

    It is often useful to match a pattern and then verify that it occurs again

    669 later in the file. For codegen tests, this can be useful to allow any register,
    670 but verify that that register is used consistently later. To do this, FileCheck
    671 allows named variables to be defined and substituted into patterns. Here is a
    672 simple example:

    673
    674
    675
    
                      
                    
    676 ; CHECK: test5:
    677 ; CHECK: notw [[REGISTER:%[a-z]+]]
    678 ; CHECK: andw {{.*}}[[REGISTER]]
    679
    680
    681
    682

    The first check line matches a regex (%[a-z]+) and captures it into

    683 the variables "REGISTER". The second line verifies that whatever is in REGISTER
    684 occurs later in the file after an "andw". FileCheck variable references are
    685 always contained in [[ ]] pairs, are named, and their names can be
    686 formed with the regex "[a-zA-Z][a-zA-Z0-9]*". If a colon follows the
    687 name, then it is a definition of the variable, if not, it is a use.

    688
    689

    FileCheck variables can be defined multiple times, and uses always get the

    690 latest value. Note that variables are all read at the start of a "CHECK" line
    691 and are all defined at the end. This means that if you have something like
    692 "CHECK: [[XYZ:.*]]x[[XYZ]]" that the check line will read the previous
    693 value of the XYZ variable and define a new one after the match is performed. If
    694 you need to do something like this you can probably take advantage of the fact
    695 that FileCheck is not actually line-oriented when it matches, this allows you to
    696 define two separate CHECK lines that match on the same line.
    697

    698
    699
    700
    701
    702
    703
    704
    705
    706

    Variables and substitutions

    707
    708
    709

    With a RUN line there are a number of substitutions that are permitted. In

    710 general, any Tcl variable that is available in the substitute
    711 function (in test/lib/llvm.exp) can be substituted into a RUN line.
    712 To make a substitution just write the variable's name preceded by a $.
    713 Additionally, for compatibility reasons with previous versions of the test
    714 library, certain names can be accessed with an alternate syntax: a % prefix.
    715 These alternates are deprecated and may go away in a future version.
    716

    717

    Here are the available variable names. The alternate syntax is listed in

    718 parentheses.

    719
    720
    721
    $test (%s)
    722
    The full path to the test case's source. This is suitable for passing
    723 on the command line as the input to an llvm tool.
    724
    725
    %(line), %(line+number), %(line-number)
    726
    The number of the line where this variable is used, with an optional
    727 integer offset. This can be used in tests with multiple RUN: lines, which
    728 reference test file's line numbers.
    729
    730
    $srcdir
    731
    The source directory from where the "make check" was run.
    732
    733
    objdir
    734
    The object directory that corresponds to the $srcdir.
    735
    736
    subdir
    737
    A partial path from the test directory that contains the
    738 sub-directory that contains the test source being executed.
    739
    740
    srcroot
    741
    The root directory of the LLVM src tree.
    742
    743
    objroot
    744
    The root directory of the LLVM object tree. This could be the same
    745 as the srcroot.
    746
    747
    path
    748
    The path to the directory that contains the test case source. This is
    749 for locating any supporting files that are not generated by the test, but
    750 used by the test.
    751
    752
    tmp
    753
    The path to a temporary file name that could be used for this test case.
    754 The file name won't conflict with other test cases. You can append to it if
    755 you need multiple temporaries. This is useful as the destination of some
    756 redirected output.
    757
    758
    target_triplet (%target_triplet)
    759
    The target triplet that corresponds to the current host machine (the one
    760 running the test cases). This should probably be called "host".
    761
    762
    link (%link)
    763
    This full link command used to link LLVM executables. This has all the
    764 configured -I, -L and -l options.
    765
    766
    shlibext (%shlibext)
    767
    The suffix for the host platforms share library (dll) files. This
    768 includes the period as the first character.
    769
    770

    To add more variables, two things need to be changed. First, add a line in

    771 the test/Makefile that creates the site.exp file. This will
    772 "set" the variable as a global in the site.exp file. Second, in the
    773 test/lib/llvm.exp file, in the substitute proc, add the variable name
    774 to the list of "global" declarations at the beginning of the proc. That's it,
    775 the variable can then be used in test scripts.

    776
    777
    778
    779

    Other Features

    780
    781
    782

    To make RUN line writing easier, there are several shell scripts located

    783 in the llvm/test/Scripts directory. This directory is in the PATH
    784 when running tests, so you can just call these scripts using their name. For
    785 example:

    786
    787
    ignore
    788
    This script runs its arguments and then always returns 0. This is useful
    789 in cases where the test needs to cause a tool to generate an error (e.g. to
    790 check the error output). However, any program in a pipeline that returns a
    791 non-zero result will cause the test to fail. This script overcomes that
    792 issue and nicely documents that the test case is purposefully ignoring the
    793 result code of the tool
    794
    795
    not
    796
    This script runs its arguments and then inverts the result code from
    797 it. Zero result codes become 1. Non-zero result codes become 0. This is
    798 useful to invert the result of a grep. For example "not grep X" means
    799 succeed only if you don't find X in the input.
    800
    801
    802

    Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.

    803 You can easily mark a test as XFAIL just by including XFAIL: on a
    804 line near the top of the file. This signals that the test case should succeed
    805 if the test fails. Such test cases are counted separately by the testing
    806 tool. To specify an expected fail, use the XFAIL keyword in the comments of
    807 the test program followed by a colon and one or more failure patterns. Each
    808 failure pattern can be either '*' (to specify fail everywhere), or a part of a
    809 target triple (indicating the test should fail on that platform), or the name
    810 of a configurable feature (for example, "loadable_module"). If there is a
    811 match, the test is expected to fail. If not, the test is expected to
    812 succeed. To XFAIL everywhere just specify XFAIL: *. Here is an
    813 example of an XFAIL line:

    814
    815
    816
    
                      
                    
    817 ; XFAIL: darwin,sun
    818
    819
    820
    821

    To make the output more useful, the llvm_runtest function wil

    822 scan the lines of the test case for ones that contain a pattern that matches
    823 PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
    824 is related to the test case. The number after "PR" specifies the LLVM bugzilla
    825 number. When a PR number is specified, it will be used in the pass/fail
    826 reporting. This is useful to quickly get some context when a test fails.

    827
    828

    Finally, any line that contains "END." will cause the special

    829 interpretation of lines to terminate. This is generally done right after the
    830 last RUN: line. This has two side effects: (a) it prevents special
    831 interpretation of lines that are part of the test program, not the
    832 instructions to the test case, and (b) it speeds things up for really big test
    833 cases by avoiding interpretation of the remainder of the file.

    834
    835
    836
    837
    838
    839
    840

    test-suite Overview

    841
    842
    843
    844
    845

    The test-suite module contains a number of programs that can be

    846 compiled and executed. The test-suite includes reference outputs for
    847 all of the programs, so that the output of the executed program can be checked
    848 for correctness.

    849
    850

    test-suite tests are divided into three types of tests: MultiSource,

    851 SingleSource, and External.

    852
    853
    854
  • test-suite/SingleSource
  • 855

    The SingleSource directory contains test programs that are only a single

    856 source file in size. These are usually small benchmark programs or small
    857 programs that calculate a particular value. Several such programs are grouped
    858 together in each directory.

    859
    860
  • test-suite/MultiSource
  • 861

    The MultiSource directory contains subdirectories which contain entire

    862 programs with multiple source files. Large benchmarks and whole applications
    863 go here.

    864
    865
  • test-suite/External
  • 866

    The External directory contains Makefiles for building code that is external

    867 to (i.e., not distributed with) LLVM. The most prominent members of this
    868 directory are the SPEC 95 and SPEC 2000 benchmark suites. The External
    869 directory does not contain these actual tests, but only the Makefiles that know
    870 how to properly compile these programs from somewhere else. When
    871 using LNT, use the --test-externals option to include these
    872 tests in the results.

    873
    874
    875
    876
    877

    test-suite Quickstart

    878
    879
    880
    881

    The modern way of running the test-suite is focused on testing and

    882 benchmarking complete compilers using
    883 the LNT testing infrastructure.

    884
    885

    For more information on using LNT to execute the test-suite, please

    886 see the LNT Quickstart
    887 documentation.

    888
    889
    890
    891

    test-suite Makefiles

    892
    893
    894
    895

    Historically, the test-suite was executed using a complicated setup

    896 of Makefiles. The LNT based approach above is recommended for most users, but
    897 there are some testing scenarios which are not supported by the LNT approach. In
    898 addition, LNT currently uses the Makefile setup under the covers and so
    899 developers who are interested in how LNT works under the hood may want to
    900 understand the Makefile based setup.

    901
    902

    For more information on the test-suite Makefile setup, please see

    903 the Test Suite Makefile Guide.

    904
    905
    906
    907
    908
    909
    910
    911 src="http://jigsaw.w3.org/css-validator/images/vcss-blue" alt="Valid CSS">
    912
    913 src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01">
    914
    915 John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner
    916 The LLVM Compiler Infrastructure
    917 Last modified: $Date$
    918
    919
    920
    0 =================================
    1 LLVM Testing Infrastructure Guide
    2 =================================
    3
    4 Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya
    5 Lattner
    6
    7 .. contents::
    8 :local:
    9
    10 Overview
    11 ========
    12
    13 This document is the reference manual for the LLVM testing
    14 infrastructure. It documents the structure of the LLVM testing
    15 infrastructure, the tools needed to use it, and how to add and run
    16 tests.
    17
    18 Requirements
    19 ============
    20
    21 In order to use the LLVM testing infrastructure, you will need all of
    22 the software required to build LLVM, as well as
    23 `Python `_ 2.4 or later.
    24
    25 LLVM testing infrastructure organization
    26 ========================================
    27
    28 The LLVM testing infrastructure contains two major categories of tests:
    29 regression tests and whole programs. The regression tests are contained
    30 inside the LLVM repository itself under ``llvm/test`` and are expected
    31 to always pass -- they should be run before every commit.
    32
    33 The whole programs tests are referred to as the "LLVM test suite" (or
    34 "test-suite") and are in the ``test-suite`` module in subversion. For
    35 historical reasons, these tests are also referred to as the "nightly
    36 tests" in places, which is less ambiguous than "test-suite" and remains
    37 in use although we run them much more often than nightly.
    38
    39 Regression tests
    40 ----------------
    41
    42 The regression tests are small pieces of code that test a specific
    43 feature of LLVM or trigger a specific bug in LLVM. They are usually
    44 written in LLVM assembly language, but can be written in other languages
    45 if the test targets a particular language front end (and the appropriate
    46 ``--with-llvmgcc`` options were used at ``configure`` time of the
    47 ``llvm`` module). These tests are driven by the 'lit' testing tool,
    48 which is part of LLVM.
    49
    50 These code fragments are not complete programs. The code generated from
    51 them is never executed to determine correct behavior.
    52
    53 These code fragment tests are located in the ``llvm/test`` directory.
    54
    55 Typically when a bug is found in LLVM, a regression test containing just
    56 enough code to reproduce the problem should be written and placed
    57 somewhere underneath this directory. In most cases, this will be a small
    58 piece of LLVM assembly language code, often distilled from an actual
    59 application or benchmark.
    60
    61 ``test-suite``
    62 --------------
    63
    64 The test suite contains whole programs, which are pieces of code which
    65 can be compiled and linked into a stand-alone program that can be
    66 executed. These programs are generally written in high level languages
    67 such as C or C++.
    68
    69 These programs are compiled using a user specified compiler and set of
    70 flags, and then executed to capture the program output and timing
    71 information. The output of these programs is compared to a reference
    72 output to ensure that the program is being compiled correctly.
    73
    74 In addition to compiling and executing programs, whole program tests
    75 serve as a way of benchmarking LLVM performance, both in terms of the
    76 efficiency of the programs generated as well as the speed with which
    77 LLVM compiles, optimizes, and generates code.
    78
    79 The test-suite is located in the ``test-suite`` Subversion module.
    80
    81 Debugging Information tests
    82 ---------------------------
    83
    84 The test suite contains tests to check quality of debugging information.
    85 The test are written in C based languages or in LLVM assembly language.
    86
    87 These tests are compiled and run under a debugger. The debugger output
    88 is checked to validate of debugging information. See README.txt in the
    89 test suite for more information . This test suite is located in the
    90 ``debuginfo-tests`` Subversion module.
    91
    92 Quick start
    93 ===========
    94
    95 The tests are located in two separate Subversion modules. The
    96 regressions tests are in the main "llvm" module under the directory
    97 ``llvm/test`` (so you get these tests for free with the main llvm tree).
    98 Use "make check-all" to run the regression tests after building LLVM.
    99
    100 The more comprehensive test suite that includes whole programs in C and
    101 C++ is in the ``test-suite`` module. See ```test-suite``
    102 Quickstart <#testsuitequickstart>`_ for more information on running
    103 these tests.
    104
    105 Regression tests
    106 ----------------
    107
    108 To run all of the LLVM regression tests, use master Makefile in the
    109 ``llvm/test`` directory:
    110
    111 .. code-block:: bash
    112
    113 % gmake -C llvm/test
    114
    115 or
    116
    117 .. code-block:: bash
    118
    119 % gmake check
    120
    121 If you have `Clang `_ checked out and built, you
    122 can run the LLVM and Clang tests simultaneously using:
    123
    124 or
    125
    126 .. code-block:: bash
    127
    128 % gmake check-all
    129
    130 To run the tests with Valgrind (Memcheck by default), just append
    131 ``VG=1`` to the commands above, e.g.:
    132
    133 .. code-block:: bash
    134
    135 % gmake check VG=1
    136
    137 To run individual tests or subsets of tests, you can use the 'llvm-lit'
    138 script which is built as part of LLVM. For example, to run the
    139 'Integer/BitPacked.ll' test by itself you can run:
    140
    141 .. code-block:: bash
    142
    143 % llvm-lit ~/llvm/test/Integer/BitPacked.ll
    144
    145 or to run all of the ARM CodeGen tests:
    146
    147 .. code-block:: bash
    148
    149 % llvm-lit ~/llvm/test/CodeGen/ARM
    150
    151 For more information on using the 'lit' tool, see 'llvm-lit --help' or
    152 the 'lit' man page.
    153
    154 Debugging Information tests
    155 ---------------------------
    156
    157 To run debugging information tests simply checkout the tests inside
    158 clang/test directory.
    159
    160 .. code-block:: bash
    161
    162 % cd clang/test
    163 % svn co http://llvm.org/svn/llvm-project/debuginfo-tests/trunk debuginfo-tests
    164
    165 These tests are already set up to run as part of clang regression tests.
    166
    167 Regression test structure
    168 =========================
    169
    170 The LLVM regression tests are driven by 'lit' and are located in the
    171 ``llvm/test`` directory.
    172
    173 This directory contains a large array of small tests that exercise
    174 various features of LLVM and to ensure that regressions do not occur.
    175 The directory is broken into several sub-directories, each focused on a
    176 particular area of LLVM. A few of the important ones are:
    177
    178 - ``Analysis``: checks Analysis passes.
    179 - ``Archive``: checks the Archive library.
    180 - ``Assembler``: checks Assembly reader/writer functionality.
    181 - ``Bitcode``: checks Bitcode reader/writer functionality.
    182 - ``CodeGen``: checks code generation and each target.
    183 - ``Features``: checks various features of the LLVM language.
    184 - ``Linker``: tests bitcode linking.
    185 - ``Transforms``: tests each of the scalar, IPO, and utility transforms
    186 to ensure they make the right transformations.
    187 - ``Verifier``: tests the IR verifier.
    188
    189 Writing new regression tests
    190 ----------------------------
    191
    192 The regression test structure is very simple, but does require some
    193 information to be set. This information is gathered via ``configure``
    194 and is written to a file, ``lit.site.cfg`` in ``llvm/test``. The
    195 ``llvm/test`` Makefile does this work for you.
    196
    197 In order for the regression tests to work, each directory of tests must
    198 have a ``lit.local.cfg`` file. Lit looks for this file to determine how
    199 to run the tests. This file is just Python code and thus is very
    200 flexible, but we've standardized it for the LLVM regression tests. If
    201 you're adding a directory of tests, just copy ``lit.local.cfg`` from
    202 another directory to get running. The standard ``lit.local.cfg`` simply
    203 specifies which files to look in for tests. Any directory that contains
    204 only directories does not need the ``lit.local.cfg`` file. Read the `Lit
    205 documentation `_ for more information.
    206
    207 The ``llvm-runtests`` function looks at each file that is passed to it
    208 and gathers any lines together that match "RUN:". These are the "RUN"
    209 lines that specify how the test is to be run. So, each test script must
    210 contain RUN lines if it is to do anything. If there are no RUN lines,
    211 the ``llvm-runtests`` function will issue an error and the test will
    212 fail.
    213
    214 RUN lines are specified in the comments of the test program using the
    215 keyword ``RUN`` followed by a colon, and lastly the command (pipeline)
    216 to execute. Together, these lines form the "script" that
    217 ``llvm-runtests`` executes to run the test case. The syntax of the RUN
    218 lines is similar to a shell's syntax for pipelines including I/O
    219 redirection and variable substitution. However, even though these lines
    220 may *look* like a shell script, they are not. RUN lines are interpreted
    221 directly by the Tcl ``exec`` command. They are never executed by a
    222 shell. Consequently the syntax differs from normal shell script syntax
    223 in a few ways. You can specify as many RUN lines as needed.
    224
    225 lit performs substitution on each RUN line to replace LLVM tool names
    226 with the full paths to the executable built for each tool (in
    227 $(LLVM\_OBJ\_ROOT)/$(BuildMode)/bin). This ensures that lit does not
    228 invoke any stray LLVM tools in the user's path during testing.
    229
    230 Each RUN line is executed on its own, distinct from other lines unless
    231 its last character is ``\``. This continuation character causes the RUN
    232 line to be concatenated with the next one. In this way you can build up
    233 long pipelines of commands without making huge line lengths. The lines
    234 ending in ``\`` are concatenated until a RUN line that doesn't end in
    235 ``\`` is found. This concatenated set of RUN lines then constitutes one
    236 execution. Tcl will substitute variables and arrange for the pipeline to
    237 be executed. If any process in the pipeline fails, the entire line (and
    238 test case) fails too.
    239
    240 Below is an example of legal RUN lines in a ``.ll`` file:
    241
    242 .. code-block:: llvm
    243
    244 ; RUN: llvm-as < %s | llvm-dis > %t1
    245 ; RUN: llvm-dis < %s.bc-13 > %t2
    246 ; RUN: diff %t1 %t2
    247
    248 As with a Unix shell, the RUN: lines permit pipelines and I/O
    249 redirection to be used. However, the usage is slightly different than
    250 for Bash. To check what's legal, see the documentation for the `Tcl
    251 exec `_ command and the
    252 `tutorial `_. The
    253 major differences are:
    254
    255 - You can't do ``2>&1``. That will cause Tcl to write to a file named
    256 ``&1``. Usually this is done to get stderr to go through a pipe. You
    257 can do that in tcl with ``|&`` so replace this idiom:
    258 ``... 2>&1 | grep`` with ``... |& grep``
    259 - You can only redirect to a file, not to another descriptor and not
    260 from a here document.
    261 - tcl supports redirecting to open files with the @ syntax but you
    262 shouldn't use that here.
    263
    264 There are some quoting rules that you must pay attention to when writing
    265 your RUN lines. In general nothing needs to be quoted. Tcl won't strip
    266 off any quote characters so they will get passed to the invoked program.
    267 For example:
    268
    269 .. code-block:: bash
    270
    271 ... | grep 'find this string'
    272
    273 This will fail because the ' characters are passed to grep. This would
    274 instruction grep to look for ``'find`` in the files ``this`` and
    275 ``string'``. To avoid this use curly braces to tell Tcl that it should
    276 treat everything enclosed as one value. So our example would become:
    277
    278 .. code-block:: bash
    279
    280 ... | grep {find this string}
    281
    282 Additionally, the characters ``[`` and ``]`` are treated specially by
    283 Tcl. They tell Tcl to interpret the content as a command to execute.
    284 Since these characters are often used in regular expressions this can
    285 have disastrous results and cause the entire test run in a directory to
    286 fail. For example, a common idiom is to look for some basicblock number:
    287
    288 .. code-block:: bash
    289
    290 ... | grep bb[2-8]
    291
    292 This, however, will cause Tcl to fail because its going to try to
    293 execute a program named "2-8". Instead, what you want is this:
    294
    295 .. code-block:: bash
    296
    297 ... | grep {bb\[2-8\]}
    298
    299 Finally, if you need to pass the ``\`` character down to a program, then
    300 it must be doubled. This is another Tcl special character. So, suppose
    301 you had:
    302
    303 .. code-block:: bash
    304
    305 ... | grep 'i32\*'
    306
    307 This will fail to match what you want (a pointer to i32). First, the
    308 ``'`` do not get stripped off. Second, the ``\`` gets stripped off by
    309 Tcl so what grep sees is: ``'i32*'``. That's not likely to match
    310 anything. To resolve this you must use ``\\`` and the ``{}``, like this:
    311
    312 .. code-block:: bash
    313
    314 ... | grep {i32\\*}
    315
    316 If your system includes GNU ``grep``, make sure that ``GREP_OPTIONS`` is
    317 not set in your environment. Otherwise, you may get invalid results
    318 (both false positives and false negatives).
    319
    320 The FileCheck utility
    321 ---------------------
    322
    323 A powerful feature of the RUN: lines is that it allows any arbitrary
    324 commands to be executed as part of the test harness. While standard
    325 (portable) unix tools like 'grep' work fine on run lines, as you see
    326 above, there are a lot of caveats due to interaction with Tcl syntax,
    327 and we want to make sure the run lines are portable to a wide range of
    328 systems. Another major problem is that grep is not very good at checking
    329 to verify that the output of a tools contains a series of different
    330 output in a specific order. The FileCheck tool was designed to help with
    331 these problems.
    332
    333 FileCheck (whose basic command line arguments are described in `the
    334 FileCheck man page `_ is designed
    335 to read a file to check from standard input, and the set of things to
    336 verify from a file specified as a command line argument. A simple
    337 example of using FileCheck from a RUN line looks like this:
    338
    339 .. code-block:: llvm
    340
    341 ; RUN: llvm-as < %s | llc -march=x86-64 | FileCheck %s
    342
    343 This syntax says to pipe the current file ("%s") into llvm-as, pipe that
    344 into llc, then pipe the output of llc into FileCheck. This means that
    345 FileCheck will be verifying its standard input (the llc output) against
    346 the filename argument specified (the original .ll file specified by
    347 "%s"). To see how this works, let's look at the rest of the .ll file
    348 (after the RUN line):
    349
    350 .. code-block:: llvm
    351
    352 define void @sub1(i32* %p, i32 %v) {
    353 entry:
    354 ; CHECK: sub1:
    355 ; CHECK: subl
    356 %0 = tail call i32 @llvm.atomic.load.sub.i32.p0i32(i32* %p, i32 %v)
    357 ret void
    358 }
    359
    360 define void @inc4(i64* %p) {
    361 entry:
    362 ; CHECK: inc4:
    363 ; CHECK: incq
    364 %0 = tail call i64 @llvm.atomic.load.add.i64.p0i64(i64* %p, i64 1)
    365 ret void
    366 }
    367
    368 Here you can see some "CHECK:" lines specified in comments. Now you can
    369 see how the file is piped into llvm-as, then llc, and the machine code
    370 output is what we are verifying. FileCheck checks the machine code
    371 output to verify that it matches what the "CHECK:" lines specify.
    372
    373 The syntax of the CHECK: lines is very simple: they are fixed strings
    374 that must occur in order. FileCheck defaults to ignoring horizontal
    375 whitespace differences (e.g. a space is allowed to match a tab) but
    376 otherwise, the contents of the CHECK: line is required to match some
    377 thing in the test file exactly.
    378
    379 One nice thing about FileCheck (compared to grep) is that it allows
    380 merging test cases together into logical groups. For example, because
    381 the test above is checking for the "sub1:" and "inc4:" labels, it will
    382 not match unless there is a "subl" in between those labels. If it
    383 existed somewhere else in the file, that would not count: "grep subl"
    384 matches if subl exists anywhere in the file.
    385
    386 The FileCheck -check-prefix option
    387 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    388
    389 The FileCheck -check-prefix option allows multiple test configurations
    390 to be driven from one .ll file. This is useful in many circumstances,
    391 for example, testing different architectural variants with llc. Here's a
    392 simple example:
    393
    394 .. code-block:: llvm
    395
    396 ; RUN: llvm-as < %s | llc -mtriple=i686-apple-darwin9 -mattr=sse41 \
    397 ; RUN: | FileCheck %s -check-prefix=X32
    398 ; RUN: llvm-as < %s | llc -mtriple=x86_64-apple-darwin9 -mattr=sse41 \
    399 ; RUN: | FileCheck %s -check-prefix=X64
    400
    401 define <4 x i32> @pinsrd_1(i32 %s, <4 x i32> %tmp) nounwind {
    402 %tmp1 = insertelement <4 x i32> %tmp, i32 %s, i32 1
    403 ret <4 x i32> %tmp1
    404 ; X32: pinsrd_1:
    405 ; X32: pinsrd $1, 4(%esp), %xmm0
    406
    407 ; X64: pinsrd_1:
    408 ; X64: pinsrd $1, %edi, %xmm0
    409 }
    410
    411 In this case, we're testing that we get the expected code generation
    412 with both 32-bit and 64-bit code generation.
    413
    414 The "CHECK-NEXT:" directive
    415 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    416
    417 Sometimes you want to match lines and would like to verify that matches
    418 happen on exactly consecutive lines with no other lines in between them.
    419 In this case, you can use CHECK: and CHECK-NEXT: directives to specify
    420 this. If you specified a custom check prefix, just use "-NEXT:".
    421 For example, something like this works as you'd expect:
    422
    423 .. code-block:: llvm
    424
    425 define void @t2(<2 x double>* %r, <2 x double>* %A, double %B) {
    426 %tmp3 = load <2 x double>* %A, align 16
    427 %tmp7 = insertelement <2 x double> undef, double %B, i32 0
    428 %tmp9 = shufflevector <2 x double> %tmp3,
    429 <2 x double> %tmp7,
    430 <2 x i32> < i32 0, i32 2 >
    431 store <2 x double> %tmp9, <2 x double>* %r, align 16
    432 ret void
    433
    434 ; CHECK: t2:
    435 ; CHECK: movl 8(%esp), %eax
    436 ; CHECK-NEXT: movapd (%eax), %xmm0
    437 ; CHECK-NEXT: movhpd 12(%esp), %xmm0
    438 ; CHECK-NEXT: movl 4(%esp), %eax
    439 ; CHECK-NEXT: movapd %xmm0, (%eax)
    440 ; CHECK-NEXT: ret
    441 }
    442
    443 CHECK-NEXT: directives reject the input unless there is exactly one
    444 newline between it an the previous directive. A CHECK-NEXT cannot be the
    445 first directive in a file.
    446
    447 The "CHECK-NOT:" directive
    448 ^^^^^^^^^^^^^^^^^^^^^^^^^^
    449
    450 The CHECK-NOT: directive is used to verify that a string doesn't occur
    451 between two matches (or the first match and the beginning of the file).
    452 For example, to verify that a load is removed by a transformation, a
    453 test like this can be used:
    454
    455 .. code-block:: llvm
    456
    457 define i8 @coerce_offset0(i32 %V, i32* %P) {
    458 store i32 %V, i32* %P
    459
    460 %P2 = bitcast i32* %P to i8*
    461 %P3 = getelementptr i8* %P2, i32 2
    462
    463 %A = load i8* %P3
    464 ret i8 %A
    465 ; CHECK: @coerce_offset0
    466 ; CHECK-NOT: load
    467 ; CHECK: ret i8
    468 }
    469
    470 FileCheck Pattern Matching Syntax
    471 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    472
    473 The CHECK: and CHECK-NOT: directives both take a pattern to match. For
    474 most uses of FileCheck, fixed string matching is perfectly sufficient.
    475 For some things, a more flexible form of matching is desired. To support
    476 this, FileCheck allows you to specify regular expressions in matching
    477 strings, surrounded by double braces: **{{yourregex}}**. Because we want
    478 to use fixed string matching for a majority of what we do, FileCheck has
    479 been designed to support mixing and matching fixed string matching with
    480 regular expressions. This allows you to write things like this:
    481
    482 .. code-block:: llvm
    483
    484 ; CHECK: movhpd {{[0-9]+}}(%esp), {{%xmm[0-7]}}
    485
    486 In this case, any offset from the ESP register will be allowed, and any
    487 xmm register will be allowed.
    488
    489 Because regular expressions are enclosed with double braces, they are
    490 visually distinct, and you don't need to use escape characters within
    491 the double braces like you would in C. In the rare case that you want to
    492 match double braces explicitly from the input, you can use something
    493 ugly like **{{[{][{]}}** as your pattern.
    494
    495 FileCheck Variables
    496 ^^^^^^^^^^^^^^^^^^^
    497
    498 It is often useful to match a pattern and then verify that it occurs
    499 again later in the file. For codegen tests, this can be useful to allow
    500 any register, but verify that that register is used consistently later.
    501 To do this, FileCheck allows named variables to be defined and
    502 substituted into patterns. Here is a simple example:
    503
    504 .. code-block:: llvm
    505
    506 ; CHECK: test5:
    507 ; CHECK: notw [[REGISTER:%[a-z]+]]
    508 ; CHECK: andw {{.*}}[[REGISTER]]
    509
    510 The first check line matches a regex (``%[a-z]+``) and captures it into
    511 the variables "REGISTER". The second line verifies that whatever is in
    512 REGISTER occurs later in the file after an "andw". FileCheck variable
    513 references are always contained in ``[[ ]]`` pairs, are named, and their
    514 names can be formed with the regex "``[a-zA-Z][a-zA-Z0-9]*``". If a
    515 colon follows the name, then it is a definition of the variable, if not,
    516 it is a use.
    517
    518 FileCheck variables can be defined multiple times, and uses always get
    519 the latest value. Note that variables are all read at the start of a
    520 "CHECK" line and are all defined at the end. This means that if you have
    521 something like "``CHECK: [[XYZ:.*]]x[[XYZ]]``" that the check line will
    522 read the previous value of the XYZ variable and define a new one after
    523 the match is performed. If you need to do something like this you can
    524 probably take advantage of the fact that FileCheck is not actually
    525 line-oriented when it matches, this allows you to define two separate
    526 CHECK lines that match on the same line.
    527
    528 Variables and substitutions
    529 ---------------------------
    530
    531 With a RUN line there are a number of substitutions that are permitted.
    532 In general, any Tcl variable that is available in the ``substitute``
    533 function (in ``test/lib/llvm.exp``) can be substituted into a RUN line.
    534 To make a substitution just write the variable's name preceded by a $.
    535 Additionally, for compatibility reasons with previous versions of the
    536 test library, certain names can be accessed with an alternate syntax: a
    537 % prefix. These alternates are deprecated and may go away in a future
    538 version.
    539
    540 Here are the available variable names. The alternate syntax is listed in
    541 parentheses.
    542
    543 ``$test`` (``%s``)
    544 The full path to the test case's source. This is suitable for passing on
    545 the command line as the input to an llvm tool.
    546
    547 ``%(line)``, ``%(line+)``, ``%(line-)``
    548 The number of the line where this variable is used, with an optional
    549 integer offset. This can be used in tests with multiple RUN: lines,
    550 which reference test file's line numbers.
    551
    552 ``$srcdir``
    553 The source directory from where the "``make check``" was run.
    554
    555 ``objdir``
    556 The object directory that corresponds to the ``$srcdir``.
    557
    558 ``subdir``
    559 A partial path from the ``test`` directory that contains the
    560 sub-directory that contains the test source being executed.
    561
    562 ``srcroot``
    563 The root directory of the LLVM src tree.
    564
    565 ``objroot``
    566 The root directory of the LLVM object tree. This could be the same as
    567 the srcroot.
    568
    569 ``path``
    570 The path to the directory that contains the test case source. This is
    571 for locating any supporting files that are not generated by the test,
    572 but used by the test.
    573
    574 ``tmp``
    575 The path to a temporary file name that could be used for this test case.
    576 The file name won't conflict with other test cases. You can append to it
    577 if you need multiple temporaries. This is useful as the destination of
    578 some redirected output.
    579
    580 ``target_triplet`` (``%target_triplet``)
    581 The target triplet that corresponds to the current host machine (the one
    582 running the test cases). This should probably be called "host".
    583
    584 ``link`` (``%link``)
    585 This full link command used to link LLVM executables. This has all the
    586 configured -I, -L and -l options.
    587
    588 ``shlibext`` (``%shlibext``)
    589 The suffix for the host platforms share library (dll) files. This
    590 includes the period as the first character.
    591
    592 To add more variables, two things need to be changed. First, add a line
    593 in the ``test/Makefile`` that creates the ``site.exp`` file. This will
    594 "set" the variable as a global in the site.exp file. Second, in the
    595 ``test/lib/llvm.exp`` file, in the substitute proc, add the variable
    596 name to the list of "global" declarations at the beginning of the proc.
    597 That's it, the variable can then be used in test scripts.
    598
    599 Other Features
    600 --------------
    601
    602 To make RUN line writing easier, there are several shell scripts located
    603 in the ``llvm/test/Scripts`` directory. This directory is in the PATH
    604 when running tests, so you can just call these scripts using their name.
    605 For example:
    606
    607 ``ignore``
    608 This script runs its arguments and then always returns 0. This is useful
    609 in cases where the test needs to cause a tool to generate an error (e.g.
    610 to check the error output). However, any program in a pipeline that
    611 returns a non-zero result will cause the test to fail. This script
    612 overcomes that issue and nicely documents that the test case is
    613 purposefully ignoring the result code of the tool
    614 ``not``
    615 This script runs its arguments and then inverts the result code from it.
    616 Zero result codes become 1. Non-zero result codes become 0. This is
    617 useful to invert the result of a grep. For example "not grep X" means
    618 succeed only if you don't find X in the input.
    619
    620 Sometimes it is necessary to mark a test case as "expected fail" or
    621 XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
    622 on a line near the top of the file. This signals that the test case
    623 should succeed if the test fails. Such test cases are counted separately
    624 by the testing tool. To specify an expected fail, use the XFAIL keyword
    625 in the comments of the test program followed by a colon and one or more
    626 failure patterns. Each failure pattern can be either ``*`` (to specify
    627 fail everywhere), or a part of a target triple (indicating the test
    628 should fail on that platform), or the name of a configurable feature
    629 (for example, ``loadable_module``). If there is a match, the test is
    630 expected to fail. If not, the test is expected to succeed. To XFAIL
    631 everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
    632 line:
    633
    634 .. code-block:: llvm
    635
    636 ; XFAIL: darwin,sun
    637
    638 To make the output more useful, the ``llvm_runtest`` function wil scan
    639 the lines of the test case for ones that contain a pattern that matches
    640 ``PR[0-9]+``. This is the syntax for specifying a PR (Problem Report) number
    641 that is related to the test case. The number after "PR" specifies the
    642 LLVM bugzilla number. When a PR number is specified, it will be used in
    643 the pass/fail reporting. This is useful to quickly get some context when
    644 a test fails.
    645
    646 Finally, any line that contains "END." will cause the special
    647 interpretation of lines to terminate. This is generally done right after
    648 the last RUN: line. This has two side effects:
    649
    650 (a) it prevents special interpretation of lines that are part of the test
    651 program, not the instructions to the test case, and
    652
    653 (b) it speeds things up for really big test cases by avoiding
    654 interpretation of the remainder of the file.
    655
    656 ``test-suite`` Overview
    657 =======================
    658
    659 The ``test-suite`` module contains a number of programs that can be
    660 compiled and executed. The ``test-suite`` includes reference outputs for
    661 all of the programs, so that the output of the executed program can be
    662 checked for correctness.
    663
    664 ``test-suite`` tests are divided into three types of tests: MultiSource,
    665 SingleSource, and External.
    666
    667 - ``test-suite/SingleSource``
    668
    669 The SingleSource directory contains test programs that are only a
    670 single source file in size. These are usually small benchmark
    671 programs or small programs that calculate a particular value. Several
    672 such programs are grouped together in each directory.
    673
    674 - ``test-suite/MultiSource``
    675
    676 The MultiSource directory contains subdirectories which contain
    677 entire programs with multiple source files. Large benchmarks and
    678 whole applications go here.
    679
    680 - ``test-suite/External``
    681
    682 The External directory contains Makefiles for building code that is
    683 external to (i.e., not distributed with) LLVM. The most prominent
    684 members of this directory are the SPEC 95 and SPEC 2000 benchmark
    685 suites. The ``External`` directory does not contain these actual
    686 tests, but only the Makefiles that know how to properly compile these
    687 programs from somewhere else. When using ``LNT``, use the
    688 ``--test-externals`` option to include these tests in the results.
    689
    690 ``test-suite`` Quickstart
    691 -------------------------
    692
    693 The modern way of running the ``test-suite`` is focused on testing and
    694 benchmarking complete compilers using the
    695 `LNT `_ testing infrastructure.
    696
    697 For more information on using LNT to execute the ``test-suite``, please
    698 see the `LNT Quickstart `_
    699 documentation.
    700
    701 ``test-suite`` Makefiles
    702 ------------------------
    703
    704 Historically, the ``test-suite`` was executed using a complicated setup
    705 of Makefiles. The LNT based approach above is recommended for most
    706 users, but there are some testing scenarios which are not supported by
    707 the LNT approach. In addition, LNT currently uses the Makefile setup
    708 under the covers and so developers who are interested in how LNT works
    709 under the hood may want to understand the Makefile based setup.
    710
    711 For more information on the ``test-suite`` Makefile setup, please see
    712 the `Test Suite Makefile Guide. `_
    1919 HowToSubmitABug
    2020 SphinxQuickstartTemplate
    2121 Phabricator
    22 TestingGuide
    2223
    2324 * :ref:`getting_started`
    2425
    7677 A template + tutorial for writing new Sphinx documentation. It is meant
    7778 to be read in source form.
    7879
    79 * `LLVM Testing Infrastructure Guide `_
    80 * :doc:`LLVM Testing Infrastructure Guide `
    8081
    8182 A reference manual for using the LLVM testing infrastructure.
    8283