llvm.org GIT mirror llvm / cd3b117
docs/TestingGuide: Minimal update to describe 'lit' based regression testing instead of DejaGNU. Still a bit kooky, since the current test format still has some strong Tcl roots. Oh well! git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@110005 91177308-0d34-0410-b5e6-96231b3b80d8 Daniel Dunbar 9 years ago
1 changed file(s) with 97 addition(s) and 95 deletion(s). Raw diff Collapse all Expand all
1515
  • Requirements
  • 1616
  • LLVM testing infrastructure organization
  • 1717
    18
  • DejaGNU tests
  • 18
  • Regression tests
  • 1919
  • Test suite
  • 2020
    2121
    2222
  • Quick start
  • 2323
    24
  • DejaGNU tests
  • 24
  • Regression tests
  • 2525
  • Test suite
  • 2626
    2727
    28
  • DejaGNU structure
  • 28
  • Regression test structure
  • 2929
    30
  • Writing new DejaGNU tests
  • 30
  • Writing new regression tests
  • 3131
  • The FileCheck utility
  • 32
  • Variables and substitutions
  • 33
  • Other features
  • 32
  • Variables and substitutions
  • 33
  • Other features
  • 3434
    3535
    3636
  • Test suite structure
  • 4545
    4646
    4747
    48

    Written by John T. Criswell,

    49 href="http://llvm.x10sys.com/rspencer">Reid Spencer, and Tanya Lattner

    48

    Written by John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner

    5049
    5150
    5251
    5554
    5655
    5756
    58

    This document is the reference manual for the LLVM testing infrastructure. It documents

    59 the structure of the LLVM testing infrastructure, the tools needed to use it,
    60 and how to add and run tests.

    57

    This document is the reference manual for the LLVM testing infrastructure. It

    58 documents the structure of the LLVM testing infrastructure, the tools needed to
    59 use it, and how to add and run tests.

    6160
    6261
    6362
    6766
    6867
    6968
    70

    In order to use the LLVM testing infrastructure, you will need all of the software

    71 required to build LLVM, plus the following:

    72
    73
    74
    DejaGNU
    75
    The Feature and Regressions tests are organized and run by DejaGNU.
    76
    Expect
    77
    Expect is required by DejaGNU.
    78
    tcl
    79
    Tcl is required by DejaGNU.
    80 </dl>
    69 <p>In order to use the LLVM testing infrastructure, you will need all of the
    70 software required to build LLVM, as well
    71 as Python 2.4 or later.

    8172
    8273
    8374
    8778
    8879
    8980
    90

    The LLVM testing infrastructure contains two major categories of tests: code

    91 fragments and whole programs. Code fragments are referred to as the "DejaGNU
    92 tests" and are in the llvm module in subversion under the
    93 llvm/test directory. The whole programs tests are referred to as the
    94 "Test suite" and are in the test-suite module in subversion.
    81

    The LLVM testing infrastructure contains two major categories of tests:

    82 regression tests and whole programs. The regression tests are contained inside
    83 the LLVM repository itself under llvm/test and are expected to always
    84 pass -- they should be run before every commit. The whole programs tests are
    85 referred to as the "LLVM test suite" and are in the test-suite module
    86 in subversion.
    9587

    9688
    9789
    9890
    9991
    100
    101
    102
    103
    104
    105

    Code fragments are small pieces of code that test a specific

    106 feature of LLVM or trigger a specific bug in LLVM. They are usually
    107 written in LLVM assembly language, but can be written in other
    108 languages if the test targets a particular language front end (and the
    109 appropriate --with-llvmgcc options were used
    110 at configure time of the llvm module). These tests
    111 are driven by the DejaGNU testing framework, which is hidden behind a
    112 few simple makefiles.>
    92
    93
    94
    95
    96
    97

    The regression tests are small pieces of code that test a specific feature of

    98 LLVM or trigger a specific bug in LLVM. They are usually written in LLVM
    99 assembly language, but can be written in other languages if the test targets a
    100 particular language front end (and the appropriate --with-llvmgcc
    101 options were used at configure time of the llvm module). These
    102 tests are driven by the 'lit' testing tool, which is part of LLVM.

    113103
    114104

    These code fragments are not complete programs. The code generated

    115105 from them is never executed to determine correct behavior.

    156146
    157147
    158148
    159

    The tests are located in two separate Subversion modules. The

    160 DejaGNU tests are in the main "llvm" module under the directory
    149

    The tests are located in two separate Subversion modules. The regressions

    150 tests are in the main "llvm" module under the directory
    161151 llvm/test (so you get these tests for free with the main llvm tree).
    162152 The more comprehensive test suite that includes whole
    163153 programs in C and C++ is in the test-suite module. This module should
    169159 Alternatively, you can configure the test-suite module manually.

    170160
    171161
    172
    173
    174

    To run all of the simple tests in LLVM using DejaGNU, use the master Makefile

    175 in the llvm/test directory:>
    162
    163
    164

    To run all of the LLVM regression tests, use master Makefile in

    165 the llvm/test directory:

    176166
    177167
    178168
    
                      
                    
    188178
    189179
    190180
    191

    To run only a subdirectory of tests in llvm/test using DejaGNU (ie.

    192 Transforms), just set the TESTSUITE variable to the path of the
    193 subdirectory (relative to llvm/test):

    194
    195
    196
    
                      
                    
    197 % gmake TESTSUITE=Transforms check
    198
    199
    200
    201

    Note: If you are running the tests with objdir != subdir, you

    202 must have run the complete testsuite before you can specify a
    203 subdirectory.

    204
    205

    To run only a single test, set TESTONE to its path (relative to

    206 llvm/test) and make the check-one target:

    207
    208
    209
    
                      
                    
    210 % gmake TESTONE=Feature/basictest.ll check-one
    181

    If you have Clang checked out and built,

    182 you can run the LLVM and Clang tests simultaneously using:

    183
    184

    or

    185
    186
    187
    
                      
                    
    188 % gmake check-all
    211189
    212190
    213191
    219197 % gmake check VG=1
    220198
    221199
    200
    201

    To run individual tests or subsets of tests, you can use the 'llvm-lit'

    202 script which is built as part of LLVM. For example, to run the
    203 'Integer/BitCast.ll' test by itself you can run:

    204
    205
    206
    
                      
                    
    207 % llvm-lit ~/llvm/test/Integer/BitCast.ll
    208
    209
    210
    211

    or to run all of the ARM CodeGen tests:

    212
    213
    214
    
                      
                    
    215 % llvm-lit ~/llvm/test/CodeGen/ARM
    216
    217
    218
    219

    For more information on using the 'lit' tool, see 'llvm-lit --help' or the

    220 'lit' man page.

    222221
    223222
    224223
    273272
    274273
    275274
    276
    277
    278
    279

    The LLVM DejaGNU tests are driven by DejaGNU together with GNU Make and are

    280 located in the llvm/test directory.
    275
    276
    277
    278

    The LLVM regression tests are driven by 'lit' and are located in

    279 the llvm/test directory.
    281280
    282281

    This directory contains a large array of small tests

    283282 that exercise various features of LLVM and to ensure that regressions do not
    300299
    301300
    302301
    303
    304
    305
    306

    The DejaGNU structure is very simple, but does require some information to

    307 be set. This information is gathered via configure and is written
    308 to a file, site.exp in llvm/test. The llvm/test
    309 Makefile does this work for you.

    310
    311

    In order for DejaGNU to work, each directory of tests must have a

    312 dg.exp file. DejaGNU looks for this file to determine how to run the
    313 tests. This file is just a Tcl script and it can do anything you want, but
    314 we've standardized it for the LLVM regression tests. If you're adding a
    302
    303
    304
    305

    The regression test structure is very simple, but does require some

    306 information to be set. This information is gathered via configure and
    307 is written to a file, lit.site.cfg
    308 in llvm/test. The llvm/test Makefile does this work for
    309 you.

    310
    311

    In order for the regression tests to work, each directory of tests must

    312 have a dg.exp file. Lit looks for this file to determine how to
    313 run the tests. This file is just a Tcl script and it can do anything you want,
    314 but we've standardized it for the LLVM regression tests. If you're adding a
    315315 directory of tests, just copy dg.exp from another directory to get
    316 running. The standard dg.exp simply loads a Tcl
    317 library (test/lib/llvm.exp) and calls the llvm_runtests
    318 function defined in that library with a list of file names to run. The names
    319 are obtained by using Tcl's glob command. Any directory that contains only
    316 running. The standard dg.exp simply loads a Tcl library
    317 (test/lib/llvm.exp) and calls the llvm_runtests function
    318 defined in that library with a list of file names to run. The names are
    319 obtained by using Tcl's glob command. Any directory that contains only
    320320 directories does not need the dg.exp file.

    321321
    322322

    The llvm-runtests function lookas at each file that is passed to

    377377
    378378

    There are some quoting rules that you must pay attention to when writing

    379379 your RUN lines. In general nothing needs to be quoted. Tcl won't strip off any
    380 ' or " so they will get passed to the invoked program. For example:

    380 quote characters so they will get passed to the invoked program. For
    381 example:

    381382
    382383
    383384
    
                      
                    
    694695
    695696
    696697
    697
    698
    698699 substitutions
    699700
    700701
    790791
    791792
    792793
    793
    794
    794795
    795796
    796797

    To make RUN line writing easier, there are several shell scripts located

    816817

    Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.

    817818 You can easily mark a test as XFAIL just by including XFAIL: on a
    818819 line near the top of the file. This signals that the test case should succeed
    819 if the test fails. Such test cases are counted separately by DejaGnu. To
    820 if the test fails. Such test cases are counted separately by the testing tool. To
    820821 specify an expected fail, use the XFAIL keyword in the comments of the test
    821822 program followed by a colon and one or more regular expressions (separated by
    822823 a comma). The regular expressions allow you to XFAIL the test conditionally by
    904905 organizations should be relatively self explanatory.

    905906
    906907

    Some tests are known to fail. Some are bugs that we have not fixed yet;

    907 others are features that we haven't added yet (or may never add). In DejaGNU,
    908 the result for such tests will be XFAIL (eXpected FAILure). In this way, you
    909 can tell the difference between an expected and unexpected failure.

    908 others are features that we haven't added yet (or may never add). In the
    909 regression tests, the result for such tests will be XFAIL (eXpected FAILure).
    910 In this way, you can tell the difference between an expected and unexpected
    911 failure.

    910912
    911913

    The tests in the test suite have no such feature at this time. If the

    912914 test passes, only warnings and other miscellaneous output will be generated. If
    11421144
    11431145 src="http://www.w3.org/Icons/valid-html401-blue" alt="Valid HTML 4.01">
    11441146
    1145 John T. Criswell, Reid Spencer, and Tanya Lattner
    1147 John T. Criswell, Daniel Dunbar, Reid Spencer, and Tanya Lattner
    11461148 The LLVM Compiler Infrastructure
    11471149 Last modified: $Date$
    11481150