llvm.org GIT mirror llvm / af19f2e
Reformat. A few minor edits. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@9434 91177308-0d34-0410-b5e6-96231b3b80d8 Brian Gaeke 15 years ago
1 changed file(s) with 338 addition(s) and 262 deletion(s). Raw diff Collapse all Expand all
0
11
2
3 LLVM Test Suite Guide
4
5
6
7
8

LLVM Test Suite Guide

9
10
11

Overview

12
13
14 This document is the reference manual for the LLVM test suite. It
2
3
4
5 LLVM Test Suite Guide
6
7
8
9
10
11 LLVM Test Suite Guide
12
13
14
15
  • Overview
  • 16
  • Requirements
  • 17
  • Quick Start
  • 18
  • LLVM Test Suite Organization
  • 19
    20
  • Code Fragments
  • 21
  • Whole Programs
  • 22
    23
  • LLVM Test Suite Tree
  • 24
  • QMTest Structure
  • 25
  • Programs Structure
  • 26
  • Running the LLVM Tests
  • 27

    Written by John T. Criswell

    28
    29
    30
    31
    32
    33
    34
    35

    36 This document is the reference manual for the LLVM test suite. It
    1537 documents the structure of the LLVM test suite, the tools needed to
    1638 use it, and how to add and run tests.
    17
    18
    19

    Requirements

    20
    21
    39

    40
    41
    42
    43
    44
    45
    46
    47

    2248 In order to use the LLVM test suite, you will need all of the software
    2349 required to build LLVM, plus the following:
    50

    2451
    25
    QMTest
    26
    27 The LLVM test suite uses QMTest to organize and run tests.
    28

    29
    30
    Python
    31
    32 You will need a python interpreter that works with QMTest.
    33 Python will need zlib and SAX support enabled.
    34 <p>
    52 <dt>QMTest>
    53
    The LLVM test suite uses QMTest to organize and
    54 run tests.
    55
    56
    Python
    57
    You will need a Python interpreter that works with
    58 QMTest. Python will need zlib and SAX support
    59 enabled.
    3560
    36
    37
    38

    Quick Start

    39
    40
    41 To run all of the tests in LLVM, use the Master Makefile in llvm/test:
    61
    62
    63
    64
    65
    66
    67
    68

    69 The tests are located in the LLVM source tree under the directory
    70 llvm/test. To run all of the tests in LLVM, use the Master
    71 Makefile in that directory:
    72

    73
    
                      
                    
    74 % make -C llvm/test
    75
    76
    4277

    43
    44 cd test
    45
    46 make
    47
    48
    49

    50
    5178 To run only the code fragment tests (i.e. those that do basic testing of
    5279 LLVM), run the tests organized by QMTest:
    80

    81
    82
    
                      
                    
    83 % make -C llvm/test qmtest
    84
    85
    5386

    54
    55
    56 cd test
    57
    58 make qmtest
    59
    60
    61

    62
    6387 To run only the tests that compile and execute whole programs, run the
    6488 Programs tests:
    65

    66
    67
    68 cd test/Programs
    69
    70 make
    71
    72

    73
    74
    75

    LLVM Test Suite Organization

    76
    77
    78 The LLVM test suite contains two major types of tests:
    79
    80
  • Code Fragments>
  • 89 >
    90
    91
    
                      
                    
    92 % make -C llvm/test/Programs
    93
    94
    95
    96
    97
    98 Organization
    99
    100
    101
    102

    The LLVM test suite contains two major categories of tests: code

    103 fragments and whole programs.

    104
    105
    106
    107
    108
    109
    110

    81111 Code fragments are small pieces of code that test a specific
    82112 feature of LLVM or trigger a specific bug in LLVM. They are
    83113 usually written in LLVM assembly language, but can be
    84 written in other languages if the test targets a particular language
    85 front end.
    114 written in other languages if the test targets a
    115 particular language front end.
    116

    117 Code fragments are not complete programs, and they are
    118 never executed to determine correct behavior.
    119

    120 The tests in the Features and
    121 Regression directories contain code fragments.
    122

    123
    124
    125
    126
    127
    128
    86129

    87 Code fragments are not complete programs, and they are never executed
    88 to determine correct behavior.
    130 Whole Programs are pieces of code which can be compiled and
    131 linked into a stand-alone program that can be executed. These
    132 programs are generally written in high level languages such as C
    133 or C++, but sometimes they are written straight in LLVM
    134 assembly.
    135

    136 These programs are compiled and then executed using several
    137 different methods (native compiler, LLVM C backend, LLVM JIT,
    138 LLVM native code generation, etc). The output of these programs
    139 is compared to ensure that LLVM is compiling the program
    140 correctly.
    141

    142 In addition to compiling and executing programs, whole program
    143 tests serve as a way of benchmarking LLVM performance, both in
    144 terms of the efficiency of the programs generated as well as the
    145 speed with which LLVM compiles, optimizes, and generates code.
    146

    147 The Programs directory contains all tests which compile and
    148 benchmark whole programs.
    149

    150
    151
    152
    153
    154
    155
    156
    157
    158

    Each type of test in the LLVM test suite has its own directory. The

    159 major subtrees of the test suite directory tree are as follows:

    160
    161
    162
  • Features
  • 89163

    90 The tests in the llvm/test/Features and llvm/test/Regression directories
    91 contain code fragments.
    92
    93
  • Whole Programs
  • 94 Whole Programs are pieces of code which can be compiled and
    95 linked into a stand-alone program that can be executed. These programs
    96 are generally written in high level languages such as C or C++, but
    97 sometimes they are written straight in LLVM assembly.
    98

    99 These programs are compiled and then executed using several different
    100 methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code
    101 generation, etc). The output of these programs is compared to ensure
    102 that LLVM is compiling the program correctly.
    103
    104

    105 In addition to compiling and executing programs, whole program tests
    106 serve as a way of benchmarking LLVM performance, both in terms of the
    107 efficiency of the programs generated as well as the speed with which
    108 LLVM compiles, optimizes, and generates code.
    109
    110

    111 The test/Programs directory contains all tests which compile and
    112 benchmark whole programs.
    113
    114
    115
    116

    LLVM Test Suite Tree

    117
    118
    119 The LLVM test suite is broken up into the following directory
    120 hierarchy:
    121
    122
    123
  • Features
  • 124 This directory contains sample codes that test various features
    164 This directory contains sample codes that test various features
    125165 of the LLVM language. These pieces of sample code are run
    126166 through various assembler, disassembler, and optimizer passes.
    127

    128
    129 <li>Regression>
    167 </p>
    168
    169
  • Regression
  • 170

    130171 This directory contains regression tests for LLVM. When a bug
    131172 is found in LLVM, a regression test containing just enough
    132173 code to reproduce the problem should be written and placed
    133174 somewhere underneath this directory. In most cases, this
    134175 will be a small piece of LLVM assembly language code, often
    135176 distilled from an actual application or benchmark.
    177

    178
    179
  • Programs
  • 136180

    137
    138
  • Programs
  • 139 The Programs directory contains programs that can be compiled
    181 The Programs directory contains programs that can be compiled
    140182 with LLVM and executed. These programs are compiled using the
    141183 native compiler and various LLVM backends. The output from the
    142184 program compiled with the native compiler is assumed correct;
    143185 the results from the other programs are compared to the native
    144186 program output and pass if they match.
    145 <p>
    187 </p><p>
    146188 In addition for testing correctness, the Programs directory
    147189 also performs timing tests of various LLVM optimizations.
    148190 It also records compilation times for the compilers and the
    149191 JIT. This information can be used to compare the
    150192 effectiveness of LLVM's optimizations and code generation.
    151 <p>
    193 </p><p>
    152194 The Programs directory is subdivided into several smaller
    153195 subdirectories:
    196

    197
    154198
    155
  • SingleSource
  • 199
  • Programs/SingleSource
  • 200

    156201 The SingleSource directory contains test programs that
    157202 are only a single source file in size. These are
    158203 usually small benchmark programs or small programs that
    159 calculate a particular value. Several such programs are grouped
    160 together in each directory.
    161

    162
    163
  • MultiSource
  • 164 The MultiSource directory contains subdirectories which contain
    165 entire programs with multiple source files. Large benchmarks and
    166 whole applications go here.
    167

    168
    169
  • External
  • 204 calculate a particular value. Several such programs are
    205 grouped together in each directory.
    206

    207
    208
  • Programs/MultiSource
  • 209

    210 The MultiSource directory contains subdirectories which
    211 contain entire programs with multiple source files.
    212 Large benchmarks and whole applications go here.
    213

    214
    215
  • Programs/External
  • 216

    170217 The External directory contains Makefiles for building
    171218 code that is external to (i.e. not distributed with)
    172219 LLVM. The most prominent member of this directory is
    173 the SPEC 2000 benchmark suite. The presence and location
    174 of these external programs is configured by the LLVM
    175 configure script.
    220 the SPEC 2000 benchmark suite. The presence and
    221 location of these external programs is configured by the
    222 LLVM configure script.
    223

    176224
    177225
    178226

    179227
    180
  • QMTest
  • 181 This directory contains the QMTest information files. Inside this
    182 directory are QMTest administration files and the Python code that
    183 implements the LLVM test and database classes.
    228
  • QMTest
  • 229

    230 This directory contains the QMTest information files. Inside
    231 this directory are QMTest administration files and the Python
    232 code that implements the LLVM test and database classes.
    233

    184234
    185
    186
    187

    QMTest Structure

    188
    189
    190 The LLVM test suite is partially driven by QMTest and partially
    235
    236
    237
    238
    239
    240
    241
    242
    243

    244 The LLVM test suite is partially driven by QMTest and partially
    191245 driven by GNU Make. Specifically, the Features and Regression tests
    192246 are all driven by QMTest. The Programs directory is currently
    193247 driven by a set of Makefiles.
    194

    195
    248

    196249 The QMTest system needs to have several pieces of information
    197250 available; these pieces of configuration information are known
    198251 collectively as the "context" in QMTest parlance. Since the context
    199252 for LLVM is relatively large, the master Makefile in llvm/test
    200253 sets it for you.
    201
    202

    203
    204 The LLVM database class makes the directory tree underneath llvm/test a
    254

    255 The LLVM database class makes the subdirectories of llvm/test a
    205256 QMTest test database. For each directory that contains tests driven by
    206257 QMTest, it knows what type of test the source file is and how to run it.
    207
    208

    209
    210 Hence, the QMTest namespace is essentially what you see in
    211 llvm/test/Feature and llvm/test/Regression, but there is some magic that
    258

    259 Hence, the QMTest namespace is essentially what you see in the
    260 Feature and Regression directories, but there is some magic that
    212261 the database class performs (as described below).
    213
    214

    215
    262

    216263 The QMTest namespace is currently composed of the following tests and
    217264 test suites:
    265

    218266
    219267
    220
  • Feature
  • 221 These are the feature tests found in llvm/test/Feature. They are broken
    222 up into the following categories:
    268
  • Feature
  • 269

    270 These are the feature tests found in the Feature directory.
    271 They are broken up into the following categories:
    272

    223273
    224
  • ad
  • 225 Assembler/Disassembler tests. These tests verify that a piece of
    226 LLVM assembly language can be assembled into bytecode and then
    227 disassembled into the original assembly language code.
    228 It does this several times to ensure that assembled
    229 output can be disassembled and disassembler output can
    230 be assembled. It also verifies that the give assembly language file
    231 can be assembled correctly.
    274
  • ad
  • 232275

    233
    234
  • opt
  • 276 Assembler/Disassembler tests. These tests verify that a
    277 piece of LLVM assembly language can be assembled into
    278 bytecode and then disassembled into the original
    279 assembly language code. It does this several times to
    280 ensure that assembled output can be disassembled and
    281 disassembler output can be assembled. It also verifies
    282 that the give assembly language file can be assembled
    283 correctly.
    284

    285
    286
  • opt
  • 287

    235288 Optimizer tests. These tests verify that two of the
    236289 optimizer passes completely optimize a program (i.e.
    237290 after a single pass, they cannot optimize a program
    238291 any further).
    292

    293
    294
  • mc
  • 239295

    240
    241
  • mc
  • 242 Machine code tests. These tests verify that the LLVM assembly
    243 language file can be translated into native assembly code.
    296 Machine code tests. These tests verify that the LLVM
    297 assembly language file can be translated into native
    298 assembly code.
    299

    300
    301
  • cc
  • 244302

    245
    246
  • cc
  • 247 C code tests. These tests verify that the specified LLVM assembly
    248 code can be converted into C source code using the C backend.
    303 C code tests. These tests verify that the specified
    304 LLVM assembly code can be converted into C source code
    305 using the C backend.
    306

    249307
    250308
    251309

    252
    253 The LLVM database class looks at every file in llvm/test/Feature and
    254 creates a fake test hierarchy containing
    255 Feature.<testtype>.<testname>.
    256 So, if you add an LLVM assembly language file to llvm/test/Feature, it
    257 actually creates 5 news test: assembler/disassembler, assembler,
    310 The LLVM database class looks at every file in the Feature
    311 directory and creates a fake test hierarchy containing
    312 Feature.<testtype>.<testname>. So, if you
    313 add an LLVM assembly language file to the Feature directory, it
    314 actually creates 5 new tests: assembler/disassembler, assembler,
    258315 optimizer, machine code, and C code.
    259
    260
  • Regression
  • 261 These are the regression tests. There is one suite for each directory
    262 in llvm/test/Regression.
    263

    264
    265 If you add a new directory to llvm/test/Regression, you will need to
    266 modify llvm/test/QMTest/llvmdb.py so that it knows what sorts of tests
    267 are in it and how to run them.
    316

    317
    318
  • Regression
  • 319

    320 These are the regression tests. There is one suite for each
    321 subdirectory of the Regression directory. If you add a new
    322 subdirectory there, you will need to modify, at least, the
    323 RegressionMap variable in QMTest/llvmdb.py so
    324 that QMTest knows how to run the tests in the new subdirectory.
    325

    268326
    269
    270
    271

    Programs Structure

    272
    273 As mentioned previously, the Programs tree in llvm/test provides three types
    274 of tests: MultiSource, SingleSource, and External. Each tree is then
    275 subdivided into several categories, including applications, benchmarks,
    276 regression tests, code that is strange grammatically, etc. These
    277 organizations should be relatively self explanatory.
    278

    279 In addition to the regular Programs tests, the Programs tree also provides a
    280 mechanism for compiling the programs in different ways. If the variable TEST
    281 is defined on the gmake command line, the test system will include a Makefile
    282 named TEST.<value of TEST variable>.Makefile. This Makefile can modify
    283 build rules that yield different results.
    284

    285 For example, the LLVM nightly tester uses TEST.nightly.Makefile to create the
    286 nightly test reports. To run the nightly tests, run gmake
    287 TEST=nightly.
    288

    289 There are several TEST Makefiles available in the tree. Some of them are
    290 designed for internal LLVM research and will not work outside of the LLVM
    291 research group. They may still be valuable, however, as a guide to writing
    292 your own TEST Makefile for any optimization or analysis passes that you
    293 develop with LLVM.
    294
    295
    296

    Running the LLVM Tests

    297
    298
    299 First, all tests are executed within the LLVM object directory tree. They
    300 are not executed inside of the LLVM source tree. This is because
    301 the test suite creates temporary files during execution.
    302
    303

    304
    327
    328
    329
    330
    331 Structure
    332
    333
    334
    335

    336 As mentioned previously, the Programs tree in llvm/test provides three
    337 types of tests: MultiSource, SingleSource, and External. Each tree is
    338 then subdivided into several categories, including applications,
    339 benchmarks, regression tests, code that is strange grammatically, etc.
    340 These organizations should be relatively self explanatory.
    341

    342 In addition to the regular Programs tests, the Programs tree also
    343 provides a mechanism for compiling the programs in different ways. If
    344 the variable TEST is defined on the gmake command line, the test system
    345 will include a Makefile named TEST.<value of TEST
    346 variable>.Makefile. This Makefile can modify build rules to
    347 yield different results.
    348

    349 For example, the LLVM nightly tester uses TEST.nightly.Makefile
    350 to create the nightly test reports. To run the nightly tests, run
    351 gmake TEST=nightly.
    352

    353 There are several TEST Makefiles available in the tree. Some of them
    354 are designed for internal LLVM research and will not work outside of the
    355 LLVM research group. They may still be valuable, however, as a guide to
    356 writing your own TEST Makefile for any optimization or analysis passes
    357 that you develop with LLVM.
    358

    359
    360
    361
    362
    363
    364
    365
    366
    367

    368 First, all tests are executed within the LLVM object directory tree.
    369 They are not executed inside of the LLVM source tree. This is
    370 because the test suite creates temporary files during execution.
    371

    305372 The master Makefile in llvm/test is capable of running both the
    306373 QMTest driven tests and the Programs tests. By default, it will run
    307374 all of the tests.
    308 <p>
    375 </p><p>
    309376 To run only the QMTest driven tests, run make qmtest at the
    310 command line in llvm/tests. To run a specific qmtest, suffix the test name
    311 with ".t" when running make.
    312

    377 command line in llvm/tests. To run a specific qmtest, suffix the test
    378 name with ".t" when running make.
    379

    313380 For example, to run the Regression.LLC tests, type
    314381 make Regression.LLC.t in llvm/tests.
    382

    383 Note that the Makefiles in llvm/test/Features and llvm/test/Regression
    384 are gone. You must now use QMTest from the llvm/test directory to run
    385 them.
    386

    387 To run the Programs test, cd into the llvm/test/Programs directory and
    388 type make. Alternatively, you can type make
    389 TEST=<type> test to run one of the specialized tests in
    390 llvm/test/Programs/TEST.<type>.Makefile. For example, you could
    391 run the nightly tester tests using the following commands:
    392

    393
    394
    
                      
                    
    395 % cd llvm/test/Programs
    396 % make TEST=nightly test
    397
    398
    315399

    316 Note that the Makefiles in llvm/test/Features and llvm/test/Regression
    317 are gone. You must now use QMTest from the llvm/test directory to run them.
    318

    319
    320 To run the Programs test, cd into the llvm/test/Programs directory
    321 and type make. Alternatively, you can type make
    322 TEST=<type> test to run one of the specialized tests in
    323 llvm/test/Programs/TEST.<type>.Makefile. For example, you could run
    324 the nightly tester tests using the following commands:
    325

    326
    327 cd llvm/test/Programs
    328
    329 make TEST=nightly test
    330
    331
    332

    333 Regardless of which test you're running, the results are printed on standard
    334 output and standard error. You can redirect these results to a file if you
    335 choose.
    336

    400 Regardless of which test you're running, the results are printed on
    401 standard output and standard error. You can redirect these results to a
    402 file if you choose.
    403

    337404 Some tests are known to fail. Some are bugs that we have not fixed yet;
    338 others are features that we haven't added yet (or may never add). In QMTest,
    339 the result for such tests will be XFAIL (eXpected FAILure). In this way, you
    340 can tell the difference between an expected and unexpected failure.
    341

    342 The Programs tests have no such feature as of this time. If the test passes,
    343 only warnings and other miscellaneous output will be generated. If a test
    344 fails, a large <program> FAILED message will be displayed. This will
    345 help you separate benign warnings from actual test failures.
    346
    347
    348
    349
    405 others are features that we haven't added yet (or may never add). In
    406 QMTest, the result for such tests will be XFAIL (eXpected FAILure). In
    407 this way, you can tell the difference between an expected and unexpected
    408 failure.
    409

    410 The Programs tests have no such feature as of this time. If the test
    411 passes, only warnings and other miscellaneous output will be generated.
    412 If a test fails, a large <program> FAILED message will be
    413 displayed. This will help you separate benign warnings from actual test
    414 failures.
    415

    416
    417
    418
    419
    420
    421
    John T. Criswell
    422 Last modified: $Date$
    423
    424
    425
    350426