llvm.org GIT mirror llvm / e26b62c
Documentation for lit: formatting improvements. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168902 91177308-0d34-0410-b5e6-96231b3b80d8 Dmitri Gribenko 6 years ago
1 changed file(s) with 42 addition(s) and 118 deletion(s). Raw diff Collapse all Expand all
0 lit - LLVM Integrated Tester
11 ============================
22
3
43 SYNOPSIS
54 --------
65
7
86 **lit** [*options*] [*tests*]
9
107
118 DESCRIPTION
129 -----------
13
1410
1511 **lit** is a portable tool for executing LLVM and Clang style test suites,
1612 summarizing their results, and providing indication of failures. **lit** is
1915
2016 **lit** should be run with one or more *tests* to run specified on the command
2117 line. Tests can be either individual test files or directories to search for
22 tests (see "TEST DISCOVERY").
18 tests (see :ref:`test-discovery`).
2319
2420 Each specified test will be executed (potentially in parallel) and once all
2521 tests have been run **lit** will print summary information on the number of tests
26 which passed or failed (see "TEST STATUS RESULTS"). The **lit** program will
22 which passed or failed (see :ref:`test-status-results`). The **lit** program will
2723 execute with a non-zero exit code if any tests fail.
2824
2925 By default **lit** will use a succinct progress display and will only print
30 summary information for test failures. See "OUTPUT OPTIONS" for options
26 summary information for test failures. See :ref:`output-options` for options
3127 controlling the **lit** progress display and output.
3228
3329 **lit** also includes a number of options for controlling how tests are executed
34 (specific features may depend on the particular test format). See "EXECUTION
35 OPTIONS" for more information.
30 (specific features may depend on the particular test format). See
31 :ref:`execution-options` for more information.
3632
3733 Finally, **lit** also supports additional options for only running a subset of
38 the options specified on the command line, see "SELECTION OPTIONS" for
34 the options specified on the command line, see :ref:`selection-options` for
3935 more information.
4036
4137 Users interested in the **lit** architecture or designing a **lit** testing
42 implementation should see "LIT INFRASTRUCTURE"
43
38 implementation should see :ref:`lit-infrastructure`.
4439
4540 GENERAL OPTIONS
4641 ---------------
4742
48
49
5043 **-h**, **--help**
5144
5245 Show the **lit** help message.
53
54
5546
5647 **-j** *N*, **--threads**\ =\ *N*
5748
5849 Run *N* tests in parallel. By default, this is automatically chosen to match
5950 the number of detected available CPUs.
6051
61
62
6352 **--config-prefix**\ =\ *NAME*
6453
6554 Search for *NAME.cfg* and *NAME.site.cfg* when searching for test suites,
6655 instead of *lit.cfg* and *lit.site.cfg*.
67
68
6956
7057 **--param** *NAME*, **--param** *NAME*\ =\ *VALUE*
7158
7360 string if not given). The meaning and use of these parameters is test suite
7461 dependent.
7562
76
77
63 .. _output-options:
7864
7965 OUTPUT OPTIONS
8066 --------------
8167
82
83
8468 **-q**, **--quiet**
8569
8670 Suppress any output except for test failures.
8771
88
89
9072 **-s**, **--succinct**
9173
9274 Show less output, for example don't show information on tests that pass.
93
94
9575
9676 **-v**, **--verbose**
9777
9878 Show more information on test failures, for example the entire test output
9979 instead of just the test result.
10080
101
102
10381 **--no-progress-bar**
10482
10583 Do not use curses based progress bar.
10684
107
108
85 .. _execution-options:
10986
11087 EXECUTION OPTIONS
11188 -----------------
11289
113
114
11590 **--path**\ =\ *PATH*
11691
11792 Specify an addition *PATH* to use when searching for executables in tests.
118
119
12093
12194 **--vg**
12295
128101 "valgrind" feature that can be used to conditionally disable (or expect failure
129102 in) certain tests.
130103
131
132
133104 **--vg-arg**\ =\ *ARG*
134105
135106 When *--vg* is used, specify an additional argument to pass to valgrind itself.
136
137
138107
139108 **--vg-leak**
140109
141110 When *--vg* is used, enable memory leak checks. When this option is enabled,
142111 **lit** will also automatically provide a "vg_leak" feature that can be
143112 used to conditionally disable (or expect failure in) certain tests.
144
145
146
147113
148114 **--time-tests**
149115
152118 take the most time to execute. Note that this option is most useful with *-j
153119 1*.
154120
155
156
121 .. _selection-options:
157122
158123 SELECTION OPTIONS
159124 -----------------
160125
161
162
163126 **--max-tests**\ =\ *N*
164127
165128 Run at most *N* tests and then terminate.
166129
167
168
169130 **--max-time**\ =\ *N*
170131
171132 Spend at most *N* seconds (approximately) running tests and then terminate.
172133
173
174
175134 **--shuffle**
176135
177136 Run the tests in a random order.
178
179
180
181137
182138 ADDITIONAL OPTIONS
183139 ------------------
184140
185
186
187141 **--debug**
188142
189143 Run **lit** in debug mode, for debugging configuration issues and **lit** itself.
190144
191
192
193145 **--show-suites**
194146
195147 List the discovered test suites as part of the standard output.
196148
197
198
199149 **--no-tcl-as-sh**
200150
201151 Run Tcl scripts internally (instead of converting to shell scripts).
202
203
204152
205153 **--repeat**\ =\ *N*
206154
207155 Run each test *N* times. Currently this is primarily useful for timing tests,
208156 other results are not collated in any reasonable fashion.
209157
210
211
212
213158 EXIT STATUS
214159 -----------
215
216160
217161 **lit** will exit with an exit code of 1 if there are any FAIL or XPASS
218162 results. Otherwise, it will exit with the status 0. Other exit codes are used
219163 for non-test related failures (for example a user error or an internal program
220164 error).
221165
166 .. _test-discovery:
222167
223168 TEST DISCOVERY
224169 --------------
225
226170
227171 The inputs passed to **lit** can be either individual tests, or entire
228172 directories or hierarchies of tests to run. When **lit** starts up, the first
247191 configured projects, this allows **lit** to provide convenient and flexible
248192 support for out-of-tree builds.
249193
194 .. _test-status-results:
250195
251196 TEST STATUS RESULTS
252197 -------------------
253198
254
255199 Each test ultimately produces one of the following six results:
256200
257
258201 **PASS**
259202
260203 The test succeeded.
261
262
263204
264205 **XFAIL**
265206
267208 specifying that a test does not currently work, but wish to leave it in the test
268209 suite.
269210
270
271
272211 **XPASS**
273212
274213 The test succeeded, but it was expected to fail. This is used for tests which
275214 were specified as expected to fail, but are now succeeding (generally because
276215 the feature they test was broken and has been fixed).
277216
278
279
280217 **FAIL**
281218
282219 The test failed.
283
284
285220
286221 **UNRESOLVED**
287222
288223 The test result could not be determined. For example, this occurs when the test
289224 could not be run, the test itself is invalid, or the test was interrupted.
290225
291
292
293226 **UNSUPPORTED**
294227
295228 The test is not supported in this environment. This is used by test formats
296229 which can report unsupported tests.
297230
298
299
300231 Depending on the test format tests may produce additional information about
301 their status (generally only for failures). See the Output|"OUTPUT OPTIONS"
232 their status (generally only for failures). See the :ref:`output-options`
302233 section for more information.
303234
235 .. _lit-infrastructure:
304236
305237 LIT INFRASTRUCTURE
306238 ------------------
307
308239
309240 This section describes the **lit** testing architecture for users interested in
310241 creating a new **lit** testing implementation, or extending an existing one.
317248 TEST SUITES
318249 ~~~~~~~~~~~
319250
320
321 As described in "TEST DISCOVERY", tests are always located inside a *test
251 As described in :ref:`test-discovery`, tests are always located inside a *test
322252 suite*. Test suites serve to define the format of the tests they contain, the
323253 logic for finding those tests, and any additional information to run the tests.
324254
332262 themselves are Python modules which will be executed. When the config file is
333263 executed, two important global variables are predefined:
334264
335
336265 **lit**
337266
338267 The global **lit** configuration object (a *LitConfig* instance), which defines
339268 the builtin test formats, global configuration parameters, and other helper
340269 routines for implementing test configurations.
341
342
343270
344271 **config**
345272
389316 *on_clone* function will generally modify), and (3) the test path to the new
390317 directory being scanned.
391318
392
393
394
395319 TEST DISCOVERY
396320 ~~~~~~~~~~~~~~
397
398321
399322 Once test suites are located, **lit** recursively traverses the source directory
400323 (following *test_src_root*) looking for tests. When **lit** enters a
401324 sub-directory, it first checks to see if a nested test suite is defined in that
402325 directory. If so, it loads that test suite recursively, otherwise it
403 instantiates a local test config for the directory (see "LOCAL CONFIGURATION
404 FILES").
326 instantiates a local test config for the directory (see
327 :ref:`local-configuration-files`).
405328
406329 Tests are identified by the test suite they are contained within, and the
407330 relative path inside that suite. Note that the relative path may not refer to an
409332 tests" which have a path that contains both the path to the actual test file and
410333 a subpath to identify the virtual test.
411334
335 .. _local-configuration-files:
412336
413337 LOCAL CONFIGURATION FILES
414338 ~~~~~~~~~~~~~~~~~~~~~~~~~
415339
416
417340 When **lit** loads a subdirectory in a test suite, it instantiates a local test
418 configuration by cloning the configuration for the parent direction -- the root
341 configuration by cloning the configuration for the parent direction --- the root
419342 of this configuration chain will always be a test suite. Once the test
420343 configuration is cloned **lit** checks for a *lit.local.cfg* file in the
421344 subdirectory. If present, this file will be loaded and can be used to specialize
422345 the configuration for each individual directory. This facility can be used to
423346 define subdirectories of optional tests, or to change other configuration
424 parameters -- for example, to change the test format, or the suffixes which
347 parameters --- for example, to change the test format, or the suffixes which
425348 identify test files.
426
427349
428350 TEST RUN OUTPUT FORMAT
429351 ~~~~~~~~~~~~~~~~~~~~~~
430
431352
432353 The **lit** output for a test run conforms to the following schema, in both
433354 short and verbose modes (although in short mode no PASS lines will be shown).
434355 This schema has been chosen to be relatively easy to reliably parse by a machine
435356 (for example in buildbot log scraping), and for other tools to generate.
436357
437 Each test result is expected to appear on a line that matches::
358 Each test result is expected to appear on a line that matches:
359
360 .. code-block:: none
438361
439362 : ()
440363
441 where is a standard test result such as PASS, FAIL, XFAIL, XPASS,
442 UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and
364 where ```` is a standard test result such as PASS, FAIL, XFAIL,
365 XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and
443366 REGRESSED are also allowed.
444367
445 The field can consist of an arbitrary string containing no newline.
446
447 The field can be used to report progress information such as
448 (1/300) or can be empty, but even when empty the parentheses are required.
368 The ```` field can consist of an arbitrary string containing no
369 newline.
370
371 The ```` field can be used to report progress information such
372 as (1/300) or can be empty, but even when empty the parentheses are required.
449373
450374 Each test result may include additional (multiline) log information in the
451 following format::
375 following format:
376
377 .. code-block:: none
452378
453379 TEST '()'
454380 ... log message ...
455381
456382
457 where should be the name of a preceding reported test,
458 delineator> is a string of '\*' characters *at least* four characters long (the
459 recommended length is 20), and is an arbitrary (unparsed)
460 string.
383 where ```` should be the name of a preceding reported test, ``
384 delineator>`` is a string of "*" characters *at least* four characters long
385 (the recommended length is 20), and ```` is an arbitrary
386 (unparsed) string.
461387
462388 The following is an example of a test run output which consists of four tests A,
463 B, C, and D, and a log message for the failing test C::
389 B, C, and D, and a log message for the failing test C:
390
391 .. code-block:: none
464392
465393 PASS: A (1 of 4)
466394 PASS: B (2 of 4)
467395 FAIL: C (3 of 4)
468 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* TEST 'C' FAILED \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
396 ******************** TEST 'C' FAILED ********************
469397 Test 'C' failed as a result of exit code 1.
470 \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*
398 ********************
471399 PASS: D (4 of 4)
472
473400
474401 LIT EXAMPLE TESTS
475402 ~~~~~~~~~~~~~~~~~
476403
477
478404 The **lit** distribution contains several example implementations of test suites
479405 in the *ExampleTests* directory.
480406
481
482407 SEE ALSO
483408 --------
484409
485
486410 valgrind(1)