llvm.org GIT mirror llvm / bc5fb06
Documentation for lit: more formatting: use 'option' and 'program' directives. This enables cross-referencing and now '--' in option names are no more turned into en dashes. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@168906 91177308-0d34-0410-b5e6-96231b3b80d8 Dmitri Gribenko 6 years ago
1 changed file(s) with 176 addition(s) and 171 deletion(s). Raw diff Collapse all Expand all
33 SYNOPSIS
44 --------
55
6 **lit** [*options*] [*tests*]
6 :program:`lit` [*options*] [*tests*]
77
88 DESCRIPTION
99 -----------
1010
11 **lit** is a portable tool for executing LLVM and Clang style test suites,
12 summarizing their results, and providing indication of failures. **lit** is
13 designed to be a lightweight testing tool with as simple a user interface as
14 possible.
15
16 **lit** should be run with one or more *tests* to run specified on the command
17 line. Tests can be either individual test files or directories to search for
18 tests (see :ref:`test-discovery`).
11 :program:`lit` is a portable tool for executing LLVM and Clang style test
12 suites, summarizing their results, and providing indication of failures.
13 :program:`lit` is designed to be a lightweight testing tool with as simple a
14 user interface as possible.
15
16 :program:`lit` should be run with one or more *tests* to run specified on the
17 command line. Tests can be either individual test files or directories to
18 search for tests (see :ref:`test-discovery`).
1919
2020 Each specified test will be executed (potentially in parallel) and once all
21 tests have been run **lit** will print summary information on the number of tests
22 which passed or failed (see :ref:`test-status-results`). The **lit** program will
23 execute with a non-zero exit code if any tests fail.
24
25 By default **lit** will use a succinct progress display and will only print
26 summary information for test failures. See :ref:`output-options` for options
27 controlling the **lit** progress display and output.
28
29 **lit** also includes a number of options for controlling how tests are executed
30 (specific features may depend on the particular test format). See
21 tests have been run :program:`lit` will print summary information on the number
22 of tests which passed or failed (see :ref:`test-status-results`). The
23 :program:`lit` program will execute with a non-zero exit code if any tests
24 fail.
25
26 By default :program:`lit` will use a succinct progress display and will only
27 print summary information for test failures. See :ref:`output-options` for
28 options controlling the :program:`lit` progress display and output.
29
30 :program:`lit` also includes a number of options for controlling how tests are
31 executed (specific features may depend on the particular test format). See
3132 :ref:`execution-options` for more information.
3233
33 Finally, **lit** also supports additional options for only running a subset of
34 the options specified on the command line, see :ref:`selection-options` for
35 more information.
36
37 Users interested in the **lit** architecture or designing a **lit** testing
38 implementation should see :ref:`lit-infrastructure`.
34 Finally, :program:`lit` also supports additional options for only running a
35 subset of the options specified on the command line, see
36 :ref:`selection-options` for more information.
37
38 Users interested in the :program:`lit` architecture or designing a
39 :program:`lit` testing implementation should see :ref:`lit-infrastructure`.
3940
4041 GENERAL OPTIONS
4142 ---------------
4243
43 **-h**, **--help**
44
45 Show the **lit** help message.
46
47 **-j** *N*, **--threads**\ =\ *N*
48
49 Run *N* tests in parallel. By default, this is automatically chosen to match
50 the number of detected available CPUs.
51
52 **--config-prefix**\ =\ *NAME*
53
54 Search for *NAME.cfg* and *NAME.site.cfg* when searching for test suites,
55 instead of *lit.cfg* and *lit.site.cfg*.
56
57 **--param** *NAME*, **--param** *NAME*\ =\ *VALUE*
58
59 Add a user defined parameter *NAME* with the given *VALUE* (or the empty
60 string if not given). The meaning and use of these parameters is test suite
44 .. option:: -h, --help
45
46 Show the :program:`lit` help message.
47
48 .. option:: -j N, --threads=N
49
50 Run ``N`` tests in parallel. By default, this is automatically chosen to
51 match the number of detected available CPUs.
52
53 .. option:: --config-prefix=NAME
54
55 Search for :file:`{NAME}.cfg` and :file:`{NAME}.site.cfg` when searching for
56 test suites, instead of :file:`lit.cfg` and :file:`lit.site.cfg`.
57
58 .. option:: --param NAME, --param NAME=VALUE
59
60 Add a user defined parameter ``NAME`` with the given ``VALUE`` (or the empty
61 string if not given). The meaning and use of these parameters is test suite
6162 dependent.
6263
6364 .. _output-options:
6566 OUTPUT OPTIONS
6667 --------------
6768
68 **-q**, **--quiet**
69 .. option:: -q, --quiet
6970
7071 Suppress any output except for test failures.
7172
72 **-s**, **--succinct**
73 .. option:: -s, --succinct
7374
7475 Show less output, for example don't show information on tests that pass.
7576
76 **-v**, **--verbose**
77 .. option:: -v, --verbose
7778
7879 Show more information on test failures, for example the entire test output
7980 instead of just the test result.
8081
81 **--no-progress-bar**
82 .. option:: --no-progress-bar
8283
8384 Do not use curses based progress bar.
8485
8788 EXECUTION OPTIONS
8889 -----------------
8990
90 **--path**\ =\ *PATH*
91
92 Specify an addition *PATH* to use when searching for executables in tests.
93
94 **--vg**
95
96 Run individual tests under valgrind (using the memcheck tool). The
97 *--error-exitcode* argument for valgrind is used so that valgrind failures will
98 cause the program to exit with a non-zero status.
99
100 When this option is enabled, **lit** will also automatically provide a
101 "valgrind" feature that can be used to conditionally disable (or expect failure
102 in) certain tests.
103
104 **--vg-arg**\ =\ *ARG*
105
106 When *--vg* is used, specify an additional argument to pass to valgrind itself.
107
108 **--vg-leak**
109
110 When *--vg* is used, enable memory leak checks. When this option is enabled,
111 **lit** will also automatically provide a "vg_leak" feature that can be
112 used to conditionally disable (or expect failure in) certain tests.
113
114 **--time-tests**
115
116 Track the wall time individual tests take to execute and includes the results in
117 the summary output. This is useful for determining which tests in a test suite
118 take the most time to execute. Note that this option is most useful with *-j
119 1*.
91 .. option:: --path=PATH
92
93 Specify an additional ``PATH`` to use when searching for executables in tests.
94
95 .. option:: --vg
96
97 Run individual tests under valgrind (using the memcheck tool). The
98 ``--error-exitcode`` argument for valgrind is used so that valgrind failures
99 will cause the program to exit with a non-zero status.
100
101 When this option is enabled, :program:`lit` will also automatically provide a
102 "``valgrind``" feature that can be used to conditionally disable (or expect
103 failure in) certain tests.
104
105 .. option:: --vg-arg=ARG
106
107 When :option:`--vg` is used, specify an additional argument to pass to
108 :program:`valgrind` itself.
109
110 .. option:: --vg-leak
111
112 When :option:`--vg` is used, enable memory leak checks. When this option is
113 enabled, :program:`lit` will also automatically provide a "``vg_leak``"
114 feature that can be used to conditionally disable (or expect failure in)
115 certain tests.
116
117 .. option:: --time-tests
118
119 Track the wall time individual tests take to execute and includes the results
120 in the summary output. This is useful for determining which tests in a test
121 suite take the most time to execute. Note that this option is most useful
122 with ``-j 1``.
120123
121124 .. _selection-options:
122125
123126 SELECTION OPTIONS
124127 -----------------
125128
126 **--max-tests**\ =\ *N*
127
128 Run at most *N* tests and then terminate.
129
130 **--max-time**\ =\ *N*
131
132 Spend at most *N* seconds (approximately) running tests and then terminate.
133
134 **--shuffle**
129 .. option:: --max-tests=N
130
131 Run at most ``N`` tests and then terminate.
132
133 .. option:: --max-time=N
134
135 Spend at most ``N`` seconds (approximately) running tests and then terminate.
136
137 .. option:: --shuffle
135138
136139 Run the tests in a random order.
137140
138141 ADDITIONAL OPTIONS
139142 ------------------
140143
141 **--debug**
142
143 Run **lit** in debug mode, for debugging configuration issues and **lit** itself.
144
145 **--show-suites**
144 .. option:: --debug
145
146 Run :program:`lit` in debug mode, for debugging configuration issues and
147 :program:`lit` itself.
148
149 .. option:: --show-suites
146150
147151 List the discovered test suites as part of the standard output.
148152
149 **--no-tcl-as-sh**
153 .. option:: --no-tcl-as-sh
150154
151155 Run Tcl scripts internally (instead of converting to shell scripts).
152156
153 **--repeat**\ =\ *N*
154
155 Run each test *N* times. Currently this is primarily useful for timing tests,
156 other results are not collated in any reasonable fashion.
157 .. option:: --repeat=N
158
159 Run each test ``N`` times. Currently this is primarily useful for timing
160 tests, other results are not collated in any reasonable fashion.
157161
158162 EXIT STATUS
159163 -----------
160164
161 **lit** will exit with an exit code of 1 if there are any FAIL or XPASS
162 results. Otherwise, it will exit with the status 0. Other exit codes are used
165 :program:`lit` will exit with an exit code of 1 if there are any FAIL or XPASS
166 results. Otherwise, it will exit with the status 0. Other exit codes are used
163167 for non-test related failures (for example a user error or an internal program
164168 error).
165169
168172 TEST DISCOVERY
169173 --------------
170174
171 The inputs passed to **lit** can be either individual tests, or entire
172 directories or hierarchies of tests to run. When **lit** starts up, the first
173 thing it does is convert the inputs into a complete list of tests to run as part
174 of *test discovery*.
175
176 In the **lit** model, every test must exist inside some *test suite*. **lit**
177 resolves the inputs specified on the command line to test suites by searching
178 upwards from the input path until it finds a *lit.cfg* or *lit.site.cfg*
179 file. These files serve as both a marker of test suites and as configuration
180 files which **lit** loads in order to understand how to find and run the tests
181 inside the test suite.
182
183 Once **lit** has mapped the inputs into test suites it traverses the list of
184 inputs adding tests for individual files and recursively searching for tests in
185 directories.
175 The inputs passed to :program:`lit` can be either individual tests, or entire
176 directories or hierarchies of tests to run. When :program:`lit` starts up, the
177 first thing it does is convert the inputs into a complete list of tests to run
178 as part of *test discovery*.
179
180 In the :program:`lit` model, every test must exist inside some *test suite*.
181 :program:`lit` resolves the inputs specified on the command line to test suites
182 by searching upwards from the input path until it finds a :file:`lit.cfg` or
183 :file:`lit.site.cfg` file. These files serve as both a marker of test suites
184 and as configuration files which :program:`lit` loads in order to understand
185 how to find and run the tests inside the test suite.
186
187 Once :program:`lit` has mapped the inputs into test suites it traverses the
188 list of inputs adding tests for individual files and recursively searching for
189 tests in directories.
186190
187191 This behavior makes it easy to specify a subset of tests to run, while still
188192 allowing the test suite configuration to control exactly how tests are
189 interpreted. In addition, **lit** always identifies tests by the test suite they
190 are in, and their relative path inside the test suite. For appropriately
191 configured projects, this allows **lit** to provide convenient and flexible
192 support for out-of-tree builds.
193 interpreted. In addition, :program:`lit` always identifies tests by the test
194 suite they are in, and their relative path inside the test suite. For
195 appropriately configured projects, this allows :program:`lit` to provide
196 convenient and flexible support for out-of-tree builds.
193197
194198 .. _test-status-results:
195199
204208
205209 **XFAIL**
206210
207 The test failed, but that is expected. This is used for test formats which allow
211 The test failed, but that is expected. This is used for test formats which allow
208212 specifying that a test does not currently work, but wish to leave it in the test
209213 suite.
210214
211215 **XPASS**
212216
213 The test succeeded, but it was expected to fail. This is used for tests which
217 The test succeeded, but it was expected to fail. This is used for tests which
214218 were specified as expected to fail, but are now succeeding (generally because
215219 the feature they test was broken and has been fixed).
216220
220224
221225 **UNRESOLVED**
222226
223 The test result could not be determined. For example, this occurs when the test
227 The test result could not be determined. For example, this occurs when the test
224228 could not be run, the test itself is invalid, or the test was interrupted.
225229
226230 **UNSUPPORTED**
227231
228 The test is not supported in this environment. This is used by test formats
232 The test is not supported in this environment. This is used by test formats
229233 which can report unsupported tests.
230234
231235 Depending on the test format tests may produce additional information about
232 their status (generally only for failures). See the :ref:`output-options`
236 their status (generally only for failures). See the :ref:`output-options`
233237 section for more information.
234238
235239 .. _lit-infrastructure:
237241 LIT INFRASTRUCTURE
238242 ------------------
239243
240 This section describes the **lit** testing architecture for users interested in
241 creating a new **lit** testing implementation, or extending an existing one.
242
243 **lit** proper is primarily an infrastructure for discovering and running
244 This section describes the :program:`lit` testing architecture for users interested in
245 creating a new :program:`lit` testing implementation, or extending an existing one.
246
247 :program:`lit` proper is primarily an infrastructure for discovering and running
244248 arbitrary tests, and to expose a single convenient interface to these
245 tests. **lit** itself doesn't know how to run tests, rather this logic is
249 tests. :program:`lit` itself doesn't know how to run tests, rather this logic is
246250 defined by *test suites*.
247251
248252 TEST SUITES
249253 ~~~~~~~~~~~
250254
251255 As described in :ref:`test-discovery`, tests are always located inside a *test
252 suite*. Test suites serve to define the format of the tests they contain, the
256 suite*. Test suites serve to define the format of the tests they contain, the
253257 logic for finding those tests, and any additional information to run the tests.
254258
255 **lit** identifies test suites as directories containing *lit.cfg* or
256 *lit.site.cfg* files (see also **--config-prefix**). Test suites are initially
257 discovered by recursively searching up the directory hierarchy for all the input
258 files passed on the command line. You can use **--show-suites** to display the
259 discovered test suites at startup.
260
261 Once a test suite is discovered, its config file is loaded. Config files
262 themselves are Python modules which will be executed. When the config file is
259 :program:`lit` identifies test suites as directories containing ``lit.cfg`` or
260 ``lit.site.cfg`` files (see also :option:`--config-prefix`). Test suites are
261 initially discovered by recursively searching up the directory hierarchy for
262 all the input files passed on the command line. You can use
263 :option:`--show-suites` to display the discovered test suites at startup.
264
265 Once a test suite is discovered, its config file is loaded. Config files
266 themselves are Python modules which will be executed. When the config file is
263267 executed, two important global variables are predefined:
264268
265269 **lit**
271275 **config**
272276
273277 This is the config object (a *TestingConfig* instance) for the test suite,
274 which the config file is expected to populate. The following variables are also
278 which the config file is expected to populate. The following variables are also
275279 available on the *config* object, some of which must be set by the config and
276280 others are optional or predefined:
277281
279283 diagnostics.
280284
281285 **test_format** *[required]* The test format object which will be used to
282 discover and run tests in the test suite. Generally this will be a builtin test
286 discover and run tests in the test suite. Generally this will be a builtin test
283287 format available from the *lit.formats* module.
284288
285 **test_src_root** The filesystem path to the test suite root. For out-of-dir
289 **test_src_root** The filesystem path to the test suite root. For out-of-dir
286290 builds this is the directory that will be scanned for tests.
287291
288292 **test_exec_root** For out-of-dir builds, the path to the test suite root inside
289 the object directory. This is where tests will be run and temporary output files
293 the object directory. This is where tests will be run and temporary output files
290294 placed.
291295
292296 **environment** A dictionary representing the environment to use when executing
293297 tests in the suite.
294298
295299 **suffixes** For **lit** test formats which scan directories for tests, this
296 variable is a list of suffixes to identify test files. Used by: *ShTest*,
300 variable is a list of suffixes to identify test files. Used by: *ShTest*,
297301 *TclTest*.
298302
299303 **substitutions** For **lit** test formats which substitute variables into a test
300 script, the list of substitutions to perform. Used by: *ShTest*, *TclTest*.
304 script, the list of substitutions to perform. Used by: *ShTest*, *TclTest*.
301305
302306 **unsupported** Mark an unsupported directory, all tests within it will be
303 reported as unsupported. Used by: *ShTest*, *TclTest*.
307 reported as unsupported. Used by: *ShTest*, *TclTest*.
304308
305309 **parent** The parent configuration, this is the config object for the directory
306310 containing the test suite, or None.
307311
308 **root** The root configuration. This is the top-most **lit** configuration in
312 **root** The root configuration. This is the top-most :program:`lit` configuration in
309313 the project.
310314
311315 **on_clone** The config is actually cloned for every subdirectory inside a test
312 suite, to allow local configuration on a per-directory basis. The *on_clone*
316 suite, to allow local configuration on a per-directory basis. The *on_clone*
313317 variable can be set to a Python function which will be called whenever a
314 configuration is cloned (for a subdirectory). The function should takes three
318 configuration is cloned (for a subdirectory). The function should takes three
315319 arguments: (1) the parent configuration, (2) the new configuration (which the
316320 *on_clone* function will generally modify), and (3) the test path to the new
317321 directory being scanned.
319323 TEST DISCOVERY
320324 ~~~~~~~~~~~~~~
321325
322 Once test suites are located, **lit** recursively traverses the source directory
323 (following *test_src_root*) looking for tests. When **lit** enters a
324 sub-directory, it first checks to see if a nested test suite is defined in that
325 directory. If so, it loads that test suite recursively, otherwise it
326 instantiates a local test config for the directory (see
326 Once test suites are located, :program:`lit` recursively traverses the source
327 directory (following *test_src_root*) looking for tests. When :program:`lit`
328 enters a sub-directory, it first checks to see if a nested test suite is
329 defined in that directory. If so, it loads that test suite recursively,
330 otherwise it instantiates a local test config for the directory (see
327331 :ref:`local-configuration-files`).
328332
329333 Tests are identified by the test suite they are contained within, and the
330 relative path inside that suite. Note that the relative path may not refer to an
331 actual file on disk; some test formats (such as *GoogleTest*) define "virtual
332 tests" which have a path that contains both the path to the actual test file and
333 a subpath to identify the virtual test.
334 relative path inside that suite. Note that the relative path may not refer to
335 an actual file on disk; some test formats (such as *GoogleTest*) define
336 "virtual tests" which have a path that contains both the path to the actual
337 test file and a subpath to identify the virtual test.
334338
335339 .. _local-configuration-files:
336340
337341 LOCAL CONFIGURATION FILES
338342 ~~~~~~~~~~~~~~~~~~~~~~~~~
339343
340 When **lit** loads a subdirectory in a test suite, it instantiates a local test
341 configuration by cloning the configuration for the parent direction --- the root
342 of this configuration chain will always be a test suite. Once the test
343 configuration is cloned **lit** checks for a *lit.local.cfg* file in the
344 subdirectory. If present, this file will be loaded and can be used to specialize
345 the configuration for each individual directory. This facility can be used to
346 define subdirectories of optional tests, or to change other configuration
347 parameters --- for example, to change the test format, or the suffixes which
348 identify test files.
344 When :program:`lit` loads a subdirectory in a test suite, it instantiates a
345 local test configuration by cloning the configuration for the parent direction
346 --- the root of this configuration chain will always be a test suite. Once the
347 test configuration is cloned :program:`lit` checks for a *lit.local.cfg* file
348 in the subdirectory. If present, this file will be loaded and can be used to
349 specialize the configuration for each individual directory. This facility can
350 be used to define subdirectories of optional tests, or to change other
351 configuration parameters --- for example, to change the test format, or the
352 suffixes which identify test files.
349353
350354 TEST RUN OUTPUT FORMAT
351355 ~~~~~~~~~~~~~~~~~~~~~~
352356
353 The **lit** output for a test run conforms to the following schema, in both
354 short and verbose modes (although in short mode no PASS lines will be shown).
355 This schema has been chosen to be relatively easy to reliably parse by a machine
356 (for example in buildbot log scraping), and for other tools to generate.
357 The :program:`lit` output for a test run conforms to the following schema, in
358 both short and verbose modes (although in short mode no PASS lines will be
359 shown). This schema has been chosen to be relatively easy to reliably parse by
360 a machine (for example in buildbot log scraping), and for other tools to
361 generate.
357362
358363 Each test result is expected to appear on a line that matches:
359364
362367 : ()
363368
364369 where ```` is a standard test result such as PASS, FAIL, XFAIL,
365 XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and
370 XPASS, UNRESOLVED, or UNSUPPORTED. The performance result codes of IMPROVED and
366371 REGRESSED are also allowed.
367372
368373 The ```` field can consist of an arbitrary string containing no
401406 LIT EXAMPLE TESTS
402407 ~~~~~~~~~~~~~~~~~
403408
404 The **lit** distribution contains several example implementations of test suites
405 in the *ExampleTests* directory.
409 The :program:`lit` distribution contains several example implementations of
410 test suites in the *ExampleTests* directory.
406411
407412 SEE ALSO
408413 --------