llvm.org GIT mirror llvm / bbb2a7a
For PR1319: Rewrite much of the DejaGnu section to bring it in line with the new facilities in llvm.exp. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@36015 91177308-0d34-0410-b5e6-96231b3b80d8 Reid Spencer 12 years ago
1 changed file(s) with 193 addition(s) and 75 deletion(s). Raw diff Collapse all Expand all
274274
275275
276276
277
278
279

The LLVM test suite is partially driven by DejaGNU and partially

280 driven by GNU Make. Specifically, the Features and Regression tests
281 are all driven by DejaGNU. The llvm-test
282 module is currently driven by a set of Makefiles.

283
284

The DejaGNU structure is very simple, but does require some

285 information to be set. This information is gathered via configure and
286 is written to a file, site.exp in llvm/test. The
287 llvm/test
288 Makefile does this work for you.

289
290

In order for DejaGNU to work, each directory of tests must have a

291 dg.exp file. This file is a program written in tcl that calls
292 the llvm-runtests procedure on each test file. The
293 llvm-runtests procedure is defined in
294 llvm/test/lib/llvm-dg.exp. Any directory that contains only
295 directories does not need the dg.exp file.

296
297

In order for a test to be run, it must contain information within

298 the test file on how to run the test. These are called RUN
299 lines. Run lines are specified in the comments of the test program
300 using the keyword RUN followed by a colon, and lastly the
301 commands to execute. These commands will be executed in a bash script,
302 so any bash syntax is acceptable. You can specify as many RUN lines as
303 necessary. Each RUN line translates to one line in the resulting bash
304 script. Below is an example of legal RUN lines in a .ll
305 file:

306

                  
                
307 ; RUN: llvm-as < %s | llvm-dis > %t1
308 ; RUN: llvm-dis < %s.bc-13 > %t2
309 ; RUN: diff %t1 %t2
310
311

There are a couple patterns within a RUN line that the

312 llvm-runtest procedure looks for and replaces with the appropriate
313 syntax:

314
315
316
%p
317
The path to the source directory. This is for locating
318 any supporting files that are not generated by the test, but used by
319 the test.
320
%s
321
The test file.
322
323
%t
324
Temporary filename: testscript.test_filename.tmp, where
325 test_filename is the name of the test file. All temporary files are
326 placed in the Output directory within the directory the test is
327 located.
328
329
%prcontext
330
Path to a script that performs grep -C. Use this since not all
331 platforms support grep -C.
332
333
%llvmgcc
Full path to the llvm-gcc executable.
334
%llvmgxx
Full path to the llvm-g++ executable.
335
336
337

There are also several scripts in the llvm/test/Scripts directory

338 that you might find useful when writing RUN lines.

339
340

Lastly, you can easily mark a test that is expected to fail on a

341 specific platform or with a specific version of llvmgcc by using the
342 XFAIL keyword. Xfail lines are
343 specified in the comments of the test program using XFAIL,
344 followed by a colon, and one or more regular expressions (separated by
345 a comma) that will match against the target triplet or llvmgcc version for the
346 machine. You can use * to match all targets. You can specify the major or full
347 version (i.e. 3.4) for llvmgcc. Here is an example of an
348 XFAIL line:

349

                  
                
350 ; XFAIL: darwin,sun,llvmgcc4
351 </pre>
277 <div class="doc_text">
278

The LLVM test suite is partially driven by DejaGNU and partially driven by

279 GNU Make. Specifically, the Features and Regression tests are all driven by
280 DejaGNU. The llvm-test module is currently driven by a set of
281 Makefiles.

282
283

The DejaGNU structure is very simple, but does require some information to

284 be set. This information is gathered via configure and is written
285 to a file, site.exp in llvm/test. The llvm/test
286 Makefile does this work for you.

287
288

In order for DejaGNU to work, each directory of tests must have a

289 dg.exp file. DejaGNU looks for this file to determine how to run the
290 tests. This file is just a Tcl script and it can do anything you want, but
291 we've standardized it for the LLVM regression tests. It simply loads a Tcl
292 library (test/lib/llvm.exp) and calls the llvm_runtests
293 function defined in that library with a list of file names to run. The names
294 are obtained by using Tcl's glob command. Any directory that contains only
295 directories does not need the dg.exp file.

296
297

The llvm-runtests function lookas at each file that is passed to

298 it and gathers any lines together that match "RUN:". This are the "RUN" lines
299 that specify how the test is to be run. So, each test script must contain
300 RUN lines if it is to do anything. If there are no RUN lines, the
301 llvm-runtests function will issue an error and the test will
302 fail.

303
304

RUN lines are specified in the comments of the test program using the

305 keyword RUN followed by a colon, and lastly the command (pipeline)
306 to execute. Together, these lines form the "script" that
307 llvm-runtests executes to run the test case. The syntax of the
308 RUN lines is similar to a shell's syntax for pipelines including I/O
309 redirection and variable substitution. However, even though these lines
310 may look like a shell script, they are not. RUN lines are interpreted
311 directly by the Tcl exec command. They are never executed by a
312 shell. Consequently the syntax differs from normal shell script syntax in a
313 few ways. You can specify as many RUN lines as needed.

314
315

Each RUN line is executed on its own, distinct from other lines unless

316 its last character is \. This continuation character causes the RUN
317 line to be concatenated with the next one. In this way you can build up long
318 pipelines of commands without making huge line lengths. The lines ending in
319 \ are concatenated until a RUN line that doesn't end in \ is
320 found. This concatenated set or RUN lines then constitutes one execution.
321 Tcl will substitute variables and arrange for the pipeline to be executed. If
322 any process in the pipeline fails, the entire line (and test case) fails too.
323

324
325

Below is an example of legal RUN lines in a .ll file:

326

                  
                
327 ; RUN: llvm-as < %s | llvm-dis > %t1
328 ; RUN: llvm-dis < %s.bc-13 > %t2
329 ; RUN: diff %t1 %t2
330
331
332
333
334
335
336

With a RUN line there are a number of substitutions that are permitted. In

337 general, any Tcl variable that is available in the substitute
338 function (in test/lib/llvm.exp) can be substituted into a RUN line.
339 To make a substitution just write the variable's name preceded by a $.
340 Additionally, for compatibility reasons with previous versions of the test
341 library, certain names can be accessed with an alternate syntax: a % prefix.
342 These alternates are deprecated and may go away in a future version.
343

344 Here are the available variable names. The alternate syntax is listed in
345 parentheses.

346
347
$test (%s)
348
The full path to the test case's source. This is suitable for passing
349 on the command line as the input to an llvm tool.
350
$srcdir
351
The source directory from where the "make check" was run.
352
objdir
353
The object directory that corresponds to the $srcdir.
354
subdir
355
A partial path from the test directory that contains the
356 sub-directory that contains the test source being executed.
357
srcroot
358
The root directory of the LLVM src tree.
359
objroot
360
The root directory of the LLVM object tree. This could be the same
361 as the srcroot.
362
path
363
The path to the directory that contains the test case source. This is
364 for locating any supporting files that are not generated by the test, but
365 used by the test.
366
tmp
367
The path to a temporary file name that could be used for this test case.
368 The file name won't conflict with other test cases. You can append to it if
369 you need multiple temporaries. This is useful as the destination of some
370 redirected output.
371
llvmlibsdir (%llvmlibsdir)
372
The directory where the LLVM libraries are located.
373
target_triplet (%target_triplet)
374
The target triplet that corresponds to the current host machine (the one
375 running the test cases). This should probably be called "host".
376
prcontext (%prcontext)
377
Path to the prcontext tcl script that prints some context around a
378 line that matches a pattern. This isn't strictly necessary as the test suite
379 is run with its PATH altered to include the test/Scripts directory where
380 the prcontext script is located. Note that this script is similar to
381 grep -C but you should use the prcontext script because
382 not all platforms support grep -C.
383
llvmgcc (%llvmgcc)
384
The full path to the llvm-gcc executable as specified in the
385 configured LLVM environment
386
llvmgxx (%llvmgxx)
387
The full path to the llvm-gxx executable as specified in the
388 configured LLVM environment
389
llvmgcc_version (%llvmgcc_version)
390
The full version number of the llvm-gcc executable.
391
llvmgccmajvers (%llvmgccmajvers)
392
The major version number of the llvm-gcc executable.
393
gccpath
394
The full path to the C compiler used to build LLVM. Note that
395 this might not be gcc.
396
gxxpath
397
The full path to the C++ compiler used to build LLVM. Note that
398 this might not be g++.
399
compile_c (%compile_c)
400
The full command line used to compile LLVM C source code. This has all
401 the configured -I, -D and optimization options.
402
compile_cxx (%compile_cxx)
403
The full command used to compile LLVM C++ source code. This has
404 all the configured -I, -D and optimization options.
405
link (%link)
406
This full link command used to link LLVM executables. This has all the
407 configured -I, -L and -l options.
408
shlibext (%shlibext)
409
The suffix for the host platforms share library (dll) files. This
410 includes the period as the first character.
411
412

To add more variables, two things need to be changed. First, add a line in

413 the test/Makefile that creates the site.exp file. This will
414 "set" the variable as a global in the site.exp file. Second, in the
415 test/lib/llvm.exp file, in the substitute proc, add the variable name
416 to the list of "global" declarations at the beginning of the proc. That's it,
417 the variable can then be used in test scripts.

418
419
420
421
422
423

To make RUN line writing easier, there are several shell scripts located

424 in the llvm/test/Scripts directory. For example:

425
426
ignore
427
This script runs its arguments and then always returns 0. This is useful
428 in cases where the test needs to cause a tool to generate an error (e.g. to
429 check the error output). However, any program in a pipeline that returns a
430 non-zero result will cause the test to fail. This script overcomes that
431 issue and nicely documents that the test case is purposefully ignoring the
432 result code of the tool
433
not
434
This script runs its arguments and then inverts the result code from
435 it. Zero result codes become 1. Non-zero result codes become 0. This is
436 useful to invert the result of a grep. For example "not grep X" means
437 succeed only if you don't find X in the input.
438
439
440

Sometimes it is necessary to mark a test case as "expected fail" or XFAIL.

441 You can easily mark a test as XFAIL just by including XFAIL: on a
442 line near the top of the file. This signals that the test case should succeed
443 if the test fails. Such test cases are counted separately by DejaGnu. To
444 specify an expected fail, use the XFAIL keyword in the comments of the test
445 program followed by a colon and one or more regular expressions (separated by
446 a comma). The regular expressions allow you to XFAIL the test conditionally
447 by host platform. The regular expressions following the : are matched against
448 the target triplet or llvmgcc version number for the host machine. If there is
449 a match, the test is expected to fail. If not, the test is expected to
450 succeed. To XFAIL everywhere just specify XFAIL: *. When matching
451 the llvm-gcc version, you can specify the major (e.g. 3) or full version
452 (i.e. 3.4) number. Here is an example of an XFAIL line:

453

                  
                
454 ; XFAIL: darwin,sun,llvmgcc4
455
456
457

To make the output more useful, the llvm_runtest function wil

458 scan the lines of the test case for ones that contain a pattern that matches
459 PR[0-9]+. This is the syntax for specifying a PR (Problem Report) number that
460 is related to the test case. The numer after "PR" specifies the LLVM bugzilla
461 number. When a PR number is specified, it will be used in the pass/fail
462 reporting. This is useful to quickly get some context when a test fails.

463
464

Finally, any line that contains "END." will cause the special

465 interpretation of lines to terminate. This is generally done right after the
466 last RUN: line. This has two side effects: (a) it prevents special
467 interpretation of lines that are part of the test program, not the
468 instructions to the test case, and (b) it speeds things up for really big test
469 cases by avoiding interpretation of the remainder of the file.

352470
353471
354472