llvm.org GIT mirror llvm / 4a50c7f
Reinstate "r292904 - [lit] Allow boolean expressions in REQUIRES and XFAIL and UNSUPPORTED" This reverts the revert in r292942. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@293007 91177308-0d34-0410-b5e6-96231b3b80d8 Greg Parker 2 years ago
16 changed file(s) with 607 addition(s) and 117 deletion(s). Raw diff Collapse all Expand all
386386 triple, test with the specific FileCheck and put it into the specific
387387 directory that will filter out all other architectures.
388388
389 REQUIRES and REQUIRES-ANY directive
390 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
391
392 Some tests can be enabled only in specific situation - like having
393 debug build. Use ``REQUIRES`` directive to specify those requirements.
389
390 Constraining test execution
391 ---------------------------
392
393 Some tests can be run only in specific configurations, such as
394 with debug builds or on particular platforms. Use ``REQUIRES``
395 and ``UNSUPPORTED`` to control when the test is enabled.
396
397 Some tests are expected to fail. For example, there may be a known bug
398 that the test detect. Use ``XFAIL`` to mark a test as an expected failure.
399 An ``XFAIL`` test will be successful if its execution fails, and
400 will be a failure if its execution succeeds.
394401
395402 .. code-block:: llvm
396403
397 ; This test will be only enabled in the build with asserts
404 ; This test will be only enabled in the build with asserts.
398405 ; REQUIRES: asserts
399
400 You can separate requirements by a comma.
401 ``REQUIRES`` means all listed requirements must be satisfied.
402 ``REQUIRES-ANY`` means at least one must be satisfied.
403
404 List of features that can be used in ``REQUIRES`` and ``REQUIRES-ANY`` can be
405 found in lit.cfg files.
406 ; This test is disabled on Linux.
407 ; UNSUPPORTED: -linux-
408 ; This test is expected to fail on PowerPC.
409 ; XFAIL: powerpc
410
411 ``REQUIRES`` and ``UNSUPPORTED`` and ``XFAIL`` all accept a comma-separated
412 list of boolean expressions. The values in each expression may be:
413
414 - Features added to ``config.available_features`` by
415 configuration files such as ``lit.cfg``.
416 - Substrings of the target triple (``UNSUPPORTED`` and ``XFAIL`` only).
417
418 | ``REQUIRES`` enables the test if all expressions are true.
419 | ``UNSUPPORTED`` disables the test if any expression is true.
420 | ``XFAIL`` expects the test to fail if any expression is true.
421
422 As a special case, ``XFAIL: *`` is expected to fail everywhere.
423
424 .. code-block:: llvm
425
426 ; This test is disabled on Windows,
427 ; and is disabled on Linux, except for Android Linux.
428 ; UNSUPPORTED: windows, linux && !android
429 ; This test is expected to fail on both PowerPC and ARM.
430 ; XFAIL: powerpc || arm
431
406432
407433 Substitutions
408434 -------------
518544 ``not``
519545 This program runs its arguments and then inverts the result code from it.
520546 Zero result codes become 1. Non-zero result codes become 0.
521
522 Sometimes it is necessary to mark a test case as "expected fail" or
523 XFAIL. You can easily mark a test as XFAIL just by including ``XFAIL:``
524 on a line near the top of the file. This signals that the test case
525 should succeed if the test fails. Such test cases are counted separately
526 by the testing tool. To specify an expected fail, use the XFAIL keyword
527 in the comments of the test program followed by a colon and one or more
528 failure patterns. Each failure pattern can be either ``*`` (to specify
529 fail everywhere), or a part of a target triple (indicating the test
530 should fail on that platform), or the name of a configurable feature
531 (for example, ``loadable_module``). If there is a match, the test is
532 expected to fail. If not, the test is expected to succeed. To XFAIL
533 everywhere just specify ``XFAIL: *``. Here is an example of an ``XFAIL``
534 line:
535
536 .. code-block:: llvm
537
538 ; XFAIL: darwin,sun
539547
540548 To make the output more useful, :program:`lit` will scan
541549 the lines of the test case for ones that contain a pattern that matches
0 import re
1
2 class BooleanExpression:
3 # A simple evaluator of boolean expressions.
4 #
5 # Grammar:
6 # expr :: or_expr
7 # or_expr :: and_expr ('||' and_expr)*
8 # and_expr :: not_expr ('&&' not_expr)*
9 # not_expr :: '!' not_expr
10 # '(' or_expr ')'
11 # identifier
12 # identifier :: [-+=._a-zA-Z0-9]+
13
14 # Evaluates `string` as a boolean expression.
15 # Returns True or False. Throws a ValueError on syntax error.
16 #
17 # Variables in `variables` are true.
18 # Substrings of `triple` are true.
19 # 'true' is true.
20 # All other identifiers are false.
21 @staticmethod
22 def evaluate(string, variables, triple=""):
23 try:
24 parser = BooleanExpression(string, set(variables), triple)
25 return parser.parseAll()
26 except ValueError as e:
27 raise ValueError(str(e) + ('\nin expression: %r' % string))
28
29 #####
30
31 def __init__(self, string, variables, triple=""):
32 self.tokens = BooleanExpression.tokenize(string)
33 self.variables = variables
34 self.variables.add('true')
35 self.triple = triple
36 self.value = None
37 self.token = None
38
39 # Singleton end-of-expression marker.
40 END = object()
41
42 # Tokenization pattern.
43 Pattern = re.compile(r'\A\s*([()]|[-+=._a-zA-Z0-9]+|&&|\|\||!)\s*(.*)\Z')
44
45 @staticmethod
46 def tokenize(string):
47 while True:
48 m = re.match(BooleanExpression.Pattern, string)
49 if m is None:
50 if string == "":
51 yield BooleanExpression.END;
52 return
53 else:
54 raise ValueError("couldn't parse text: %r" % string)
55
56 token = m.group(1)
57 string = m.group(2)
58 yield token
59
60 def quote(self, token):
61 if token is BooleanExpression.END:
62 return ''
63 else:
64 return repr(token)
65
66 def accept(self, t):
67 if self.token == t:
68 self.token = next(self.tokens)
69 return True
70 else:
71 return False
72
73 def expect(self, t):
74 if self.token == t:
75 if self.token != BooleanExpression.END:
76 self.token = next(self.tokens)
77 else:
78 raise ValueError("expected: %s\nhave: %s" %
79 (self.quote(t), self.quote(self.token)))
80
81 def isIdentifier(self, t):
82 if (t is BooleanExpression.END or t == '&&' or t == '||' or
83 t == '!' or t == '(' or t == ')'):
84 return False
85 return True
86
87 def parseNOT(self):
88 if self.accept('!'):
89 self.parseNOT()
90 self.value = not self.value
91 elif self.accept('('):
92 self.parseOR()
93 self.expect(')')
94 elif not self.isIdentifier(self.token):
95 raise ValueError("expected: '!' or '(' or identifier\nhave: %s" %
96 self.quote(self.token))
97 else:
98 self.value = (self.token in self.variables or
99 self.token in self.triple)
100 self.token = next(self.tokens)
101
102 def parseAND(self):
103 self.parseNOT()
104 while self.accept('&&'):
105 left = self.value
106 self.parseNOT()
107 right = self.value
108 # this is technically the wrong associativity, but it
109 # doesn't matter for this limited expression grammar
110 self.value = left and right
111
112 def parseOR(self):
113 self.parseAND()
114 while self.accept('||'):
115 left = self.value
116 self.parseAND()
117 right = self.value
118 # this is technically the wrong associativity, but it
119 # doesn't matter for this limited expression grammar
120 self.value = left or right
121
122 def parseAll(self):
123 self.token = next(self.tokens)
124 self.parseOR()
125 self.expect(BooleanExpression.END)
126 return self.value
127
128
129 #######
130 # Tests
131
132 import unittest
133
134 class TestBooleanExpression(unittest.TestCase):
135 def test_variables(self):
136 variables = {'its-true', 'false-lol-true', 'under_score',
137 'e=quals', 'd1g1ts'}
138 self.assertTrue(BooleanExpression.evaluate('true', variables))
139 self.assertTrue(BooleanExpression.evaluate('its-true', variables))
140 self.assertTrue(BooleanExpression.evaluate('false-lol-true', variables))
141 self.assertTrue(BooleanExpression.evaluate('under_score', variables))
142 self.assertTrue(BooleanExpression.evaluate('e=quals', variables))
143 self.assertTrue(BooleanExpression.evaluate('d1g1ts', variables))
144
145 self.assertFalse(BooleanExpression.evaluate('false', variables))
146 self.assertFalse(BooleanExpression.evaluate('True', variables))
147 self.assertFalse(BooleanExpression.evaluate('true-ish', variables))
148 self.assertFalse(BooleanExpression.evaluate('not_true', variables))
149 self.assertFalse(BooleanExpression.evaluate('tru', variables))
150
151 def test_triple(self):
152 triple = 'arch-vendor-os'
153 self.assertTrue(BooleanExpression.evaluate('arch-', {}, triple))
154 self.assertTrue(BooleanExpression.evaluate('ar', {}, triple))
155 self.assertTrue(BooleanExpression.evaluate('ch-vend', {}, triple))
156 self.assertTrue(BooleanExpression.evaluate('-vendor-', {}, triple))
157 self.assertTrue(BooleanExpression.evaluate('-os', {}, triple))
158 self.assertFalse(BooleanExpression.evaluate('arch-os', {}, triple))
159
160 def test_operators(self):
161 self.assertTrue(BooleanExpression.evaluate('true || true', {}))
162 self.assertTrue(BooleanExpression.evaluate('true || false', {}))
163 self.assertTrue(BooleanExpression.evaluate('false || true', {}))
164 self.assertFalse(BooleanExpression.evaluate('false || false', {}))
165
166 self.assertTrue(BooleanExpression.evaluate('true && true', {}))
167 self.assertFalse(BooleanExpression.evaluate('true && false', {}))
168 self.assertFalse(BooleanExpression.evaluate('false && true', {}))
169 self.assertFalse(BooleanExpression.evaluate('false && false', {}))
170
171 self.assertFalse(BooleanExpression.evaluate('!true', {}))
172 self.assertTrue(BooleanExpression.evaluate('!false', {}))
173
174 self.assertTrue(BooleanExpression.evaluate(' ((!((false) )) ) ', {}))
175 self.assertTrue(BooleanExpression.evaluate('true && (true && (true))', {}))
176 self.assertTrue(BooleanExpression.evaluate('!false && !false && !! !false', {}))
177 self.assertTrue(BooleanExpression.evaluate('false && false || true', {}))
178 self.assertTrue(BooleanExpression.evaluate('(false && false) || true', {}))
179 self.assertFalse(BooleanExpression.evaluate('false && (false || true)', {}))
180
181 # Evaluate boolean expression `expr`.
182 # Fail if it does not throw a ValueError containing the text `error`.
183 def checkException(self, expr, error):
184 try:
185 BooleanExpression.evaluate(expr, {})
186 self.fail("expression %r didn't cause an exception" % expr)
187 except ValueError as e:
188 if -1 == str(e).find(error):
189 self.fail(("expression %r caused the wrong ValueError\n" +
190 "actual error was:\n%s\n" +
191 "expected error was:\n%s\n") % (expr, e, error))
192 except BaseException as e:
193 self.fail(("expression %r caused the wrong exception; actual " +
194 "exception was: \n%r") % (expr, e))
195
196 def test_errors(self):
197 self.checkException("ba#d",
198 "couldn't parse text: '#d'\n" +
199 "in expression: 'ba#d'")
200
201 self.checkException("true and true",
202 "expected: \n" +
203 "have: 'and'\n" +
204 "in expression: 'true and true'")
205
206 self.checkException("|| true",
207 "expected: '!' or '(' or identifier\n" +
208 "have: '||'\n" +
209 "in expression: '|| true'")
210
211 self.checkException("true &&",
212 "expected: '!' or '(' or identifier\n" +
213 "have: \n" +
214 "in expression: 'true &&'")
215
216 self.checkException("",
217 "expected: '!' or '(' or identifier\n" +
218 "have: \n" +
219 "in expression: ''")
220
221 self.checkException("*",
222 "couldn't parse text: '*'\n" +
223 "in expression: '*'")
224
225 self.checkException("no wait stop",
226 "expected: \n" +
227 "have: 'wait'\n" +
228 "in expression: 'no wait stop'")
229
230 self.checkException("no-$-please",
231 "couldn't parse text: '$-please'\n" +
232 "in expression: 'no-$-please'")
233
234 self.checkException("(((true && true) || true)",
235 "expected: ')'\n" +
236 "have: \n" +
237 "in expression: '(((true && true) || true)'")
238
239 self.checkException("true (true)",
240 "expected: \n" +
241 "have: '('\n" +
242 "in expression: 'true (true)'")
243
244 self.checkException("( )",
245 "expected: '!' or '(' or identifier\n" +
246 "have: ')'\n" +
247 "in expression: '( )'")
248
249 if __name__ == '__main__':
250 unittest.main()
0 import os
11 from xml.sax.saxutils import escape
22 from json import JSONEncoder
3
4 from lit.BooleanExpression import BooleanExpression
35
46 # Test result codes.
57
179181 self.path_in_suite = path_in_suite
180182 self.config = config
181183 self.file_path = file_path
182 # A list of conditions under which this test is expected to fail. These
183 # can optionally be provided by test format handlers, and will be
184 # honored when the test result is supplied.
184
185 # A list of conditions under which this test is expected to fail.
186 # Each condition is a boolean expression of features and target
187 # triple parts. These can optionally be provided by test format
188 # handlers, and will be honored when the test result is supplied.
185189 self.xfails = []
190
191 # A list of conditions that must be satisfied before running the test.
192 # Each condition is a boolean expression of features. All of them
193 # must be True for the test to run.
194 # FIXME should target triple parts count here too?
195 self.requires = []
196
197 # A list of conditions that prevent execution of the test.
198 # Each condition is a boolean expression of features and target
199 # triple parts. All of them must be False for the test to run.
200 self.unsupported = []
201
186202 # The test result, once complete.
187203 self.result = None
188204
195211 self.result = result
196212
197213 # Apply the XFAIL handling to resolve the result exit code.
198 if self.isExpectedToFail():
199 if self.result.code == PASS:
200 self.result.code = XPASS
201 elif self.result.code == FAIL:
202 self.result.code = XFAIL
214 try:
215 if self.isExpectedToFail():
216 if self.result.code == PASS:
217 self.result.code = XPASS
218 elif self.result.code == FAIL:
219 self.result.code = XFAIL
220 except ValueError as e:
221 # Syntax error in an XFAIL line.
222 self.result.code = UNRESOLVED
223 self.result.output = str(e)
203224
204225 def getFullName(self):
205226 return self.suite.config.name + ' :: ' + '/'.join(self.path_in_suite)
223244 configuration. This check relies on the test xfails property which by
224245 some test formats may not be computed until the test has first been
225246 executed.
226 """
247 Throws ValueError if an XFAIL line has a syntax error.
248 """
249
250 features = self.config.available_features
251 triple = getattr(self.suite.config, 'target_triple', "")
227252
228253 # Check if any of the xfails match an available feature or the target.
229254 for item in self.xfails:
231256 if item == '*':
232257 return True
233258
234 # If this is an exact match for one of the features, it fails.
235 if item in self.config.available_features:
236 return True
237
238 # If this is a part of the target triple, it fails.
239 if item and item in self.suite.config.target_triple:
240 return True
259 # If this is a True expression of features and target triple parts,
260 # it fails.
261 try:
262 if BooleanExpression.evaluate(item, features, triple):
263 return True
264 except ValueError as e:
265 raise ValueError('Error in XFAIL list:\n%s' % str(e))
241266
242267 return False
268
269 def isWithinFeatureLimits(self):
270 """
271 isWithinFeatureLimits() -> bool
272
273 A test is within the feature limits set by run_only_tests if
274 1. the test's requirements ARE satisfied by the available features
275 2. the test's requirements ARE NOT satisfied after the limiting
276 features are removed from the available features
277
278 Throws ValueError if a REQUIRES line has a syntax error.
279 """
280
281 if not self.config.limit_to_features:
282 return True # No limits. Run it.
283
284 # Check the requirements as-is (#1)
285 if self.getMissingRequiredFeatures():
286 return False
287
288 # Check the requirements after removing the limiting features (#2)
289 featuresMinusLimits = [f for f in self.config.available_features
290 if not f in self.config.limit_to_features]
291 if not self.getMissingRequiredFeaturesFromList(featuresMinusLimits):
292 return False
293
294 return True
295
296 def getMissingRequiredFeaturesFromList(self, features):
297 try:
298 return [item for item in self.requires
299 if not BooleanExpression.evaluate(item, features)]
300 except ValueError as e:
301 raise ValueError('Error in REQUIRES list:\n%s' % str(e))
302
303 def getMissingRequiredFeatures(self):
304 """
305 getMissingRequiredFeatures() -> list of strings
306
307 Returns a list of features from REQUIRES that are not satisfied."
308 Throws ValueError if a REQUIRES line has a syntax error.
309 """
310
311 features = self.config.available_features
312 return self.getMissingRequiredFeaturesFromList(features)
313
314 def getUnsupportedFeatures(self):
315 """
316 getUnsupportedFeatures() -> list of strings
317
318 Returns a list of features from UNSUPPORTED that are present
319 in the test configuration's features or target triple.
320 Throws ValueError if an UNSUPPORTED line has a syntax error.
321 """
322
323 features = self.config.available_features
324 triple = getattr(self.suite.config, 'target_triple', "")
325
326 try:
327 return [item for item in self.unsupported
328 if BooleanExpression.evaluate(item, features, triple)]
329 except ValueError as e:
330 raise ValueError('Error in UNSUPPORTED list:\n%s' % str(e))
243331
244332 def isEarlyTest(self):
245333 """
88 import lit.Test as Test
99 import lit.util
1010 from lit.util import to_bytes, to_string
11 from lit.BooleanExpression import BooleanExpression
1112
1213 class InternalShellError(Exception):
1314 def __init__(self, command, message):
745746 command.
746747
747748 TAG: A keyword taking no value. Ex 'END.'
748 COMMAND: A Keyword taking a list of shell commands. Ex 'RUN:'
749 LIST: A keyword taking a comma separated list of value. Ex 'XFAIL:'
749 COMMAND: A keyword taking a list of shell commands. Ex 'RUN:'
750 LIST: A keyword taking a comma-separated list of values.
751 BOOLEAN_EXPR: A keyword taking a comma-separated list of
752 boolean expressions. Ex 'XFAIL:'
750753 CUSTOM: A keyword with custom parsing semantics.
751754 """
752755 TAG = 0
753756 COMMAND = 1
754757 LIST = 2
755 CUSTOM = 3
758 BOOLEAN_EXPR = 3
759 CUSTOM = 4
760
761 @staticmethod
762 def allowedKeywordSuffixes(value):
763 return { ParserKind.TAG: ['.'],
764 ParserKind.COMMAND: [':'],
765 ParserKind.LIST: [':'],
766 ParserKind.BOOLEAN_EXPR: [':'],
767 ParserKind.CUSTOM: [':', '.']
768 } [value]
769
770 @staticmethod
771 def str(value):
772 return { ParserKind.TAG: 'TAG',
773 ParserKind.COMMAND: 'COMMAND',
774 ParserKind.LIST: 'LIST',
775 ParserKind.BOOLEAN_EXPR: 'BOOLEAN_EXPR',
776 ParserKind.CUSTOM: 'CUSTOM'
777 } [value]
756778
757779
758780 class IntegratedTestKeywordParser(object):
764786 ParserKind.CUSTOM.
765787 """
766788 def __init__(self, keyword, kind, parser=None, initial_value=None):
767 if not keyword.endswith('.') and not keyword.endswith(':'):
768 raise ValueError("keyword '%s' must end with either '.' or ':' "
769 % keyword)
770 if keyword.endswith('.') and kind in \
771 [ParserKind.LIST, ParserKind.COMMAND]:
772 raise ValueError("Keyword '%s' should end in ':'" % keyword)
773
774 elif keyword.endswith(':') and kind in [ParserKind.TAG]:
775 raise ValueError("Keyword '%s' should end in '.'" % keyword)
789 allowedSuffixes = ParserKind.allowedKeywordSuffixes(kind)
790 if len(keyword) == 0 or keyword[-1] not in allowedSuffixes:
791 if len(allowedSuffixes) == 1:
792 raise ValueError("Keyword '%s' of kind '%s' must end in '%s'"
793 % (keyword, ParserKind.str(kind),
794 allowedSuffixes[0]))
795 else:
796 raise ValueError("Keyword '%s' of kind '%s' must end in "
797 " one of '%s'"
798 % (keyword, ParserKind.str(kind),
799 ' '.join(allowedSuffixes)))
800
776801 if parser is not None and kind != ParserKind.CUSTOM:
777802 raise ValueError("custom parsers can only be specified with "
778803 "ParserKind.CUSTOM")
786811 self.parser = self._handleCommand
787812 elif kind == ParserKind.LIST:
788813 self.parser = self._handleList
814 elif kind == ParserKind.BOOLEAN_EXPR:
815 self.parser = self._handleBooleanExpr
789816 elif kind == ParserKind.TAG:
790 if not keyword.endswith('.'):
791 raise ValueError("keyword '%s' should end with '.'" % keyword)
792817 self.parser = self._handleTag
793818 elif kind == ParserKind.CUSTOM:
794819 if parser is None:
798823 raise ValueError("Unknown kind '%s'" % kind)
799824
800825 def parseLine(self, line_number, line):
801 self.parsed_lines += [(line_number, line)]
802 self.value = self.parser(line_number, line, self.value)
826 try:
827 self.parsed_lines += [(line_number, line)]
828 self.value = self.parser(line_number, line, self.value)
829 except ValueError as e:
830 raise ValueError(str(e) + ("\nin %s directive on test line %d" %
831 (self.keyword, line_number)))
803832
804833 def getValue(self):
805834 return self.value
840869 output.extend([s.strip() for s in line.split(',')])
841870 return output
842871
872 @staticmethod
873 def _handleBooleanExpr(line_number, line, output):
874 """A parser for BOOLEAN_EXPR type keywords"""
875 if output is None:
876 output = []
877 output.extend([s.strip() for s in line.split(',')])
878 # Evaluate each expression to verify syntax.
879 # We don't want any results, just the raised ValueError.
880 for s in output:
881 if s != '*':
882 BooleanExpression.evaluate(s, [])
883 return output
884
885 @staticmethod
886 def _handleRequiresAny(line_number, line, output):
887 """A custom parser to transform REQUIRES-ANY: into REQUIRES:"""
888
889 # Extract the conditions specified in REQUIRES-ANY: as written.
890 conditions = []
891 IntegratedTestKeywordParser._handleList(line_number, line, conditions)
892
893 # Output a `REQUIRES: a || b || c` expression in its place.
894 expression = ' || '.join(conditions)
895 IntegratedTestKeywordParser._handleBooleanExpr(line_number,
896 expression, output)
897 return output
843898
844899 def parseIntegratedTestScript(test, additional_parsers=[],
845900 require_script=True):
846901 """parseIntegratedTestScript - Scan an LLVM/Clang style integrated test
847902 script and extract the lines to 'RUN' as well as 'XFAIL' and 'REQUIRES'
848 'REQUIRES-ANY' and 'UNSUPPORTED' information.
903 and 'UNSUPPORTED' information.
849904
850905 If additional parsers are specified then the test is also scanned for the
851906 keywords they specify and all matches are passed to the custom parser.
854909 may be returned. This can be used for test formats where the actual script
855910 is optional or ignored.
856911 """
857 # Collect the test lines from the script.
858 sourcepath = test.getSourcePath()
912
913 # Install the built-in keyword parsers.
859914 script = []
860 requires = []
861 requires_any = []
862 unsupported = []
863915 builtin_parsers = [
864916 IntegratedTestKeywordParser('RUN:', ParserKind.COMMAND,
865917 initial_value=script),
866 IntegratedTestKeywordParser('XFAIL:', ParserKind.LIST,
918 IntegratedTestKeywordParser('XFAIL:', ParserKind.BOOLEAN_EXPR,
867919 initial_value=test.xfails),
868 IntegratedTestKeywordParser('REQUIRES:', ParserKind.LIST,
869 initial_value=requires),
870 IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.LIST,
871 initial_value=requires_any),
872 IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.LIST,
873 initial_value=unsupported),
920 IntegratedTestKeywordParser('REQUIRES:', ParserKind.BOOLEAN_EXPR,
921 initial_value=test.requires),
922 IntegratedTestKeywordParser('REQUIRES-ANY:', ParserKind.CUSTOM,
923 IntegratedTestKeywordParser._handleRequiresAny,
924 initial_value=test.requires),
925 IntegratedTestKeywordParser('UNSUPPORTED:', ParserKind.BOOLEAN_EXPR,
926 initial_value=test.unsupported),
874927 IntegratedTestKeywordParser('END.', ParserKind.TAG)
875928 ]
876929 keyword_parsers = {p.keyword: p for p in builtin_parsers}
930
931 # Install user-defined additional parsers.
877932 for parser in additional_parsers:
878933 if not isinstance(parser, IntegratedTestKeywordParser):
879934 raise ValueError('additional parser must be an instance of '
882937 raise ValueError("Parser for keyword '%s' already exists"
883938 % parser.keyword)
884939 keyword_parsers[parser.keyword] = parser
885
940
941 # Collect the test lines from the script.
942 sourcepath = test.getSourcePath()
886943 for line_number, command_type, ln in \
887944 parseIntegratedTestScriptCommands(sourcepath,
888945 keyword_parsers.keys()):
900957 return lit.Test.Result(Test.UNRESOLVED,
901958 "Test has unterminated run lines (with '\\')")
902959
903 # Check that we have the required features:
904 missing_required_features = [f for f in requires
905 if f not in test.config.available_features]
960 # Enforce REQUIRES:
961 missing_required_features = test.getMissingRequiredFeatures()
906962 if missing_required_features:
907963 msg = ', '.join(missing_required_features)
908964 return lit.Test.Result(Test.UNSUPPORTED,
909 "Test requires the following features: %s"
910 % msg)
911 requires_any_features = [f for f in requires_any
912 if f in test.config.available_features]
913 if requires_any and not requires_any_features:
914 msg = ' ,'.join(requires_any)
915 return lit.Test.Result(Test.UNSUPPORTED,
916 "Test requires any of the following features: "
917 "%s" % msg)
918 unsupported_features = [f for f in unsupported
919 if f in test.config.available_features]
965 "Test requires the following unavailable "
966 "features: %s" % msg)
967
968 # Enforce UNSUPPORTED:
969 unsupported_features = test.getUnsupportedFeatures()
920970 if unsupported_features:
921971 msg = ', '.join(unsupported_features)
922972 return lit.Test.Result(
923973 Test.UNSUPPORTED,
924 "Test is unsupported with the following features: %s" % msg)
925
926 unsupported_targets = [f for f in unsupported
927 if f in test.suite.config.target_triple]
928 if unsupported_targets:
929 return lit.Test.Result(
930 Test.UNSUPPORTED,
931 "Test is unsupported with the following triple: %s" % (
932 test.suite.config.target_triple,))
933
934 if test.config.limit_to_features:
935 # Check that we have one of the limit_to_features features in requires.
936 limit_to_features_tests = [f for f in test.config.limit_to_features
937 if f in requires]
938 if not limit_to_features_tests:
939 msg = ', '.join(test.config.limit_to_features)
940 return lit.Test.Result(
941 Test.UNSUPPORTED,
942 "Test requires one of the limit_to_features features %s" % msg)
974 "Test does not support the following features "
975 "and/or targets: %s" % msg)
976
977 # Enforce limit_to_features.
978 if not test.isWithinFeatureLimits():
979 msg = ', '.join(test.config.limit_to_features)
980 return lit.Test.Result(Test.UNSUPPORTED,
981 "Test does not require any of the features "
982 "specified in limit_to_features: %s" % msg)
983
943984 return script
944985
945986
None RUN: true
1 REQUIRES: a-missing-feature
0 # REQUIRES with a false clause. Test should not run.
1 REQUIRES: true
2 REQUIRES: a-missing-feature, true
3 REQUIRES: true
4 RUN: false
0 # REQUIRES with only true clauses. Test should run.
1 REQUIRES: a-present-feature, true, !not-true
2 REQUIRES: true
3 RUN: true
1 REQUIRES: a-present-feature
0 # '*' only works in XFAIL
1 REQUIRES: *
2 RUN: false
0 # REQUIRES line that uses target triple, which doesn't work. Test should not run
1 REQUIRES: x86_64
2 RUN: false
0 # UNSUPPORTED with only false clauses. Test should run.
1 UNSUPPORTED: false
2 UNSUPPORTED: false, not-true
3 UNSUPPORTED: false
4 UNSUPPORTED: still-not-true
5 UNSUPPORTED: false
6 UNSUPPORTED: false
7 UNSUPPORTED: false
8 RUN: true
0 # UNSUPPORTED with a true clause. Test should not run.
1 UNSUPPORTED: false
2 UNSUPPORTED: false, false, false, _64-unk && a-present-feature, false
3 RUN: false
0 # '*' only works in XFAIL
1 UNSUPPORTED: *
2 RUN: false
0 # XFAIL with only false clauses. Test should run.
1 XFAIL: false, a-missing-feature || ! a-present-feature || ! x86_64, false
2 RUN: true
0 # XFAIL with a true clause. Test should not run.
1 XFAIL: false
2 XFAIL: false, a-present-feature && ! a-missing-feature && x86_64
3 RUN: false
0 # Test the boolean expression parser
1 # used for REQUIRES and UNSUPPORTED and XFAIL
2
3 # RUN: %{python} -m lit.BooleanExpression
4949 # CHECK: PASS: shtest-format :: requires-any-present.txt
5050 # CHECK: UNSUPPORTED: shtest-format :: requires-missing.txt
5151 # CHECK: PASS: shtest-format :: requires-present.txt
52 # CHECK: UNRESOLVED: shtest-format :: requires-star.txt
53 # CHECK: UNSUPPORTED: shtest-format :: requires-triple.txt
54 # CHECK: PASS: shtest-format :: unsupported-expr-false.txt
55 # CHECK: UNSUPPORTED: shtest-format :: unsupported-expr-true.txt
56 # CHECK: UNRESOLVED: shtest-format :: unsupported-star.txt
5257 # CHECK: UNSUPPORTED: shtest-format :: unsupported_dir/some-test.txt
58 # CHECK: PASS: shtest-format :: xfail-expr-false.txt
59 # CHECK: XFAIL: shtest-format :: xfail-expr-true.txt
5360 # CHECK: XFAIL: shtest-format :: xfail-feature.txt
5461 # CHECK: XFAIL: shtest-format :: xfail-target.txt
5562 # CHECK: XFAIL: shtest-format :: xfail.txt
6976 # CHECK: shtest-format :: external_shell/fail_with_bad_encoding.txt
7077 # CHECK: shtest-format :: fail.txt
7178
72 # CHECK: Expected Passes : 5
73 # CHECK: Expected Failures : 3
74 # CHECK: Unsupported Tests : 3
75 # CHECK: Unresolved Tests : 1
79 # CHECK: Expected Passes : 7
80 # CHECK: Expected Failures : 4
81 # CHECK: Unsupported Tests : 5
82 # CHECK: Unresolved Tests : 3
7683 # CHECK: Unexpected Passes : 1
7784 # CHECK: Unexpected Failures: 3
107107 value = custom_parser.getValue()
108108 self.assertItemsEqual(value, ['a', 'b', 'c'])
109109
110 def test_bad_keywords(self):
111 def custom_parse(line_number, line, output):
112 return output
113
114 try:
115 IntegratedTestKeywordParser("TAG_NO_SUFFIX", ParserKind.TAG),
116 self.fail("TAG_NO_SUFFIX failed to raise an exception")
117 except ValueError as e:
118 pass
119 except BaseException as e:
120 self.fail("TAG_NO_SUFFIX raised the wrong exception: %r" % e)
121
122 try:
123 IntegratedTestKeywordParser("TAG_WITH_COLON:", ParserKind.TAG),
124 self.fail("TAG_WITH_COLON: failed to raise an exception")
125 except ValueError as e:
126 pass
127 except BaseException as e:
128 self.fail("TAG_WITH_COLON: raised the wrong exception: %r" % e)
129
130 try:
131 IntegratedTestKeywordParser("LIST_WITH_DOT.", ParserKind.LIST),
132 self.fail("LIST_WITH_DOT. failed to raise an exception")
133 except ValueError as e:
134 pass
135 except BaseException as e:
136 self.fail("LIST_WITH_DOT. raised the wrong exception: %r" % e)
137
138 try:
139 IntegratedTestKeywordParser("CUSTOM_NO_SUFFIX",
140 ParserKind.CUSTOM, custom_parse),
141 self.fail("CUSTOM_NO_SUFFIX failed to raise an exception")
142 except ValueError as e:
143 pass
144 except BaseException as e:
145 self.fail("CUSTOM_NO_SUFFIX raised the wrong exception: %r" % e)
146
147 # Both '.' and ':' are allowed for CUSTOM keywords.
148 try:
149 IntegratedTestKeywordParser("CUSTOM_WITH_DOT.",
150 ParserKind.CUSTOM, custom_parse),
151 except BaseException as e:
152 self.fail("CUSTOM_WITH_DOT. raised an exception: %r" % e)
153 try:
154 IntegratedTestKeywordParser("CUSTOM_WITH_COLON:",
155 ParserKind.CUSTOM, custom_parse),
156 except BaseException as e:
157 self.fail("CUSTOM_WITH_COLON: raised an exception: %r" % e)
158
159 try:
160 IntegratedTestKeywordParser("CUSTOM_NO_PARSER:",
161 ParserKind.CUSTOM),
162 self.fail("CUSTOM_NO_PARSER: failed to raise an exception")
163 except ValueError as e:
164 pass
165 except BaseException as e:
166 self.fail("CUSTOM_NO_PARSER: raised the wrong exception: %r" % e)
110167
111168 if __name__ == '__main__':
112169 TestIntegratedTestKeywordParser.load_keyword_parser_lit_tests()