llvm.org GIT mirror llvm / 64bfc0a
[lit] Raise the default soft process limit when possible It is common to have a default soft process limit, at least on some families of Linux distributions, of 1024. This is normally more than enough, but if you have many cores, and you're running tests that create many threads, this can become a problem. My POWER7 development machine has 48 cores, and when running the lld regression tests, which often want to create up to 48 threads, I run into problems. lit, by default, will want to run 48 tests in parallel, and 48*48 < 1024, and so many tests fail like this: terminate called after throwing an instance of 'std::system_error' what(): Resource temporarily unavailable or lit fails like this when launching a test: OSError: [Errno 11] Resource temporarily unavailable lit can easily detect this situation and attempt to repair it before launching tests (by raising the soft process limit to something that will allow ncpus^2 threads to be created), and should do so to prevent spurious test failures. This is the follow-up to this thread: http://lists.llvm.org/pipermail/llvm-dev/2015-October/090942.html git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@249161 91177308-0d34-0410-b5e6-96231b3b80d8 Hal Finkel 4 years ago
1 changed file(s) with 22 addition(s) and 0 deletion(s). Raw diff Collapse all Expand all
227227 canceled_flag = LockedValue(0)
228228 consumer = ThreadResultsConsumer(display)
230 # Because some tests use threads internally, and at least on Linux each
231 # of these threads counts toward the current process limit, try to
232 # raise the (soft) process limit so that tests don't fail due to
233 # resource exhaustion.
234 try:
235 cpus = lit.util.detectCPUs()
236 desired_limit = jobs * cpus * 2 # the 2 is a safety factor
238 # Import the resource module here inside this try block because it
239 # will likely fail on Windows.
240 import resource
242 max_procs_soft, max_procs_hard = resource.getrlimit(resource.RLIMIT_NPROC)
243 desired_limit = min(desired_limit, max_procs_hard)
245 if max_procs_soft < desired_limit:
246 resource.setrlimit(resource.RLIMIT_NPROC, (desired_limit, max_procs_hard))
247 self.lit_config.note('raised the process limit from %d to %d' % \
248 (max_procs_soft, desired_limit))
249 except:
250 pass
230252 # Create the test provider.
231253 provider = TestProvider(self.tests, jobs, queue_impl, canceled_flag)