llvm.org GIT mirror llvm / 61be1ad
Add flag to llvm-profdata to allow symbols in profile data to be remapped, and add a tool to generate symbol remapping files. Summary: The new tool llvm-cxxmap builds a symbol mapping table from a file containing a description of partial equivalences to apply to mangled names and files containing old and new symbol tables. Reviewers: davidxl Subscribers: mgorny, llvm-commits Differential Revision: https://reviews.llvm.org/D51470 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@342168 91177308-0d34-0410-b5e6-96231b3b80d8 Richard Smith 1 year, 6 days ago
24 changed file(s) with 507 addition(s) and 9 deletion(s). Raw diff Collapse all Expand all
2424 llvm-nm
2525 llvm-objdump
2626 llvm-config
27 llvm-cxxmap
2728 llvm-diff
2829 llvm-cov
2930 llvm-profdata
0 llvm-cxxmap - Mangled name remapping tool
1 =========================================
2
3 SYNOPSIS
4 --------
5
6 :program:`llvm-cxxmap` [*options*] *symbol-file-1* *symbol-file-2*
7
8 DESCRIPTION
9 -----------
10
11 The :program:`llvm-cxxmap` tool performs fuzzy matching of C++ mangled names,
12 based on a file describing name components that should be considered equivalent.
13
14 The symbol files should contain a list of C++ mangled names (one per line).
15 Blank lines and lines starting with ``#`` are ignored. The output is a list
16 of pairs of equivalent symbols, one per line, of the form
17
18 .. code-block:: none
19
20
21
22 where ```` is a symbol from *symbol-file-1* and ```` is
23 a symbol from *symbol-file-2*. Mappings for which the two symbols are identical
24 are omitted.
25
26 OPTIONS
27 -------
28
29 .. program:: llvm-cxxmap
30
31 .. option:: -remapping-file=file, -r=file
32
33 Specify a file containing a list of equivalence rules that should be used
34 to determine whether two symbols are equivalent. Required.
35 See :ref:`remapping-file`.
36
37 .. option:: -output=file, -o=file
38
39 Specify a file to write the list of matched names to. If unspecified, the
40 list will be written to stdout.
41
42 .. option:: -Wambiguous
43
44 Produce a warning if there are multiple equivalent (but distinct) symbols in
45 *symbol-file-2*.
46
47 .. option:: -Wincomplete
48
49 Produce a warning if *symbol-file-1* contains a symbol for which there is no
50 equivalent symbol in *symbol-file-2*.
51
52 .. _remapping-file:
53
54 REMAPPING FILE
55 --------------
56
57 The remapping file is a text file containing lines of the form
58
59 .. code-block:: none
60
61 fragmentkind fragment1 fragment2
62
63 where ``fragmentkind`` is one of ``name``, ``type``, or ``encoding``,
64 indicating whether the following mangled name fragments are
65 <`name `_>s,
66 <`type `_>s, or
67 <`encoding `_>s,
68 respectively.
69 Blank lines and lines starting with ``#`` are ignored.
70
71 For convenience, built-in s such as ``St`` and ``Ss``
72 are accepted as s (even though they technically are not s).
73
74 For example, to specify that ``absl::string_view`` and ``std::string_view``
75 should be treated as equivalent, the following remapping file could be used:
76
77 .. code-block:: none
78
79 # absl::string_view is considered equivalent to std::string_view
80 type N4absl11string_viewE St17basic_string_viewIcSt11char_traitsIcEE
81
82 # std:: might be std::__1:: in libc++ or std::__cxx11:: in libstdc++
83 name St St3__1
84 name St St7__cxx11
85
86 .. note::
87
88 Symbol remapping is currently only supported for C++ mangled names
89 following the Itanium C++ ABI mangling scheme. This covers all C++ targets
90 supported by Clang other than Windows targets.
7373 file are newline-separated. Lines starting with '#' are skipped. Entries may
7474 be of the form or ,.
7575
76 .. option:: -remapping-file=path, -r=path
77
78 Specify a file which contains a remapping from symbol names in the input
79 profile to the symbol names that should be used in the output profile. The
80 file should consist of lines of the form `` ``.
81 Blank lines and lines starting with ``#`` are skipped.
82
83 The :doc:`llvm-cxxmap ` tool can be used to generate the symbol
84 remapping file.
85
7686 .. option:: -instr (default)
7787
7888 Specify that the input profile is an instrumentation-based profile.
0 _ZN1N1fEN1M1X1YE
1 _ZN3foo6detail3qux1fEv
0 _ZN1N1fENS_1X1YE
1 _ZN1N1fEN1M1X1YE
0 _ZN3foo4quux1fEv
1 _ZN1N1fENS_1X1YE
0 _ZN3foo4quux1fEv _ZN3foo6detail3qux1fEv
1 _ZN1N1fENS_1X1YE _ZN1N1fEN1M1X1YE
0 # foo:: and foo::detail:: are equivalent
1 name 3foo N3foo6detailE
2
3 # foo::qux and foo::quux are equivalent
4 type N3foo3quxE N3foo4quuxE
5
6 # N::X and M::X are equivalent
7 name N1N1XE N1M1XE
0 RUN: llvm-cxxmap %S/Inputs/before.sym %S/Inputs/ambiguous.sym -r %S/Inputs/remap.map -o /dev/null -Wambiguous 2>&1 | FileCheck %s
1 CHECK: warning: {{.*}}:2: symbol _ZN1N1fEN1M1X1YE is equivalent to earlier symbol _ZN1N1fENS_1X1YE
0 RUN: llvm-cxxmap %S/Inputs/before.sym %S/Inputs/incomplete.sym -r %S/Inputs/remap.map -o /dev/null -Wincomplete 2>&1 | FileCheck %s
1 CHECK: warning: {{.*}}:2: no new symbol matches old symbol _ZN1N1fENS_1X1YE
0 RUN: llvm-cxxmap %S/Inputs/before.sym %S/Inputs/after.sym -r %S/Inputs/remap.map -o %t.output -Wambiguous -Wincomplete 2>&1 | FileCheck %s --allow-empty
1 RUN: diff %S/Inputs/expected %t.output
2
3 CHECK-NOT: warning
4 CHECK-NOT: error
0 # IR level Instrumentation Flag
1 :ir
2 bar
3 # Func Hash:
4 1234
5 # Num Counters:
6 2
7 # Counter Values:
8 31
9 42
10
11 bar
12 # Func Hash:
13 5678
14 # Num Counters:
15 2
16 # Counter Values:
17 500
18 600
19
20 baz
21 # Func Hash:
22 5678
23 # Num Counters:
24 2
25 # Counter Values:
26 7
27 8
28
0 # :ir is the flag to indicate this is IR level profile.
1 :ir
2 foo
3 1234
4 2
5 1
6 2
7
8 bar
9 1234
10 2
11 30
12 40
13
14 foo
15 5678
16 2
17 500
18 600
19
20 baz
21 5678
22 2
23 7
24 8
0 main:184019:0
1 4: 534
2 4.2: 534
3 5: 1075
4 5.1: 1075
5 6: 2080
6 7: 534
7 9: 2064 _Z3bazi:1471 _Z3fooi:631
8 10: inline2:2000
9 1: 2000
10 10: inline42:1000
11 1: 1000
12 _Z3bazi:40602:2437
13 1: 2437
14 _Z3fooi:7711:610
15 1: 610
0 _Z3bari:20301:1437
1 1: 1437
2 _Z3fooi:7711:610
3 1: 610
4 main:184019:0
5 4: 534
6 4.2: 534
7 5: 1075
8 5.1: 1075
9 6: 2080
10 7: 534
11 9: 2064 _Z3bari:1471 _Z3fooi:631
12 10: inline1:1000
13 1: 1000
14 10: inline2:2000
15 1: 2000
16 _Z3bazi:20301:1000
17 1: 1000
0 _Z3bari _Z3bazi
1 inline1 inline42
0 ; RUN: llvm-profdata merge -text %S/Inputs/instr-remap.proftext -r %S/Inputs/instr-remap.remap -o %t.output
1 ; RUN: diff %S/Inputs/instr-remap.expected %t.output
0 ; RUN: llvm-profdata merge -sample -text %S/Inputs/sample-remap.proftext -r %S/Inputs/sample-remap.remap -o %t.output
1 ; RUN: diff %S/Inputs/sample-remap.expected %t.output
0 set(LLVM_LINK_COMPONENTS
1 Core
2 Support
3 )
4
5 add_llvm_tool(llvm-cxxmap
6 llvm-cxxmap.cpp
7 )
0 ;===- ./tools/llvm-cxxmap/LLVMBuild.txt ------------------------*- Conf -*--===;
1 ;
2 ; The LLVM Compiler Infrastructure
3 ;
4 ; This file is distributed under the University of Illinois Open Source
5 ; License. See LICENSE.TXT for details.
6 ;
7 ;===------------------------------------------------------------------------===;
8 ;
9 ; This is an LLVMBuild description file for the components in this subdirectory.
10 ;
11 ; For more information on the LLVMBuild system, please see:
12 ;
13 ; http://llvm.org/docs/LLVMBuild.html
14 ;
15 ;===------------------------------------------------------------------------===;
16
17 [component_0]
18 type = Tool
19 name = llvm-cxxmap
20 parent = Tools
21 required_libraries = Support
0 //===- llvm-cxxmap.cpp ----------------------------------------------------===//
1 //
2 // The LLVM Compiler Infrastructure
3 //
4 // This file is distributed under the University of Illinois Open Source
5 // License. See LICENSE.TXT for details.
6 //
7 //===----------------------------------------------------------------------===//
8 //
9 // llvm-cxxmap computes a correspondence between old symbol names and new
10 // symbol names based on a symbol equivalence file.
11 //
12 //===----------------------------------------------------------------------===//
13
14 #include "llvm/ADT/DenseSet.h"
15 #include "llvm/ADT/DenseMap.h"
16 #include "llvm/ADT/StringRef.h"
17 #include "llvm/Support/CommandLine.h"
18 #include "llvm/Support/InitLLVM.h"
19 #include "llvm/Support/LineIterator.h"
20 #include "llvm/Support/MemoryBuffer.h"
21 #include "llvm/Support/SymbolRemappingReader.h"
22 #include "llvm/Support/WithColor.h"
23 #include "llvm/Support/raw_ostream.h"
24
25 using namespace llvm;
26
27 cl::opt OldSymbolFile(cl::Positional, cl::Required,
28 cl::desc(""));
29 cl::opt NewSymbolFile(cl::Positional, cl::Required,
30 cl::desc(""));
31 cl::opt RemappingFile("remapping-file", cl::Required,
32 cl::desc("Remapping file"));
33 cl::alias RemappingFileA("r", cl::aliasopt(RemappingFile));
34 cl::opt OutputFilename("output", cl::value_desc("output"),
35 cl::init("-"), cl::desc("Output file"));
36 cl::alias OutputFilenameA("o", cl::aliasopt(OutputFilename));
37
38 cl::opt WarnAmbiguous(
39 "Wambiguous",
40 cl::desc("Warn on equivalent symbols in the output symbol list"));
41 cl::opt WarnIncomplete(
42 "Wincomplete",
43 cl::desc("Warn on input symbols missing from output symbol list"));
44
45 static void warn(Twine Message, Twine Whence = "",
46 std::string Hint = "") {
47 WithColor::warning();
48 std::string WhenceStr = Whence.str();
49 if (!WhenceStr.empty())
50 errs() << WhenceStr << ": ";
51 errs() << Message << "\n";
52 if (!Hint.empty())
53 WithColor::note() << Hint << "\n";
54 }
55
56 static void exitWithError(Twine Message, Twine Whence = "",
57 std::string Hint = "") {
58 WithColor::error();
59 std::string WhenceStr = Whence.str();
60 if (!WhenceStr.empty())
61 errs() << WhenceStr << ": ";
62 errs() << Message << "\n";
63 if (!Hint.empty())
64 WithColor::note() << Hint << "\n";
65 ::exit(1);
66 }
67
68 static void exitWithError(Error E, StringRef Whence = "") {
69 exitWithError(toString(std::move(E)), Whence);
70 }
71
72 static void exitWithErrorCode(std::error_code EC, StringRef Whence = "") {
73 exitWithError(EC.message(), Whence);
74 }
75
76 static void remapSymbols(MemoryBuffer &OldSymbolFile,
77 MemoryBuffer &NewSymbolFile,
78 MemoryBuffer &RemappingFile,
79 raw_ostream &Out) {
80 // Load the remapping file and prepare to canonicalize symbols.
81 SymbolRemappingReader Reader;
82 if (Error E = Reader.read(RemappingFile))
83 exitWithError(std::move(E));
84
85 // Canonicalize the new symbols.
86 DenseMap MappedNames;
87 DenseSet UnparseableSymbols;
88 for (line_iterator LineIt(NewSymbolFile, /*SkipBlanks=*/true, '#');
89 !LineIt.is_at_eof(); ++LineIt) {
90 StringRef Symbol = *LineIt;
91
92 auto K = Reader.insert(Symbol);
93 if (!K) {
94 UnparseableSymbols.insert(Symbol);
95 continue;
96 }
97
98 auto ItAndIsNew = MappedNames.insert({K, Symbol});
99 if (WarnAmbiguous && !ItAndIsNew.second &&
100 ItAndIsNew.first->second != Symbol) {
101 warn("symbol " + Symbol + " is equivalent to earlier symbol " +
102 ItAndIsNew.first->second,
103 NewSymbolFile.getBufferIdentifier() + ":" +
104 Twine(LineIt.line_number()),
105 "later symbol will not be the target of any remappings");
106 }
107 }
108
109 // Figure out which new symbol each old symbol is equivalent to.
110 for (line_iterator LineIt(OldSymbolFile, /*SkipBlanks=*/true, '#');
111 !LineIt.is_at_eof(); ++LineIt) {
112 StringRef Symbol = *LineIt;
113
114 auto K = Reader.lookup(Symbol);
115 StringRef NewSymbol = MappedNames.lookup(K);
116
117 if (NewSymbol.empty()) {
118 if (WarnIncomplete && !UnparseableSymbols.count(Symbol)) {
119 warn("no new symbol matches old symbol " + Symbol,
120 OldSymbolFile.getBufferIdentifier() + ":" +
121 Twine(LineIt.line_number()));
122 }
123 continue;
124 }
125
126 Out << Symbol << " " << NewSymbol << "\n";
127 }
128 }
129
130 int main(int argc, const char *argv[]) {
131 InitLLVM X(argc, argv);
132
133 cl::ParseCommandLineOptions(argc, argv, "LLVM C++ mangled name remapper\n");
134
135 auto OldSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(OldSymbolFile);
136 if (!OldSymbolBufOrError)
137 exitWithErrorCode(OldSymbolBufOrError.getError(), OldSymbolFile);
138
139 auto NewSymbolBufOrError = MemoryBuffer::getFileOrSTDIN(NewSymbolFile);
140 if (!NewSymbolBufOrError)
141 exitWithErrorCode(NewSymbolBufOrError.getError(), NewSymbolFile);
142
143 auto RemappingBufOrError = MemoryBuffer::getFileOrSTDIN(RemappingFile);
144 if (!RemappingBufOrError)
145 exitWithErrorCode(RemappingBufOrError.getError(), RemappingFile);
146
147 std::error_code EC;
148 raw_fd_ostream OS(OutputFilename.data(), EC, sys::fs::F_Text);
149 if (EC)
150 exitWithErrorCode(EC, OutputFilename);
151
152 remapSymbols(*OldSymbolBufOrError.get(), *NewSymbolBufOrError.get(),
153 *RemappingBufOrError.get(), OS);
154 }
122122 }
123123 }
124124
125 namespace {
126 /// A remapper from original symbol names to new symbol names based on a file
127 /// containing a list of mappings from old name to new name.
128 class SymbolRemapper {
129 std::unique_ptr File;
130 DenseMap RemappingTable;
131
132 public:
133 /// Build a SymbolRemapper from a file containing a list of old/new symbols.
134 static std::unique_ptr create(StringRef InputFile) {
135 auto BufOrError = MemoryBuffer::getFileOrSTDIN(InputFile);
136 if (!BufOrError)
137 exitWithErrorCode(BufOrError.getError(), InputFile);
138
139 auto Remapper = llvm::make_unique();
140 Remapper->File = std::move(BufOrError.get());
141
142 for (line_iterator LineIt(*Remapper->File, /*SkipBlanks=*/true, '#');
143 !LineIt.is_at_eof(); ++LineIt) {
144 std::pair Parts = LineIt->split(' ');
145 if (Parts.first.empty() || Parts.second.empty() ||
146 Parts.second.count(' ')) {
147 exitWithError("unexpected line in remapping file",
148 (InputFile + ":" + Twine(LineIt.line_number())).str(),
149 "expected 'old_symbol new_symbol'");
150 }
151 Remapper->RemappingTable.insert(Parts);
152 }
153 return Remapper;
154 }
155
156 /// Attempt to map the given old symbol into a new symbol.
157 ///
158 /// \return The new symbol, or \p Name if no such symbol was found.
159 StringRef operator()(StringRef Name) {
160 StringRef New = RemappingTable.lookup(Name);
161 return New.empty() ? Name : New;
162 }
163 };
164 }
165
125166 struct WeightedFile {
126167 std::string Filename;
127168 uint64_t Weight;
160201 }
161202
162203 /// Load an input into a writer context.
163 static void loadInput(const WeightedFile &Input, WriterContext *WC) {
204 static void loadInput(const WeightedFile &Input, SymbolRemapper *Remapper,
205 WriterContext *WC) {
164206 std::unique_lock CtxGuard{WC->Lock};
165207
166208 // If there's a pending hard error, don't do more work.
191233 }
192234
193235 for (auto &I : *Reader) {
236 if (Remapper)
237 I.Name = (*Remapper)(I.Name);
194238 const StringRef FuncName = I.Name;
195239 bool Reported = false;
196240 WC->Writer.addRecord(std::move(I), Input.Weight, [&](Error E) {
235279 }
236280
237281 static void mergeInstrProfile(const WeightedFileVector &Inputs,
282 SymbolRemapper *Remapper,
238283 StringRef OutputFilename,
239284 ProfileFormat OutputFormat, bool OutputSparse,
240285 unsigned NumThreads) {
266311
267312 if (NumThreads == 1) {
268313 for (const auto &Input : Inputs)
269 loadInput(Input, Contexts[0].get());
314 loadInput(Input, Remapper, Contexts[0].get());
270315 } else {
271316 ThreadPool Pool(NumThreads);
272317
273318 // Load the inputs in parallel (N/NumThreads serial steps).
274319 unsigned Ctx = 0;
275320 for (const auto &Input : Inputs) {
276 Pool.async(loadInput, Input, Contexts[Ctx].get());
321 Pool.async(loadInput, Input, Remapper, Contexts[Ctx].get());
277322 Ctx = (Ctx + 1) % NumThreads;
278323 }
279324 Pool.wait();
321366 }
322367 }
323368
369 /// Make a copy of the given function samples with all symbol names remapped
370 /// by the provided symbol remapper.
371 static sampleprof::FunctionSamples
372 remapSamples(const sampleprof::FunctionSamples &Samples,
373 SymbolRemapper &Remapper, sampleprof_error &Error) {
374 sampleprof::FunctionSamples Result;
375 Result.setName(Remapper(Samples.getName()));
376 Result.addTotalSamples(Samples.getTotalSamples());
377 Result.addHeadSamples(Samples.getHeadSamples());
378 for (const auto &BodySample : Samples.getBodySamples()) {
379 Result.addBodySamples(BodySample.first.LineOffset,
380 BodySample.first.Discriminator,
381 BodySample.second.getSamples());
382 for (const auto &Target : BodySample.second.getCallTargets()) {
383 Result.addCalledTargetSamples(BodySample.first.LineOffset,
384 BodySample.first.Discriminator,
385 Remapper(Target.first()), Target.second);
386 }
387 }
388 for (const auto &CallsiteSamples : Samples.getCallsiteSamples()) {
389 sampleprof::FunctionSamplesMap &Target =
390 Result.functionSamplesAt(CallsiteSamples.first);
391 for (const auto &Callsite : CallsiteSamples.second) {
392 sampleprof::FunctionSamples Remapped =
393 remapSamples(Callsite.second, Remapper, Error);
394 MergeResult(Error, Target[Remapped.getName()].merge(Remapped));
395 }
396 }
397 return Result;
398 }
399
324400 static sampleprof::SampleProfileFormat FormatMap[] = {
325401 sampleprof::SPF_None, sampleprof::SPF_Text, sampleprof::SPF_Compact_Binary,
326402 sampleprof::SPF_GCC, sampleprof::SPF_Binary};
327403
328404 static void mergeSampleProfile(const WeightedFileVector &Inputs,
405 SymbolRemapper *Remapper,
329406 StringRef OutputFilename,
330407 ProfileFormat OutputFormat) {
331408 using namespace sampleprof;
356433 for (StringMap::iterator I = Profiles.begin(),
357434 E = Profiles.end();
358435 I != E; ++I) {
359 StringRef FName = I->first();
360 FunctionSamples &Samples = I->second;
361 sampleprof_error Result = ProfileMap[FName].merge(Samples, Input.Weight);
436 sampleprof_error Result = sampleprof_error::success;
437 FunctionSamples Remapped =
438 Remapper ? remapSamples(I->second, *Remapper, Result)
439 : FunctionSamples();
440 FunctionSamples &Samples = Remapper ? Remapped : I->second;
441 StringRef FName = Samples.getName();
442 MergeResult(Result, ProfileMap[FName].merge(Samples, Input.Weight));
362443 if (Result != sampleprof_error::success) {
363444 std::error_code EC = make_error_code(Result);
364445 handleMergeWriterError(errorCodeToError(EC), Input.Filename, FName);
460541 cl::opt DumpInputFileList(
461542 "dump-input-file-list", cl::init(false), cl::Hidden,
462543 cl::desc("Dump the list of input files and their weights, then exit"));
544 cl::opt RemappingFile("remapping-file", cl::value_desc("file"),
545 cl::desc("Symbol remapping file"));
546 cl::alias RemappingFileA("r", cl::desc("Alias for --remapping-file"),
547 cl::aliasopt(RemappingFile));
463548 cl::opt OutputFilename("output", cl::value_desc("output"),
464549 cl::init("-"), cl::Required,
465550 cl::desc("Output file"));
508593 return 0;
509594 }
510595
596 std::unique_ptr Remapper;
597 if (!RemappingFile.empty())
598 Remapper = SymbolRemapper::create(RemappingFile);
599
511600 if (ProfileKind == instr)
512 mergeInstrProfile(WeightedInputs, OutputFilename, OutputFormat,
513 OutputSparse, NumThreads);
601 mergeInstrProfile(WeightedInputs, Remapper.get(), OutputFilename,
602 OutputFormat, OutputSparse, NumThreads);
514603 else
515 mergeSampleProfile(WeightedInputs, OutputFilename, OutputFormat);
604 mergeSampleProfile(WeightedInputs, Remapper.get(), OutputFilename,
605 OutputFormat);
516606
517607 return 0;
518608 }