llvm.org GIT mirror llvm / eceee13
Updates based on post-commit review of r263834 * Renamed to be camel case, consistent with other docs. * Fixed non-ascii characters (this is what I get for writing docs on an iPad). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@263840 91177308-0d34-0410-b5e6-96231b3b80d8 Chris Bieneman 3 years ago
4 changed file(s) with 176 addition(s) and 176 deletion(s). Raw diff Collapse all Expand all
0 =============================
1 Advanced Build Configurations
2 =============================
3
4 .. contents::
5 :local:
6
7 Introduction
8 ============
9
10 `CMake `_ is a cross-platform build-generator tool. CMake
11 does not build the project, it generates the files needed by your build tool
12 (GNU make, Visual Studio, etc.) for building LLVM.
13
14 If **you are a new contributor**, please start with the :doc:`GettingStarted` or
15 :doc:`CMake` pages. This page is intended for users doing more complex builds.
16
17 Many of the examples below are written assuming specific CMake Generators.
18 Unless otherwise explicitly called out these commands should work with any CMake
19 generator.
20
21 Bootstrap Builds
22 ================
23
24 The Clang CMake build system supports bootstrap (aka multi-stage) builds. At a
25 high level a multi-stage build is a chain of builds that pass data from one
26 stage into the next. The most common and simple version of this is a traditional
27 bootstrap build.
28
29 In a simple two-stage bootstrap build, we build clang using the system compiler,
30 then use that just-built clang to build clang again. In CMake this simplest form
31 of a bootstrap build can be configured with a single option,
32 CLANG_ENABLE_BOOTSTRAP.
33
34 .. code-block:: console
35
36 $ make -G Ninja -DCLANG_ENABLE_BOOTSTRAP=On
37 $ ninja stage2
38
39 This command itself isn't terribly useful because it assumes default
40 configurations for each stage. The next series of examples utilize CMake cache
41 scripts to provide more complex options.
42
43 The clang build system refers to builds as stages. A stage1 build is a standard
44 build using the compiler installed on the host, and a stage2 build is built
45 using the stage1 compiler. This nomenclature holds up to more stages too. In
46 general a stage*n* build is built using the output from stage*n-1*.
47
48 Apple Clang Builds (A More Complex Bootstrap)
49 =============================================
50
51 Apple's Clang builds are a slightly more complicated example of the simple
52 bootstrapping scenario. Apple Clang is built using a 2-stage build.
53
54 The stage1 compiler is a host-only compiler with some options set. The stage1
55 compiler is a balance of optimization vs build time because it is a throwaway.
56 The stage2 compiler is the fully optimized compiler intended to ship to users.
57
58 Setting up these compilers requires a lot of options. To simplify the
59 configuration the Apple Clang build settings are contained in CMake Cache files.
60 You can build an Apple Clang compiler using the following commands:
61
62 .. code-block:: console
63
64 $ make -G Ninja -C /cmake/caches/Apple-stage1.cmake
65 $ ninja stage2-distribution
66
67 This CMake invocation configures the stage1 host compiler, and sets
68 CLANG_BOOTSTRAP_CMAKE_ARGS to pass the Apple-stage2.cmake cache script to the
69 stage2 configuration step.
70
71 When you build the stage2-distribution target it builds the minimal stage1
72 compiler and required tools, then configures and builds the stage2 compiler
73 based on the settings in Apple-stage2.cmake.
74
75 This pattern of using cache scripts to set complex settings, and specifically to
76 make later stage builds include cache scripts is common in our more advanced
77 build configurations.
78
79 Multi-stage PGO
80 ===============
81
82 Profile-Guided Optimizations (PGO) is a really great way to optimize the code
83 clang generates. Our multi-stage PGO builds are a workflow for generating PGO
84 profiles that can be used to optimize clang.
85
86 At a high level, the way PGO works is that you build an instrumented compiler,
87 then you run the instrumented compiler against sample source files. While the
88 instrumented compiler runs it will output a bunch of files containing
89 performance counters (.profraw files). After generating all the profraw files
90 you use llvm-profdata to merge the files into a single profdata file that you
91 can feed into the LLVM_PROFDATA_FILE option.
92
93 Our PGO.cmake cache script automates that whole process. You can use it by
94 running:
95
96 .. code-block:: console
97
98 $ make -G Ninja -C /cmake/caches/PGO.cmake
99 $ ninja stage2-instrumented-generate-profdata
100
101 If you let that run for a few hours or so, it will place a profdata file in your
102 build directory. This takes a really long time because it builds clang twice,
103 and you *must* have compiler-rt in your build tree.
104
105 This process uses any source files under the perf-training directory as training
106 data as long as the source files are marked up with LIT-style RUN lines.
107
108 After it finishes you can use “find . -name clang.profdata” to find it, but it
109 should be at a path something like:
110
111 .. code-block:: console
112
113 /tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata
114
115 You can feed that file into the LLVM_PROFDATA_FILE option when you build your
116 optimized compiler.
117
118 The PGO came cache has a slightly different stage naming scheme than other
119 multi-stage builds. It generates three stages; stage1, stage2-instrumented, and
120 stage2. Both of the stage2 builds are built using the stage1 compiler.
121
122 The PGO came cache generates the following additional targets:
123
124 **stage2-instrumented**
125 Builds a stage1 x86 compiler, runtime, and required tools (llvm-config,
126 llvm-profdata) then uses that compiler to build an instrumented stage2 compiler.
127
128 **stage2-instrumented-generate-profdata**
129 Depends on “stage2-instrumented” and will use the instrumented compiler to
130 generate profdata based on the training files in /utils/perf-training
131
132 **stage2**
133 Depends of “stage2-instrumented-generate-profdata” and will use the stage1
134 compiler with the stage2 profdata to build a PGO-optimized compiler.
135
136 **stage2-check-llvm**
137 Depends on stage2 and runs check-llvm using the stage2 compiler.
138
139 **stage2-check-clang**
140 Depends on stage2 and runs check-clang using the stage2 compiler.
141
142 **stage2-check-all**
143 Depends on stage2 and runs check-all using the stage2 compiler.
144
145 **stage2-test-suite**
146 Depends on stage2 and runs the test-suite using the stage3 compiler (requires
147 in-tree test-suite).
148
149 3-Stage Non-Determinism
150 =======================
151
152 In the ancient lore of compilers non-determinism is like the multi-headed hydra.
153 Whenever it's head pops up, terror and chaos ensue.
154
155 Historically one of the tests to verify that a compiler was deterministic would
156 be a three stage build. The idea of a three stage build is you take your sources
157 and build a compiler (stage1), then use that compiler to rebuild the sources
158 (stage2), then you use that compiler to rebuild the sources a third time
159 (stage3) with an identical configuration to the stage2 build. At the end of
160 this, you have a stage2 and stage3 compiler that should be bit-for-bit
161 identical.
162
163 You can perform one of these 3-stage builds with LLVM & clang using the
164 following commands:
165
166 .. code-block:: console
167
168 $ cmake -G Ninja -C /cmake/caches/3-stage.cmake
169 $ ninja stage3
170
171 After the build you can compare the stage2 & stage3 compilers. We have a bot
172 setup `here `_ that runs
173 this build and compare configuration.
+0
-174
docs/Advanced_Builds.rst less more
None =============================
1 Advanced Build Configurations
2 =============================
3
4 .. contents::
5 :local:
6
7 Introduction
8 ============
9
10 `CMake `_ is a cross-platform build-generator tool. CMake
11 does not build the project, it generates the files needed by your build tool
12 (GNU make, Visual Studio, etc.) for building LLVM.
13
14 If **you are a new contributor**, please start with the :doc:`GettingStarted` or
15 :doc:`CMake` pages. This page is intended for users doing more complex builds.
16
17 Many of the examples below are written assuming specific CMake Generators.
18 Unless otherwise explicitly called out these commands should work with any CMake
19 generator.
20
21 Bootstrap Builds
22 ================
23
24 The Clang CMake build system supports bootstrap (aka multi-stage) builds. At a
25 high level a multi-stage build is a chain of builds that pass data from one
26 stage into the next. The most common and simple version of this is a traditional
27 bootstrap build.
28
29 In a simple two-stage bootstrap build, we build clang using the system compiler,
30 then use that just-built clang to build clang again. In CMake this simplest form
31 of a bootstrap build can be configured with a single option,
32 CLANG_ENABLE_BOOTSTRAP.
33
34 .. code-block:: console
35
36 $ make -G Ninja -DCLANG_ENABLE_BOOTSTRAP=On
37 $ ninja stage2
38
39 This command itself isn’t terribly useful because it assumes default
40 configurations for each stage. The next series of examples utilize CMake cache
41 scripts to provide more complex options.
42
43 The clang build system refers to builds as stages. A stage1 build is a standard
44 build using the compiler installed on the host, and a stage2 build is built
45 using the stage1 compiler. This nomenclature holds up to more stages too. In
46 general a stage*n* build is built using the output from stage*n-1*.
47
48 Apple Clang Builds (A More Complex Bootstrap)
49 =============================================
50
51 Apple’s Clang builds are a slightly more complicated example of the simple
52 bootstrapping scenario. Apple Clang is built using a 2-stage build.
53
54 The stage1 compiler is a host-only compiler with some options set. The stage1
55 compiler is a balance of optimization vs build time because it is a throwaway.
56 The stage2 compiler is the fully optimized compiler intended to ship to users.
57
58 Setting up these compilers requires a lot of options. To simplify the
59 configuration the Apple Clang build settings are contained in CMake Cache files.
60 You can build an Apple Clang compiler using the following commands:
61
62 .. code-block:: console
63
64 $ make -G Ninja -C /cmake/caches/Apple-stage1.cmake
65 $ ninja stage2-distribution
66
67 This CMake invocation configures the stage1 host compiler, and sets
68 CLANG_BOOTSTRAP_CMAKE_ARGS to pass the Apple-stage2.cmake cache script to the
69 stage2 configuration step.
70
71 When you build the stage2-distribution target it builds the minimal stage1
72 compiler and required tools, then configures and builds the stage2 compiler
73 based on the settings in Apple-stage2.cmake.
74
75 This pattern of using cache scripts to set complex settings, and specifically to
76 make later stage builds include cache scripts is common in our more advanced
77 build configurations.
78
79 Multi-stage PGO
80 ===============
81
82 Profile-Guided Optimizations (PGO) is a really great way to optimize the code
83 clang generates. Our multi-stage PGO builds are a workflow for generating PGO
84 profiles that can be used to optimize clang.
85
86 At a high level, the way PGO works is that you build an instrumented compiler,
87 then you run the instrumented compiler against sample source files. While the
88 instrumented compiler runs it will output a bunch of files containing
89 performance counters (.profraw files). After generating all the profraw files
90 you use llvm-profdata to merge the files into a single profdata file that you
91 can feed into the LLVM_PROFDATA_FILE option.
92
93 Our PGO.cmake cache script automates that whole process. You can use it by
94 running:
95
96 .. code-block:: console
97
98 $ make -G Ninja -C /cmake/caches/PGO.cmake
99 $ ninja stage2-instrumented-generate-profdata
100
101 If you let that run for a few hours or so, it will place a profdata file in your
102 build directory. This takes a really long time because it builds clang twice,
103 and you *must* have compiler-rt in your build tree.
104
105 This process uses any source files under the perf-training directory as training
106 data as long as the source files are marked up with LIT-style RUN lines.
107
108 After it finishes you can use “find . -name clang.profdata” to find it, but it
109 should be at a path something like:
110
111 .. code-block:: console
112
113 /tools/clang/stage2-instrumented-bins/utils/perf-training/clang.profdata
114
115 You can feed that file into the LLVM_PROFDATA_FILE option when you build your
116 optimized compiler.
117
118 The PGO came cache has a slightly different stage naming scheme than other
119 multi-stage builds. It generates three stages; stage1, stage2-instrumented, and
120 stage2. Both of the stage2 builds are built using the stage1 compiler.
121
122 The PGO came cache generates the following additional targets:
123
124 **stage2-instrumented**
125 Builds a stage1 x86 compiler, runtime, and required tools (llvm-config,
126 llvm-profdata) then uses that compiler to build an instrumented stage2 compiler.
127
128 **stage2-instrumented-generate-profdata**
129 Depends on “stage2-instrumented” and will use the instrumented compiler to
130 generate profdata based on the training files in /utils/perf-training
131
132 **stage2**
133 Depends of “stage2-instrumented-generate-profdata” and will use the stage1
134 compiler with the stage2 profdata to build a PGO-optimized compiler.
135
136 **stage2-check-llvm**
137 Depends on stage2 and runs check-llvm using the stage2 compiler.
138
139 **stage2-check-clang**
140 Depends on stage2 and runs check-clang using the stage2 compiler.
141
142 **stage2-check-all**
143 Depends on stage2 and runs check-all using the stage2 compiler.
144
145 **stage2-test-suite**
146 Depends on stage2 and runs the test-suite using the stage3 compiler (requires
147 in-tree test-suite).
148
149 3-Stage Non-Determinism
150 =======================
151
152 In the ancient lore of compilers non-determinism is like the multi-headed hydra.
153 Whenever it's head pops up, terror and chaos ensue.
154
155 Historically one of the tests to verify that a compiler was deterministic would
156 be a three stage build. The idea of a three stage build is you take your sources
157 and build a compiler (stage1), then use that compiler to rebuild the sources
158 (stage2), then you use that compiler to rebuild the sources a third time
159 (stage3) with an identical configuration to the stage2 build. At the end of
160 this, you have a stage2 and stage3 compiler that should be bit-for-bit
161 identical.
162
163 You can perform one of these 3-stage builds with LLVM & clang using the
164 following commands:
165
166 .. code-block:: console
167
168 $ cmake -G Ninja -C /cmake/caches/3-stage.cmake
169 $ ninja stage3
170
171 After the build you can compare the stage2 & stage3 compilers. We have a bot
172 setup `here `_ that runs
173 this build and compare configuration.
506506 - It is recommended that all -D options should be specified *before* -C
507507
508508 For more information about some of the advanced build configurations supported
509 via Cache files see :doc:`Advanced_Builds`.
509 via Cache files see :doc:`AdvancedBuilds`.
510510
511511 Executing the test suite
512512 ========================
6464 :hidden:
6565
6666 CMake
67 Advanced_Builds
67 AdvancedBuilds
6868 HowToBuildOnARM
6969 HowToCrossCompileLLVM
7070 CommandGuide/index