llvm.org GIT mirror llvm / 474404b
Migrate dockerfiles to use multi-stage builds. Summary: We previously emulated multi-staged builds using two dockerfiles, native support from Docker allows us to merge them into one, simplifying our scripts. For more details about multi-stage builds, see: https://docs.docker.com/develop/develop-images/multistage-build/ Reviewers: mehdi_amini, klimek, sammccall Reviewed By: sammccall Subscribers: llvm-commits, ioeric, cfe-commits Differential Revision: https://reviews.llvm.org/D44787 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@328503 91177308-0d34-0410-b5e6-96231b3b80d8 Ilya Biryukov 1 year, 6 months ago
12 changed file(s) with 153 addition(s) and 238 deletion(s). Raw diff Collapse all Expand all
5252 LLVM components, compiled from sources. The sources are checked out from the
5353 upstream svn repository when building the image.
5454
55 Inside each subfolder we host Dockerfiles for two images:
55 The resulting image contains only the requested LLVM components and a few extra
56 packages to make the image minimally useful for C++ development, e.g. libstdc++
57 and binutils.
5658
57 - ``build/`` image is used to compile LLVM, it installs a system compiler and all
58 build dependencies of LLVM. After the build process is finished, the build
59 image will have an archive with compiled components at ``/tmp/clang.tar.gz``.
60 - ``release/`` image usually only contains LLVM components, compiled by the
61 ``build/`` image, and also libstdc++ and binutils to make image minimally
62 useful for C++ development. The assumption is that you usually want clang to
63 be one of the provided components.
64
65 To build both of those images, use ``build_docker_image.sh`` script.
66 It will checkout LLVM sources and build clang in the ``build`` container, copy results
67 of the build to the local filesystem and then build the ``release`` container using
68 those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to
69 checkout, and arguments for CMake invocation.
59 The interface to run the build is ``build_docker_image.sh`` script. It accepts a
60 list of LLVM repositories to checkout and arguments for CMake invocation.
7061
7162 If you want to write your own docker image, start with an ``example/`` subfolder.
72 It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps
63 It provides an incomplete Dockerfile with (very few) FIXMEs explaining the steps
7364 you need to take in order to make your Dockerfiles functional.
7465
7566 Usage
109100 -DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \
110101 -DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers"
111102
112 This will produce two images, a release image ``clang-debian8:staging`` and a
113 build image ``clang-debian8-build:staging`` from the latest upstream revision.
114 After the image is built you can run bash inside a container based on your
115 image like this:
103 This will produce a new image ``clang-debian8:staging`` from the latest
104 upstream revision.
105 After the image is built you can run bash inside a container based on your image
106 like this:
116107
117108 .. code-block:: bash
118109
180171
181172 Minimizing docker image size
182173 ============================
183 Due to Docker restrictions we use two images (i.e., build and release folders)
184 for the release image to be as small as possible. It's much easier to achieve
185 that using two images, because Docker would store a filesystem layer for each
186 command in the Dockerfile, i.e. if you install some packages in one command,
187 then remove those in a separate command, the size of the resulting image will
188 still be proportinal to the size of an image with installed packages.
189 Therefore, we strive to provide a very simple release image which only copies
190 compiled clang and does not do anything else.
191
192 Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the
193 image, i.e. remove the parts that were actually deleted. That is an easier way
194 to produce the smallest images possible by using just a single image. We do not
195 use it because as of today the flag is in experimental stage and not everyone
196 may have the latest docker version available. When the flag is out of
197 experimental stage, we should investigate replacing two images approach with
198 just a single image, built using ``--squash`` flag.
174 Due to how Docker's filesystem works, all intermediate writes are persisted in
175 the resulting image, even if they are removed in the following commands.
176 To minimize the resulting image size we use `multi-stage Docker builds
177 `_.
178 Internally Docker builds two images. The first image does all the work: installs
179 build dependencies, checks out LLVM source code, compiles LLVM, etc.
180 The first image is only used during build and does not have a descriptive name,
181 i.e. it is only accessible via the hash value after the build is finished.
182 The second image is our resulting image. It contains only the built binaries
183 and not any build dependencies. It is also accessible via a descriptive name
184 (specified by -d and -t flags).
162162 DOCKER_TAG=":$DOCKER_TAG"
163163 fi
164164
165 echo "Building from $IMAGE_SOURCE"
166 echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG"
167 docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
165 echo "Building ${DOCKER_REPOSITORY}${DOCKER_TAG} from $IMAGE_SOURCE"
166 docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
168167 --build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \
169 -f "$BUILD_DIR/$IMAGE_SOURCE/build/Dockerfile" \
168 -f "$BUILD_DIR/$IMAGE_SOURCE/Dockerfile" \
170169 "$BUILD_DIR"
171
172 echo "Copying clang installation to release image sources"
173 docker run -v "$BUILD_DIR/$IMAGE_SOURCE:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \
174 cp /tmp/clang.tar.gz /workspace/release
175
176 echo "Building release image"
177 docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \
178 "$BUILD_DIR/$IMAGE_SOURCE/release"
179
180170 echo "Done"
0 #===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # Stage 1. Check out LLVM source code and run the build.
9 FROM launcher.gcr.io/google/debian8:latest as builder
10 LABEL maintainer "LLVM Developers"
11 # Install build dependencies of llvm.
12 # First, Update the apt's source list and include the sources of the packages.
13 RUN grep deb /etc/apt/sources.list | \
14 sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
15 # Install compiler, python and subversion.
16 RUN apt-get update && \
17 apt-get install -y --no-install-recommends ca-certificates gnupg \
18 build-essential python wget subversion unzip && \
19 rm -rf /var/lib/apt/lists/*
20 # Install a newer ninja release. It seems the older version in the debian repos
21 # randomly crashes when compiling llvm.
22 RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
23 echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
24 | sha256sum -c && \
25 unzip ninja-linux.zip -d /usr/local/bin && \
26 rm ninja-linux.zip
27 # Import public key required for verifying signature of cmake download.
28 RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684
29 # Download, verify and install cmake version that can compile clang into /usr/local.
30 # (Version in debian8 repos is is too old)
31 RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
32 wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt.asc" && \
33 wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt" && \
34 gpg --verify cmake-3.7.2-SHA-256.txt.asc cmake-3.7.2-SHA-256.txt && \
35 wget "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" && \
36 ( grep "cmake-3.7.2-Linux-x86_64.tar.gz" cmake-3.7.2-SHA-256.txt | \
37 sha256sum -c - ) && \
38 tar xzf cmake-3.7.2-Linux-x86_64.tar.gz -C /usr/local --strip-components=1 && \
39 cd / && rm -rf /tmp/cmake-install
40
41 ADD checksums /tmp/checksums
42 ADD scripts /tmp/scripts
43 # Arguments passed to build_install_clang.sh.
44 ARG buildscript_args
45 # Run the build. Results of the build will be available at /tmp/clang-install/.
46 RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
47
48
49 # Stage 2. Produce a minimal release image with build results.
50 FROM launcher.gcr.io/google/debian8:latest
51 LABEL maintainer "LLVM Developers"
52 # Install packages for minimal useful image.
53 RUN apt-get update && \
54 apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
55 rm -rf /var/lib/apt/lists/*
56 # Copy build results of stage 1 to /usr/local.
57 COPY --from=builder /tmp/clang-install/ /usr/local/
+0
-55
utils/docker/debian8/build/Dockerfile less more
None #===- llvm/utils/docker/debian8/build/Dockerfile -------------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # Produces an image that compiles and archives clang, based on debian8.
9 FROM launcher.gcr.io/google/debian8:latest
10
11 LABEL maintainer "LLVM Developers"
12
13 # Install build dependencies of llvm.
14 # First, Update the apt's source list and include the sources of the packages.
15 RUN grep deb /etc/apt/sources.list | \
16 sed 's/^deb/deb-src /g' >> /etc/apt/sources.list
17
18 # Install compiler, python and subversion.
19 RUN apt-get update && \
20 apt-get install -y --no-install-recommends ca-certificates gnupg \
21 build-essential python wget subversion unzip && \
22 rm -rf /var/lib/apt/lists/*
23
24 # Install a newer ninja release. It seems the older version in the debian repos
25 # randomly crashes when compiling llvm.
26 RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \
27 echo "d2fea9ff33b3ef353161ed906f260d565ca55b8ca0568fa07b1d2cab90a84a07 ninja-linux.zip" \
28 | sha256sum -c && \
29 unzip ninja-linux.zip -d /usr/local/bin && \
30 rm ninja-linux.zip
31
32 # Import public key required for verifying signature of cmake download.
33 RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684
34
35 # Download, verify and install cmake version that can compile clang into /usr/local.
36 # (Version in debian8 repos is is too old)
37 RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \
38 wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt.asc" && \
39 wget "https://cmake.org/files/v3.7/cmake-3.7.2-SHA-256.txt" && \
40 gpg --verify cmake-3.7.2-SHA-256.txt.asc cmake-3.7.2-SHA-256.txt && \
41 wget "https://cmake.org/files/v3.7/cmake-3.7.2-Linux-x86_64.tar.gz" && \
42 ( grep "cmake-3.7.2-Linux-x86_64.tar.gz" cmake-3.7.2-SHA-256.txt | \
43 sha256sum -c - ) && \
44 tar xzf cmake-3.7.2-Linux-x86_64.tar.gz -C /usr/local --strip-components=1 && \
45 cd / && rm -rf /tmp/cmake-install
46
47 ADD checksums /tmp/checksums
48 ADD scripts /tmp/scripts
49
50 # Arguments passed to build_install_clang.sh.
51 ARG buildscript_args
52
53 # Run the build. Results of the build will be available as /tmp/clang.tar.gz.
54 RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+0
-21
utils/docker/debian8/release/Dockerfile less more
None #===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # A release image, containing clang installation, produced by the 'build/' image
9 # and adding libstdc++ and binutils.
10 FROM launcher.gcr.io/google/debian8:latest
11
12 LABEL maintainer "LLVM Developers"
13
14 # Install packages for minimal useful image.
15 RUN apt-get update && \
16 apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \
17 rm -rf /var/lib/apt/lists/*
18
19 # Unpack clang installation into this image.
20 ADD clang.tar.gz /usr/local/
0 #===- llvm/utils/docker/example/build/Dockerfile -------------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # This is an example Dockerfile to build an image that compiles clang.
9 # Replace FIXMEs to prepare your own image.
10
11 # Stage 1. Check out LLVM source code and run the build.
12 # FIXME: Replace 'ubuntu' with your base image
13 FROM ubuntu as builder
14 # FIXME: Change maintainer name
15 LABEL maintainer "Maintainer "
16 # FIXME: Install llvm/clang build dependencies here. Including compiler to
17 # build stage1, cmake, subversion, ninja, etc.
18
19 ADD checksums /tmp/checksums
20 ADD scripts /tmp/scripts
21 # Arguments passed to build_install_clang.sh.
22 ARG buildscript_args
23 # Run the build. Results of the build will be available as /tmp/clang-install.
24 RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
25
26
27 # Stage 2. Produce a minimal release image with build results.
28 # FIXME: Replace 'ubuntu' with your base image.
29 FROM ubuntu
30 # FIXME: Change maintainer name.
31 LABEL maintainer "Maintainer "
32 # FIXME: Install all packages you want to have in your release container.
33 # A minimal useful installation should include at least libstdc++ and binutils.
34
35 # Copy build results of stage 1 to /usr/local.
36 COPY --from=builder /tmp/clang-install/ /usr/local/
+0
-28
utils/docker/example/build/Dockerfile less more
None #===- llvm/utils/docker/example/build/Dockerfile -------------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # This is an example Dockerfile to build an image that compiles clang.
9 # Replace FIXMEs to prepare your own image.
10
11 # FIXME: Replace 'ubuntu' with your base image
12 FROM ubuntu
13
14 # FIXME: Change maintainer name
15 LABEL maintainer "Maintainer "
16
17 # FIXME: Install llvm/clang build dependencies. Including compiler to
18 # build stage1, cmake, subversion, ninja, etc.
19
20 ADD checksums /tmp/checksums
21 ADD scripts /tmp/scripts
22
23 # Arguments passed to build_install_clang.sh.
24 ARG buildscript_args
25
26 # Run the build. Results of the build will be available as /tmp/clang.tar.gz.
27 RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+0
-24
utils/docker/example/release/Dockerfile less more
None #===- llvm/utils/docker/example/release/Dockerfile -----------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # An image that unpacks a clang installation, compiled by the 'build/'
9 # container.
10 # Replace FIXMEs to prepare your own image.
11
12 # FIXME: Replace 'ubuntu' with your base image.
13 FROM ubuntu
14
15 # FIXME: Change maintainer name.
16 LABEL maintainer "Maintainer "
17
18 # FIXME: Install all packages you want to have in your release container.
19 # A minimal useful installation must include libstdc++ and binutils.
20
21 # Unpack clang installation into this container.
22 # It is copied to this directory by build_docker_image.sh script.
23 ADD clang.tar.gz /usr/local/
0 #===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # Stage 1. Check out LLVM source code and run the build.
9 FROM nvidia/cuda:8.0-devel as builder
10 LABEL maintainer "LLVM Developers"
11 # Install llvm build dependencies.
12 RUN apt-get update && \
13 apt-get install -y --no-install-recommends ca-certificates cmake python \
14 subversion ninja-build && \
15 rm -rf /var/lib/apt/lists/*
16
17 ADD checksums /tmp/checksums
18 ADD scripts /tmp/scripts
19 # Arguments passed to build_install_clang.sh.
20 ARG buildscript_args
21 # Run the build. Results of the build will be available at /tmp/clang-install/.
22 RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
23
24
25 # Stage 2. Produce a minimal release image with build results.
26 FROM nvidia/cuda:8.0-devel
27 LABEL maintainer "LLVM Developers"
28 # Copy clang installation into this container.
29 COPY --from=builder /tmp/clang-install/ /usr/local/
30 # C++ standard library and binutils are already included in the base package.
+0
-31
utils/docker/nvidia-cuda/build/Dockerfile less more
None #===- llvm/utils/docker/nvidia-cuda/build/Dockerfile ---------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # Produces an image that compiles and archives clang, based on nvidia/cuda
9 # image.
10 FROM nvidia/cuda:8.0-devel
11
12 LABEL maintainer "LLVM Developers"
13
14 # Arguments to pass to build_install_clang.sh.
15 ARG buildscript_args
16
17 # Install llvm build dependencies.
18 RUN apt-get update && \
19 apt-get install -y --no-install-recommends ca-certificates cmake python \
20 subversion ninja-build && \
21 rm -rf /var/lib/apt/lists/*
22
23 ADD checksums /tmp/checksums
24 ADD scripts /tmp/scripts
25
26 # Arguments passed to build_install_clang.sh.
27 ARG buildscript_args
28
29 # Run the build. Results of the build will be available as /tmp/clang.tar.gz.
30 RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args}
+0
-23
utils/docker/nvidia-cuda/release/Dockerfile less more
None #===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===//
1 #
2 # The LLVM Compiler Infrastructure
3 #
4 # This file is distributed under the University of Illinois Open Source
5 # License. See LICENSE.TXT for details.
6 #
7 #===----------------------------------------------------------------------===//
8 # This is an example Dockerfile that copies a clang installation, compiled
9 # by the 'build/' container into a fresh docker image to get a container of
10 # minimal size.
11 # Replace FIXMEs to prepare a new Dockerfile.
12
13 # FIXME: Replace 'ubuntu' with your base image.
14 FROM nvidia/cuda:8.0-devel
15
16 # FIXME: Change maintainer name.
17 LABEL maintainer "LLVM Developers"
18
19 # Unpack clang installation into this container.
20 ADD clang.tar.gz /usr/local/
21
22 # C++ standard library and binutils are already included in the base package.
1515
1616 Checkout svn sources and run cmake with the specified arguments. Used
1717 inside docker container.
18 Passes additional -DCMAKE_INSTALL_PREFIX and archives the contents of
19 the directory to /tmp/clang.tar.gz.
18 Passes additional -DCMAKE_INSTALL_PREFIX and puts the build results into
19 /tmp/clang-install/ directory.
2020
2121 Available options:
2222 -h|--help show this help message
243243
244244 popd
245245
246 # Pack the installed clang into an archive.
247 echo "Archiving clang installation to /tmp/clang.tar.gz"
248 cd "$CLANG_INSTALL_DIR"
249 tar -czf /tmp/clang.tar.gz *
250
251246 # Cleanup.
252 rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR"
247 rm -rf "$CLANG_BUILD_DIR"
253248
254249 echo "Done"