Search the Community

Docker images play a pivotal role in containerized application deployment. They encapsulate your application and its dependencies, ensuring consistent and efficient deployment across various environments. However, security is a paramount concern when working with Docker images. In this guide, we will explore security best practices for Docker images to help you create and maintain secure images for your containerized applications. 1. Introduction The Significance of Docker Images Docker images are at the core of containerization, offering a standardized approach to packaging applications and their dependencies. They allow developers to work in controlled environments and empower DevOps teams to deploy applications consistently across various platforms. However, the advantages of Docker images come with security challenges, making it essential to adopt best practices to protect your containerized applications. View the full article

Author: Kevin Hannon (Red Hat) A common issue in running/operating Kubernetes clusters is running out of disk space. When the node is provisioned, you should aim to have a good amount of storage space for your container images and running containers. The container runtime usually writes to /var. This can be located as a separate partition or on the root filesystem. CRI-O, by default, writes its containers and images to /var/lib/containers, while containerd writes its containers and images to /var/lib/containerd. In this blog post, we want to bring attention to ways that you can configure your container runtime to store its content separately from the default partition. This allows for more flexibility in configuring Kubernetes and provides support for adding a larger disk for the container storage while keeping the default filesystem untouched. One area that needs more explaining is where/what Kubernetes is writing to disk. Understanding Kubernetes disk usage Kubernetes has persistent data and ephemeral data. The base path for the kubelet and local Kubernetes-specific storage is configurable, but it is usually assumed to be /var/lib/kubelet. In the Kubernetes docs, this is sometimes referred to as the root or node filesystem. The bulk of this data can be categorized into: ephemeral storage logs and container runtime This is different from most POSIX systems as the root/node filesystem is not / but the disk that /var/lib/kubelet is on. Ephemeral storage Pods and containers can require temporary or transient local storage for their operation. The lifetime of the ephemeral storage does not extend beyond the life of the individual pod, and the ephemeral storage cannot be shared across pods. Logs By default, Kubernetes stores the logs of each running container, as files within /var/log. These logs are ephemeral and are monitored by the kubelet to make sure that they do not grow too large while the pods are running. You can customize the log rotation settings for each node to manage the size of these logs, and configure log shipping (using a 3rd party solution) to avoid relying on the node-local storage. Container runtime The container runtime has two different areas of storage for containers and images. read-only layer: Images are usually denoted as the read-only layer, as they are not modified when containers are running. The read-only layer can consist of multiple layers that are combined into a single read-only layer. There is a thin layer on top of containers that provides ephemeral storage for containers if the container is writing to the filesystem. writeable layer: Depending on your container runtime, local writes might be implemented as a layered write mechanism (for example, overlayfs on Linux or CimFS on Windows). This is referred to as the writable layer. Local writes could also use a writeable filesystem that is initialized with a full clone of the container image; this is used for some runtimes based on hypervisor virtualisation. The container runtime filesystem contains both the read-only layer and the writeable layer. This is considered the imagefs in Kubernetes documentation. Container runtime configurations CRI-O CRI-O uses a storage configuration file in TOML format that lets you control how the container runtime stores persistent and temporary data. CRI-O utilizes the storage library. Some Linux distributions have a manual entry for storage (man 5 containers-storage.conf). The main configuration for storage is located in /etc/containers/storage.conf and one can control the location for temporary data and the root directory. The root directory is where CRI-O stores the persistent data. [storage] # Default storage driver driver = "overlay" # Temporary storage location runroot = "/var/run/containers/storage" # Primary read/write location of container storage graphroot = "/var/lib/containers/storage" graphroot Persistent data stored from the container runtime If SELinux is enabled, this must match the /var/lib/containers/storage runroot Temporary read/write access for container Recommended to have this on a temporary filesystem Here is a quick way to relabel your graphroot directory to match /var/lib/containers/storage: semanage fcontext -a -e /var/lib/containers/storage <YOUR-STORAGE-PATH> restorecon -R -v <YOUR-STORAGE-PATH> containerd The containerd runtime uses a TOML configuration file to control where persistent and ephemeral data is stored. The default path for the config file is located at /etc/containerd/config.toml. The relevant fields for containerd storage are root and state. root The root directory for containerd metadata Default is /var/lib/containerd Root also requires SELinux labels if your OS requires it state Temporary data for containerd Default is /run/containerd Kubernetes node pressure eviction Kubernetes will automatically detect if the container filesystem is split from the node filesystem. When one separates the filesystem, Kubernetes is responsible for monitoring both the node filesystem and the container runtime filesystem. Kubernetes documentation refers to the node filesystem and the container runtime filesystem as nodefs and imagefs. If either nodefs or the imagefs are running out of disk space, then the overall node is considered to have disk pressure. Kubernetes will first reclaim space by deleting unusued containers and images, and then it will resort to evicting pods. On a node that has a nodefs and an imagefs, the kubelet will garbage collect unused container images on imagefs and will remove dead pods and their containers from the nodefs. If there is only a nodefs, then Kubernetes garbage collection includes dead containers, dead pods and unused images. Kubernetes allows more configurations for determining if your disk is full. The eviction manager within the kubelet has some configuration settings that let you control the relevant thresholds. For filesystems, the relevant measurements are nodefs.available, nodefs.inodesfree, imagefs.available, and imagefs.inodesfree. If there is not a dedicated disk for the container runtime then imagefs is ignored. Users can use the existing defaults: memory.available < 100MiB nodefs.available < 10% imagefs.available < 15% nodefs.inodesFree < 5% (Linux nodes) Kubernetes allows you to set user defined values in EvictionHard and EvictionSoft in the kubelet configuration file. EvictionHard defines limits; once these limits are exceeded, pods will be evicted without any grace period. EvictionSoft defines limits; once these limits are exceeded, pods will be evicted with a grace period that can be set per signal. If you specify a value for EvictionHard, it will replace the defaults. This means it is important to set all signals in your configuration. For example, the following kubelet configuration could be used to configure eviction signals and grace period options. apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration address: "192.168.0.8" port: 20250 serializeImagePulls: false evictionHard: memory.available: "100Mi" nodefs.available: "10%" nodefs.inodesFree: "5%" imagefs.available: "15%" imagefs.inodesFree: "5%" evictionSoft: memory.available: "100Mi" nodefs.available: "10%" nodefs.inodesFree: "5%" imagefs.available: "15%" imagefs.inodesFree: "5%" evictionSoftGracePeriod: memory.available: "1m30s" nodefs.available: "2m" nodefs.inodesFree: "2m" imagefs.available: "2m" imagefs.inodesFree: "2m" evictionMaxPodGracePeriod: 60s Problems The Kubernetes project recommends that you either use the default settings for eviction or you set all the fields for eviction. You can use the default settings or specify your own evictionHard settings. If you miss a signal, then Kubernetes will not monitor that resource. One common misconfiguration administrators or users can hit is mounting a new filesystem to /var/lib/containers/storage or /var/lib/containerd. Kubernetes will detect a separate filesystem, so you want to make sure to check that imagefs.inodesfree and imagefs.available match your needs if you've done this. Another area of confusion is that ephemeral storage reporting does not change if you define an image filesystem for your node. The image filesystem (imagefs) is used to store container image layers; if a container writes to its own root filesystem, that local write doesn't count towards the size of the container image. The place where the container runtime stores those local modifications is runtime-defined, but is often the image filesystem. If a container in a pod is writing to a filesystem-backed emptyDir volume, then this uses space from the nodefs filesystem. The kubelet always reports ephemeral storage capacity and allocations based on the filesystem represented by nodefs; this can be confusing when ephemeral writes are actually going to the image filesystem. Future work To fix the ephemeral storage reporting limitations and provide more configuration options to the container runtime, SIG Node are working on KEP-4191. In KEP-4191, Kubernetes will detect if the writeable layer is separated from the read-only layer (images). This would allow us to have all ephemeral storage, including the writeable layer, on the same disk as well as allowing for a separate disk for images. Getting involved If you would like to get involved, you can join Kubernetes Node Special-Interest-Group (SIG). If you would like to share feedback, you can do so on our #sig-node Slack channel. If you're not already part of that Slack workspace, you can visit https://slack.k8s.io/ for an invitation. Special thanks to all the contributors who provided great reviews, shared valuable insights or suggested the topic idea. Peter Hunt Mrunal Patel Ryan Phillips Gaurav Singh View the full article

Introduction Many applications built today or modernized from monoliths are done so using microservice architectures. The microservice architecture makes applications easier to scale and faster to develop, which enables innovation and accelerating time-to-market for new features. In addition, microservices also provide lifecycle autonomy enabling applications to have independent build and deploy processes, which provides technological freedom such that they can be implemented in different programming languages and provide scaling flexibility to scale up or scale down independently based on workload utilization. While microservices provide a lot of flexibility, the process of building and deploying them, ensuring the right application versions, and required dependencies are used is a tedious process. This is where containers come in. Microservices can be packaged into a single lightweight and standalone executable artifact called container image that includes everything to run an application. This process of packaging a microservice into container image is called containerization. Containerization offers a lot of benefits, such as portability, which allow containers to be deployed to different infrastructures. It also offers fault isolation, which ensures that different containers are running as isolated process within their own user space in the host OS so that one container’s crash or failure wouldn’t impact the other and provide ease-of-management for deployment and version management. With the benefits that are offered via microservices and subsequent containerization, creation of container images have increased at a rapid scale. As the use of containerized applications continue to grow, it is important to ensure that your container images are optimized, secure, and reliable. Taking these tenets into account in this post, we discuss best practices for building better container images for use on Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (Amazon EKS), and other services. For the container image repositories, we focus on Amazon Elastic Container Registry (Amazon ECR). Building better images We’ll look at the following approaches to build better images. While this isn’t an exhaustive list, these topics provide a good base for your image builds, and you can adopt them as you wish. Walkthrough Use trusted images and source Using a base image from a trusted source can improve the security and reliability of your container. This is because you can be confident that the base image has been thoroughly tested and vetted. It can also reduce the burden of establishing provenance, because you only need to consider the packages and libraries that you include in your image, rather than the entire base image. Here is an example creating a Dockerfile using the official Python base image from the Amazon ECR repository. In fact, all of the Docker official images are available on Amazon ECR public gallery. Note: The code samples in the blog are for Docker. If you don’t have Docker, then please refer to Docker installation guide for information about how to install Docker on your particular operating system. Alternatively, you can also consider using Finch an open source client for container development. FROM public.ecr.aws/docker/library/python:slim # Install necessary packages RUN pip install flask COPY app.py /app/ ENTRYPOINT ["python", "/app/app.py"] Here’s another example of using latest version (as of the data of this post) of Amazon Linux image, which is a secure and lightweight Linux distribution provided by AWS. FROM public.ecr.aws/amazonlinux/amazonlinux:2023 It is important to keep images up-to-date by regularly updating to latest secure versions of the software and libraries included in the image. As new versions of the images get created, they should be explicitly tagged with versions such as v1, v2, rc_03122023, etc instead of tagging as latest. Using explicit tags instead of latest tag could prevent situations where the image with latest tag isn’t actually updated and instead gives a false appearance that the image contains the latest version of the application. If you’re confident in your automation, then vanity tags such as latest or prod might be acceptable to use, but avoiding them also reduces ambiguity. This can avoid confusion about which application version may actually being used. Once images are created, they can be pushed into the Amazon ECR repository for secure storage and highly available distribution. Amazon ECR encrypts data at rest and offers lifecycle management, replication, and caching features. Amazon ECR can also scan the images to help in identifying software vulnerabilities through Basic and Enhanced Scanning. Stored images from Amazon ECR can then be pulled and run by services such Amazon ECS, Amazon EKS, or other services and tools. Here is an example of using AWS Command Line Interface (AWS CLI) and Docker CLI to pull a versioned image from Amazon ECR. Step 1 – Authenticate your Docker client to the Amazon Linux Public registry. Authentication tokens are valid for 12 hours. For more information, see Private registry authentication. Alternatively, you can also use Amazon ECR Docker Credential Helper, which is a credential helper for the Docker daemon that makes it easier to use Amazon Elastic Container Registry. Amazon ECR Docker credential helper automatically gets credentials for Amazon ECR on docker push and docker pull. Note that this would only be required for Amazon ECR, but ECR Public doesn’t need authentication. $ aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws Step 2 – Pull the Amazon Linux container image using the docker pull command. To view the Amazon Linux container image on the Amazon ECR Public Gallery, see Amazon ECR Public Gallery – amazonlinux. docker pull public.ecr.aws/amazonlinux/amazonlinux:2023 Step 3 – Run the container locally. docker run -it public.ecr.aws/amazonlinux/amazonlinux /bin/bash Bonus While Amazon Linux is a secure, trusted, and lightweight container image, AWS also offers purpose built container image called Bottlerocket, which is a Linux-based open-source operating system. Bottlerocket includes only the essential software required to run containers, and ensures that the underlying software is always secure. Additionally, Bottlerocket is available at no cost as an Amazon Machine Image (AMI) for Amazon Elastic Compute Cloud (Amazon EC2) and can be used on Amazon EKS and Amazon ECS setups. Sign container images Container image signing can help verify that the trusted images you have selected and vetted are in use throughout your build pipelines and deployments. This process involves trusted parties cryptographically signing images so that they can be verified when used. This can be used to also sign and verify images throughout your organization. Container image signing is fairly lightweight. Because container images and runtimes have built-in integrity checks and all image content is immutable, signing solutions can simply sign image manifests. Signatures are stored alongside images in the registry, and at any point in time a consumer of the image can retrieve its signature and verify against a trusted publisher’s identity or public key. It is a good practice to sign and verify container images as part of overall security practices, and verifying public content establishes trust in content authenticity. With signed images, you can implement a solution that blocks images from running in your container environment unless they can be verified as trusted. This not only guarantees the authenticity of container images but also reduces the need for additional work to validate the container images in other ways prior to their use. For a full walkthrough of such a solution, see the recent launch of Container Image Signing with AWS Signer and Amazon EKS. Limit the number of layers It is a good practice to limit the number of layers in your container images. Having a large number of layers can increase the size and complexity of the image, which can make it more difficult to manage and maintain. For example, consider the following Dockerfile: FROM public.ecr.aws/docker/library/alpine:3.17.2 # Install all necessary dependencies in a single layer RUN apk add --no-cache \ curl \ nginx \ && rm -rf /var/cache/apk/* # Set nginx as the entrypoint ENTRYPOINT ["nginx"] In this example, we install all necessary dependencies in a single layer, and then remove the cache to reduce the number of layers in the final image. This results in a smaller and more efficient image that is easier to manage and maintain. Make use of multi-stage builds A multi-stage build is a technique that allows you to separate the build tools and dependencies from the final image content. This can be beneficial for several reasons: Reducing image size: By separating the build stage from the runtime stage, you can include only the necessary dependencies in the final image, rather than including all of the build tools and libraries. This can significantly reduce the size of your final image as and reduce the total number of image layers. Improved security: By including only the necessary dependencies in the final image, you can reduce the attack surface of the image. This is because there are fewer packages and libraries that could potentially have vulnerabilities. Here is an example of a multi-stage build in a Dockerfile: # Build stage FROM public.ecr.aws/bitnami/golang:1.18.10 as builder WORKDIR /app COPY . . RUN go build -o app . # Runtime stage FROM public.ecr.aws/amazonlinux/amazonlinux:2023 RUN yum install -y go WORKDIR /app COPY --from=builder/app ./ ENTRYPOINT ["./app"] In this example, we first use the golang:1.18.10 image as the build stage, in which we copy the source code and build the binary. Then, in the runtime stage, we use the amazonlinux:2023 container image, which is a minimal base image, and copy the built binary from build stage. This results in a smaller and more secure final image that only includes the necessary dependencies. Secure your secrets Secrets management is a technique for securely storing and managing sensitive information such as passwords and encryption keys and also externalizing environment specific application configuration. Secrets should not be stored in an image but instead stored and managed externally in service such as AWS Secrets Manager. For secrets that are required for your application during runtime, you can retrieve them from AWS Secrets Manager in real-time or use container orchestrator service such as Amazon ECS or Amazon EKS to mount secrets as volumes on the container such that the application can read from the mounted volume. Details on how secrets can be used on Amazon EKS can be found here and how secrets can be used on Amazon ECS can be found here. If secrets are required during build time, then you can inject ephemeral secrets using Docker’s secret mount-type. For example, the following Dockerfile uses a multi-stage build process where the first stage called builder_image_stage uses mount=type=secret to load the AWS credentials. The second stage then uses the build artifacts from first builder_image_stage. The resulting final image from second stage won’t contain any build tools or secrets from the builder_image_stage thereby not maintaining secrets on it. FROM public.ecr.aws/amazonlinux/amazonlinux:2023 AS builder_image_stage WORKDIR /tmp RUN yum install -y unzip && \ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && \ unzip /tmp/awscliv2.zip && \ /tmp/aws/install RUN --mount=type=secret,id=aws,target=/root/.aws/credentials \ aws s3 cp s3://... /app/... # cd /app # make FROM public.ecr.aws/amazonlinux/amazonlinux:2023 COPY --from=builder_image_stage /app/files_from_builder_image /app Reduce the attack surface Images, by default, are not secure and can potentially be vulnerable for attacks. Following approaches can help mitigate the attack surface. Build from scratch Most images are based on an existing base image such as Amazon Linux, ubuntu, alpine, etc. However, if you need to create a completely new base image with your own distribution, or if your application or service can be deployed as a single statically linked binary, you can use the special scratch image as a starting point. The scratch base image is empty and doesn’t contain any files. All of the official base images are actually built on top of scratch, and you can place any content on subsequent image layers. Using FROM scratch is basically a signal to the image build process that the layer built by the next command in the Dockerfile should be the first layer of the image. Scratch gives you the ability to completely control the contents of your image starting from the beginning. It is important to note that using scratch, you’ll be responsible for the entire toolchain and all content in the resulting container image. Also, container scanning solutions, such as Amazon ECR Enhanced Scanning, do not support vulnerability scanning of images without an OS packaging system in place. If your container images have many environmental dependencies or use interpreted languages or need vulnerability scanning, then using a minimal base image instead of scratch may be better. Here are a few examples of using scratch. The example below adds hello-world executable to the image as a first layer. FROM scratch ADD hello-world / CMD ["/hello-wold"] The second example copies the root filesystem to the root of the image as a first layer. The container uses this root filesystem to spin off processes from binaries and libraries in that directory. FROM scratch ADD root-filesystem.tar.gz / CMD ["bash"] Remove unwanted packages It is a good practice to remove any unwanted or unnecessary packages from your container image to reduce the attack surface and size of the image. For example, if your application does not require certain libraries or utilities, you should remove them from the image. Here is an example on how to remove package manager that may be unnecessary on a final image. FROM public.ecr.aws/ubuntu/ubuntu:20.04_stable # Remove package manager RUN apt-get -y --allow-remove-essential remove apt Use a Minimal Base Image Choosing a minimal base image, such as a Distroless image, can help to restrict what is included in your runtime container. Distroless images only include your application and its runtime dependencies that is necessary for the application to run. As such they don’t contain package managers or shells. This can improve the security and performance of your container by reducing the attack surface, as there are fewer packages and libraries that could potentially have vulnerabilities or add unnecessary overhead. Here is an example of using the Distroless base image for a Go application. This base image contains a minimal Linux, glibc-based system and is intended for use directly by mostly-statically compiled languages like Go, Rust or D. For more information, see the GitHub documentation on base distroless image. FROM gcr.io/distroless/base COPY app /app ENTRYPOINT ["/app"] Secure image configuration Images by default may not be secure and can allow privileged access. It is best to ensure that they are setup with least privileged access and remove configurations that are unnecessary for your application. One such approach is to run containers as a non-root user. Processes within Docker containers have root privileges by default to both the container and the underlying host. This opens up the container and host to security vulnerabilities that can be exploited. To prevent these vulnerabilities, it is a good practice to minimize the privileges granted to your container images, only giving the necessary permissions to perform required tasks. This helps to reduce the attack surface and potential for privilege escalation. On Dockerfile, you can use USER directive to specify a non-root user to run the container, or use the securityContext field in the Kubernetes pod specification to specify a non-root user and set group ownership and file permissions. Furthermore, on Kubernetes with Amazon EKS, you can also limit the default capabilities assigned to a POD as explained on EKS best practices Guides. For example, consider the following Dockerfile: FROM public.ecr.aws/amazonlinux/amazonlinux:2023 # 1) install package for adduser # 2) create new non-root user named myuser # 3) create app folder # 4) remove package for adduser RUN yum install -y shadow-utils && \ adduser myuser && \ mkdir /app && \ yum install -y remove shadow-utils COPY . /app # Set group ownership and file permissions RUN chown -R myuser:myuser /app # Run as non-root user USER myuser # Set entrypoint ENTRYPOINT ["/app/myapp"] In this example, we create a non-root user named myuser and set the group ownership and file permissions of the application files to this user. We then set the entry point to run as the myuser user, rather than as root. This helps to reduce the potential for privilege escalation and increase the security of the container. Tag Images When building container images, it is recommended to use tags to identify them. If the image is built without any tags, then Docker will assign latest as the default tag. For instance, when building an image, you may use the following command: docker build -t my-pricing-app . With this command, Docker automatically assigns the latest tag to the image, since no specific tag was provided. However, building an updated image using the same command assigns the latest tag to the new image. This can cause ambiguity and potential deployment of outdated or unverified versions. To avoid this, consider tagging the image descriptively as below: docker build -t my-pricing-app:<git-commit-hash>-1.0-prod . This command builds the my-pricing-app image tagged as <git-commit-hash>-1.0-prod. This allows specific version of the image to be deployed unlike latest tag. Additionally, you can also use the tag command to tag existing images. It is important to note that Docker tags are mutable by design, which allows you to build a new image with an existing tag. If you need to have immutable tags, then you can consider using Amazon Elastic Container Registry (Amazon ECR) to store your images. Amazon ECR supports immutable tags, which is a capability that prevents image tags from being overwritten. This enables users to rely on the descriptive tags of an image as a reliable mechanism to track and uniquely identify images and also trust that they have not been tampered with. More details can be found at Image tag mutability – Amazon ECR. Conclusion In this post, we showed you how to build better container images using best practices. As adoption of containers increase, it becomes important to ensure the container images are secure, lightweight, and are built from trusted sources. The options described in this post can act as a starting point to building such images. Additionally, this post also shows how users can use a secure and fully managed Amazon ECR to manage container images. View the full article

re:Post has launched a new functionality for community members to add a profile picture or avatar to their account. re:Post members will now be able to better personalize their accounts by uploading a photo or image of their choice. The ability to add a profile image creates a visual identifier for the account and helps members form connections, build relationships, and foster learning in the community. View the full article

Homepage Downloads GitHub About HashiCorp Packer is easy to use and automates the creation of any type of machine image. It embraces modern configuration management by encouraging you to use automated scripts to install and configure the software within your Packer-made images. Packer brings machine images into the modern age, unlocking untapped potential and opening new opportunities. Works Out of The Box Out of the box Packer comes with support to build images for Amazon EC2, CloudStack, DigitalOcean, Docker, Google Compute Engine, Microsoft Azure, QEMU, VirtualBox, VMware, and more. Support for more platforms is on the way, and anyone can add new platforms via plugins.

Docker images are the basic deployable artifact for various container orchestrators. It is extremely important to understand what is needed by the application to run, and to only include those things in the Docker image. View the full article

Amazon Elastic Container Registry Public (Amazon ECR Public) is a fully managed registry that makes it easy for a developer to publicly share container software worldwide for anyone to download. Anyone (with or without an AWS account) can use Amazon ECR Public to pull container software for use. Amazon ECR Public Gallery is a website that allows anyone to browse and search for public container images, view developer-provided details, and see pull commands. Developers no longer need to use different private and public registries when building and sharing their public container artifacts. And everyone, including anonymous users and logged-in AWS customers, can now pull artifacts from a highly available public registry using a single, global registry URL when building and running their applications. View the full article