Search the Community

You can now leverage tag-based subnet discovery capability of Amazon VPC CNI to scale Amazon Elastic Kubernetes Service (EKS) clusters in IPv4 address space without adding operational complexity. In this new default mode, Kubernetes Pod IP addresses are allocated from all tagged and available subnets in your Amazon Virtual Private Cloud(VPC). View the full article

Users modernizing their applications using Amazon Elastic Kubernetes Service (Amazon EKS) on AWS often run into critical IPv4 address space exhaustion driven by scale. They want to maximize usage of the VPC CIDRs and subnets provisioned for the EKS pods without introducing additional operational complexity. We believe that use of IPv6 address space is the long-term solution for users to build scalable networking solutions. However, we also understand that Amazon EKS users may be constrained to IPv4 environments owing to dependencies on other networking components and applications’ support for IPv6. Therefore, Amazon EKS is introducing support for Enhanced Subnet Discovery for helping users streamline network configuration and scale IPv4 based clusters without adding operational complexity. How it works Amazon VPC Container Network Interface (CNI) plugin is deployed on each Amazon Elastic Compute Cloud (Amazon EC2) worker node in your EKS cluster. It creates and attaches Elastic Network Interfaces (ENIs) to your worker nodes, as well as assigns a private IPv4, IPv6 address from your VPC CIDR to each pod in the EKS cluster. By default, VPC CNI assigns IP addresses to pods from the same subnet as the worker node’s primary network interface, which is sometimes referred to as “usable subnet”. Without any additional configuration, a node can only attach ENIs from this usable subnet in which an EC2 instance was launched. With the new feature of VPC CNI, we are now expanding the scope of “usable subnet(s)”. When enhanced subnet discovery is enabled, pod IPs are automatically allocated from all available subnets/CIDRs in the VPC that are tagged for use. New subnets can be created and tagged using the specific tag “kubernetes.io/role/cni”, and they are integrated seamlessly into the existing network configuration. This enables you to scale your applications effectively with minimal disruption to ongoing operations. Prerequisites The following prerequisites are necessary to continue with this post: An AWS Account An EKS cluster with version 1.25 or higher – we use v1.29 in the walkthrough Amazon VPC CNI version 1.18.0 or later The latest version of AWS Command Line Interface (AWS CLI) configured on your device, or AWS CloudShell eksctl – a simple CLI tool for creating and managing EKS clusters (v0.165.0 or higher) Setup export AWS_REGION=<YOUR_AWS_REGION> #Replace with your AWS Region export AWS_ACCOUNT=<YOUR_ACCOUNT> #Replace with your AWS Account number export CLUSTER_NAME=eks-enhsubsel-demo #Replace with your EKS cluster name In this walkthrough we simulate an IP exhaustion scenario by creating an Amazon VPC with /24 CIDR block, which yields 256 IP addresses. It is divided into three public and three private subnets, and each is assigned with /27 CIDR block (28 IP addresses) as shown in the following figure. Once the VPC CIDR range is exhausted, we associate a secondary CIDR to the VPC, and then we create the VPC subnets with the “kubernetes.io/role/cni” tag so that VPC CNI can automatically discover and use the new subnets to allocate the Pod IP addresses. Figure 1: Amazon VPC setup Let’s start with creating an EKS cluster with VPC CNI version 1.18.0. cat << EOF > cluster.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: ${CLUSTER_NAME} region: ${AWS_REGION} version: "1.29" vpc: cidr: 10.0.0.0/24 addons: - name: vpc-cni version: 1.18.0 - name: coredns - name: kube-proxy managedNodeGroups: - name: ${CLUSTER_NAME}-mng instanceType: m6a.large privateNetworking: true minSize: 2 desiredCapacity: 2 maxSize: 5 EOF eksctl create cluster -f cluster.yaml Wait for the cluster creation to be complete and make sure that vpc-cni addon is up and running in the cluster. aws eks describe-addon --addon-name vpc-cni --cluster-name $CLUSTER_NAME --region $AWS_REGION { "addon": { "addonName": "vpc-cni", "clusterName": "eks-enhsubsel-demo", "status": "ACTIVE", "addonVersion": "v1.18.0-eksbuild.1", .... } } As the subnets are assigned with /27 CIDR, note that private subnets only have few or no available IP addresses: aws ec2 describe-subnets --region $AWS_REGION \ --filters Name=tag:Name,Values="eksctl-eks-enhsubsel-demo-cluster/SubnetPrivate*" \ --query "Subnets[].{VPC:VpcId,SubnetId:SubnetId,AvailableIPs:AvailableIpAddressCount}" \ --output table ----------------------------------------------------------------------- | DescribeSubnets | +--------------+----------------------------+-------------------------+ | AvailableIPs | SubnetId | VPC | +--------------+----------------------------+-------------------------+ | 16 | subnet-08411e385d62f29da | vpc-07f75e9b1d954689a | | 7 | subnet-0755097835150b642 | vpc-07f75e9b1d954689a | | 0 | subnet-0975a78066e7e76d6 | vpc-07f75e9b1d954689a | +--------------+----------------------------+-------------------------+ Deploy a sample application and simulate the IP exhaustion scenario in the EKS cluster. cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: inflate spec: replicas: 50 selector: matchLabels: app: inflate template: metadata: labels: app: inflate spec: terminationGracePeriodSeconds: 0 containers: - name: inflate image: public.ecr.aws/eks-distro/kubernetes/pause:3.7 resources: requests: cpu: 50m EOF Due to the insufficient IPs, note many pods are in the “ContainerCreating” state, as the Amazon VPC CNI is unable to allocate the IP addresses. Now, let’s explore how we can use the Enhanced Subnet discovery feature of VPC CNI to automatically discover the new VPC Subnets with the available IP space, and use it to allocate IP addresses for the k8s pods. Amazon VPC supports up to five secondary IP CIDR blocks to extend the VPC IP space. Start by adding a secondary CIDR block “10.1.0.0/16” to the Amazon EKS VPC. export EKS_VPC_ID=$(aws eks describe-cluster --name $CLUSTER_NAME \ --region $AWS_REGION --query "cluster.resourcesVpcConfig.vpcId" --output text) aws ec2 associate-vpc-cidr-block --vpc-id $EKS_VPC_ID \ --cidr-block "10.1.0.0/16" --region $AWS_REGION { "CidrBlockAssociation": { "AssociationId": "vpc-cidr-assoc-06515a22930a5d6e9", "CidrBlock": "10.1.0.0/16", "CidrBlockState": { "State": "associating" } }, "VpcId": "vpc-07f75e9b1d954689a" } Wait for the association to complete, and start creating new VPC subnets from the secondary CIDR block. We are also tagging the subnets with “kubernetes.io/role/cni=1” so that VPC CNI can auto-discover them. aws ec2 create-subnet --vpc-id $EKS_VPC_ID --region $AWS_REGION \ --availability-zone "$AWS_REGION"a --cidr-block 10.1.0.0/19 \ --tag-specifications "ResourceType=subnet,Tags=[{Key=kubernetes.io/role/cni,Value=1}]" aws ec2 create-subnet --vpc-id $EKS_VPC_ID --region $AWS_REGION \ --availability-zone "$AWS_REGION"b --cidr-block 10.1.32.0/19 \ --tag-specifications "ResourceType=subnet,Tags=[{Key=kubernetes.io/role/cni,Value=1}]" aws ec2 create-subnet --vpc-id $EKS_VPC_ID --region $AWS_REGION \ --availability-zone "$AWS_REGION"c --cidr-block 10.1.64.0/19 \ --tag-specifications "ResourceType=subnet,Tags=[{Key=kubernetes.io/role/cni,Value=1}]" In the default setup, VPC CNI assigns both the primary and secondary IP addresses of an ENI from the VPC subnet associated with the Amazon EKS worker node’s primary network interface. aws ec2 describe-network-interfaces --region $AWS_REGION \ --query "NetworkInterfaces[*].{ID:NetworkInterfaceId,DNSName:PrivateDnsName,PrimaryIP:PrivateIpAddress,SecondaryIPs:PrivateIpAddresses[].PrivateIpAddress}" \ --filters Name=tag:cluster.k8s.amazonaws.com/name,Values=$CLUSTER_NAME \ --output table -------------------------------------------------------------------------------------- | DescribeNetworkInterfaces | +-------------------------------------------+-------------------------+--------------+ | DNSName | ID | PrimaryIP | +-------------------------------------------+-------------------------+--------------+ | ip-10-0-0-155.us-west-2.compute.internal | eni-07f66d0e6b2408fc7 | 10.0.0.155 | +--------------------------------------------+------------------------+--------------+ || SecondaryIPs || |+-----------------------------------------------------------------------------------+| || 10.0.0.155 || || 10.0.0.136 || || 10.0.0.140 || || 10.0.0.141 || || 10.0.0.145 || || 10.0.0.150 || || 10.0.0.135 || || 10.0.0.151 || || 10.0.0.148 || || 10.0.0.149 || |+-----------------------------------------------------------------------------------+| ......... |+----------------------------------------------------------------------------------+| | DescribeNetworkInterfaces | +--------------------------------------------+-------------------------+-------------+ | DNSName | ID | PrimaryIP | +--------------------------------------------+-------------------------+--------------+ | ip-10-0-0-102.us-west-2.compute.internal | eni-087692b80786865e0 | 10.0.0.102 | +--------------------------------------------+-------------------------+--------------+ || SecondaryIPs || |+-----------------------------------------------------------------------------------+| || 10.0.0.102 || || 10.0.0.105 || || 10.0.0.110 || || 10.0.0.126 || || 10.0.0.124 || || 10.0.0.125 || || 10.0.0.118 || || 10.0.0.100 || || 10.0.0.116 || || 10.0.0.117 || |+-----------------------------------------------------------------------------------+| Now, verify the new Enhanced Subnet Discovery feature is enabled by checking the “ENABLE_SUBNET_DISCOVERY” environment variable in the Amazon VPC CNI Addon. You can use kubectl to verify this kubectl describe ds aws-node -n kube-system | grep ENABLE_SUBNET_DISCOVERY ENABLE_SUBNET_DISCOVERY: true As of this writing, the Enhanced Subnet Discovery feature is enabled by default in Amazon VPC CNI 1.18.0 and above. If the environment variable was not set to true, then you can use kubectl or AWS CLI to set this configuration: kubectl set env daemonset aws-node -n kube-system ENABLE_SUBNET_DISCOVERY=true \ -c aws-node or aws eks update-addon --cluster-name $CLUSTER_NAME --region $AWS_REGION \ --addon-name vpc-cni \ --configuration-values '{"env":{"ENABLE_SUBNET_DISCOVERY":"true"}}' When this feature is enabled, VPC CNI looks for the VPC Subnets tagged with “kubernetes.io/role/cni” and with the available IP Space. It attaches additional ENIs from these subnets to the Amazon EKS worker nodes so that it can assign the IP addresses to the k8s pods. In our walkthrough, as many pods are in the “ContainerCreating” state, VPC CNI automatically discovered the new subnets with /19 CIDR and attached them to the existing worker nodes. We can verify this using the following commands: kubectl get pods -o wide | grep ContainerCreating <<EMPTY OUTPUT>> Now, when we look at the ENIs attached to the worker nodes, note that an additional ENI from the secondary CIDR subnet is attached and pods are assigned from the 10.1.x.x IP range. aws ec2 describe-network-interfaces --region $AWS_REGION \ --query "NetworkInterfaces[*].{ID:NetworkInterfaceId,DNSName:PrivateDnsName,PrimaryIP:PrivateIpAddress,SecondaryIPs:PrivateIpAddresses[].PrivateIpAddress}" \ --filters Name=tag:cluster.k8s.amazonaws.com/name,Values=$CLUSTER_NAME \ --output table -------------------------------------------------------------------------------------- | DescribeNetworkInterfaces | +-------------------------------------------+-------------------------+--------------+ | DNSName | ID | PrimaryIP | +-------------------------------------------+-------------------------+--------------+ | ip-10-0-0-152.us-west-2.compute.internal | eni-0525ae09d044a6688 | 10.0.0.152 | +--------------------------------------------+-------------------------+--------------+ || SecondaryIPs || |+-----------------------------------------------------------------------------------+| || 10.0.0.152 || || 10.0.0.144 || || 10.0.0.154 || || 10.0.0.147 || || 10.0.0.133 || || 10.0.0.157 || || 10.0.0.153 || || 10.0.0.158 || || 10.0.0.146 || || 10.0.0.132 || |+-----------------------------------------------------------------------------------+| | DescribeNetworkInterfaces | +--------------------------------------------+-------------------------+--------------+ | DNSName | ID | PrimaryIP | +--------------------------------------------+-------------------------+--------------+ | ip-10-1-79-53.us-west-2.compute.internal | eni-0b2632fa77e9fbf68 | 10.1.79.53 | +--------------------------------------------+-------------------------+--------------+ || SecondaryIPs || |+-----------------------------------------------------------------------------------+| || 10.1.79.53 || || 10.1.78.231 || || 10.1.75.23 || || 10.1.81.171 || || 10.1.95.26 || || 10.1.86.60 || || 10.1.76.92 || || 10.1.65.140 || || 10.1.75.174 || || 10.1.94.30 || |+-----------------------------------------------------------------------------------+| Cleaning up To avoid ongoing charges, make sure to delete EKS cluster resources created in your AWS account. # Delete EKS cluster resources eksctl delete cluster -f cluster.yaml Key considerations Shared subnets When using this feature in the cross account scenario, where VPC and Subnets are created in the central AWS Account and shared with the participant AWS Account to deploy the EKS cluster, you should tag the subnets in the participant account where the cluster is launched. Refer to Use shared VPC Subnets in Amazon EKS for a detailed walkthrough. Custom networking Custom networking, a feature of Amazon VPC CNI, provides optionality to the IP exhaustion issue by assigning the Pod IPs from secondary VPC IP address spaces. When custom networking is enabled in VPC CNI, it creates secondary ENIs in the subnet defined under a custom resource named ENIConfig that includes an alternate subnet CIDR range created from a secondary VPC CIDR. The VPC CNI assigns Pods IP addresses from the CIDR range defined in the ENIConfig custom resource. Furthermore, the pods can use different security groups than that of the node’s primary network interface. Therefore, you might consider custom networking if you have a security requirement to run Pods on a different network with different security groups. Note that, unlike custom networking, enhanced subnet discovery does not need the creation of the ENIConfig custom resource, and thus reduces the configuration overhead. Custom networking takes precedence when both features are enabled on the VPC CNI. Pod networking use cases You can use this feature along with other VPC CNI use cases, such as “SNAT for Pods”, “Security groups for Pods”, “Kubernetes network policies”, and “Increase available IP addresses on a worker node”. Refer to Choose Pod networking use cases for a detailed comparison. Conclusion In this post we showed you how Amazon VPC CNI based subnet discovery can provide scale and flexibility to adjust your IPv4 address allocations to accommodate the growth of your EKS clusters with low operational overhead. We demonstrated how the feature enables adaptability to changes in size, and simplifies IP address management while supporting the dynamic needs of modern IT environments. Visit the Amazon EKS best practices guide for recommendations and additional considerations for securely scaling EKS clusters. For installation instructions for Amazon VPC CNI, refer to the Amazon EKS user guide. You may provide feedback on the Amazon VPC CNI plugin by leaving a comment or opening an issue on the AWS Containers Roadmap that is hosted on GitHub. View the full article