Document toolboxDocument toolbox

(v2.2.0.0) Checking EKS Environments

Owned by Yajing Wang
Last updated: May 18, 2024Version comment
3 min read

To ensure your setup is optimal for Infrastructure Optimizer, please ensure your Amazon Elastic Kubernetes Service (Amazon EKS) Cluster and Karpenter adhere to the following specifications.

EKS Prerequisites Overview

Component

Section Link

Component

Section Link

EKS Cluster

Cluster Details

Add-ons

Add-ons

Cluster Management

Toolkit

Cluster Details

  • Kubernetes Version: Version 1.29 or later

Component

Requirements

Cluster IAM

Type: AWS Managed

  • AmazonEKSClusterPolicy -> This is to allow the Kubernetes control plane to manage AWS resources on your behalf

Cluster Permission

  • Allow EKS API and ConfigMap to authorize cluster administrator

Cluster Endpoint Access

  • Allow public and private endpoint access

  • Node Group: Has at least two regular EKS nodes to run Karpenter on (Name cannot contain “-“)

Component

Requirements

Node Group IAM

Type: AWS Managed

  • AmazonEC2ContainerRegistryReadOnly → This allows read-only access to Amazon EC2 Container Registry repositories

  • AmazonEKS_CNI_Policy → This provides the Amazon VPC CNI Add-on permissions it requires to modify the IP address configuration on your EKS worker nodes

  • AmazonEKSWorkerNodePolicy → This allows Amazon EKS worker nodes to connect to Amazon EKS Clusters

  • AmazonSSMManagedInstanceCore → This is to enable AWS Systems Manager service core functionality

  • Daemonset exo-aws-node: Has the following changes

  1. Edit the Daemonset Configuration: Begin by editing the aws-node daemonset in the kube-system namespace:

    kubectl edit daemonset -n kube-system aws-node

    Within the editor, navigate to the nodeSelectorTerms section and add the following:

    - key: eks.amazonaws.com/nodegroup operator: NotIn values: - x-compute

  2. Export Current Daemonset Configuration: Save the current configuration of the aws-node daemonset to a YAML file:

    kubectl get daemonset -n kube-system aws-node -o yaml > exo-aws-node-daemon-set.yaml

  3. Modify the Daemonset File:
    Open the exo-aws-node-daemon-set.yaml file and make the following changes:

    1. Change the nodeSelectorTerms from NotIn > In for x-compute nodegroup

    2. Change only name (2 places) and app.kubernetes.io/name (2 places) of daemonset to exo-aws-node instead of aws-node

  4. Apply the Updated Configuration: Apply the updated configuration to the cluster:

    kubectl apply -f exo-aws-node-daemon-set.yaml

  5. Tune the following variables:

    kubectl set env daemonset -n kube-system exo-aws-node \ AWS_VPC_K8S_CNI_EXTERNALSNAT=true kubectl set env daemonset -n kube-system exo-aws-node \ ENABLE_PREFIX_DELEGATION=true MAX_ENI=1 \ WARM_PREFIX_TARGET=13 WARM_IP_TARGET=0 MINIMUM_IP_TARGET=0 kubectl set env daemonset -n kube-system exo-aws-node \ DISABLE_NETWORK_RESOURCE_PROVISIONING=true kubectl set env daemonset -n kube-system exo-aws-node \ POD_SECURITY_GROUP_ENFORCING_MODE=standard

  • ConfigMap: Has the following changes

  1. Edit the ConfigMap Configuration: Begin by editing the aws-auth config in the kube-system namespace:

    kubectl edit configmap aws-auth -n kube-system

  2. Insert the following to the mapRoles section:

    - groups: - system:masters rolearn: <Insert the Role ARN of your Worker IAM Role> username: admin - groups: - system:masters rolearn: <Insert the Role ARN of your Controller IAM Role> username: admin

Add-ons

  • Kube-proxy: Version 1.29.1-eksbuild.2 or later

  • Amazon VPC CNI: Version 1.16.4-eksbuild.2 or later

Amazon VPC CNI IAM Role: cni-addon-iamserviceaccount

Type: AWS Managed

Policy

Explanation

Policy

Explanation

1

AmazonEKS_CNI_Policy

This provides the Amazon VPC CNI Add-on permissions it requires to modify the IP address configuration on your EKS worker nodes

Type: Customer Inline

Policy

Explanation

Policy

Explanation

1

UnassignPrivateIpAddresses

This denies unassigning one or more secondary private IP addresses, or IPv4 Prefix Delegation prefixes from a network interface

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Deny", "Action": "ec2:UnassignPrivateIpAddresses", "Resource": "*" } ] }

  • Amazon EBS CSI Driver: Version v1.29.1-eksbuild.1 or later

Toolkit

  • Kubectl: within one minor version difference of your cluster

  • AWS CLI: version 2

  • Helm: 3.13.0 or later