/
v4.0.1 Installing with Terraform Modules (Kubernetes-native)

v4.0.1 Installing with Terraform Modules (Kubernetes-native)

The Exostellar Terraform modules - Kubernetes-native are one of the newest Exostellar installation approaches designed to simplify the setup process.

Current Terraform Modules Version: Unreleased. Use latest commit on main branch.

Version Matrix

Component

Version

Component

Version

Exostellar Terraform Modules

Latest commit on main branch

Kubernetes

1.32 - 1.33

Exo Helm Chart

4.0.1

Worker AMI ID

https://exostellar.atlassian.net/wiki/spaces/ENG/pages/314638358/EAR+Non-Marketplace+Release+Information#4.0.0-AMIs

Exostellar's CNI (Exo-CNI)

v1.20.0

Exostellar's CSI (Exo-CSI)

v1.46.0

Exostellar's Karpenter (xkarpenter)

v2.0.7

Guest Kernel Version (for Xspot Nodes)

5.15.185

@danh @Sagar Jadhav @Bhargav Ravuri Need to cut a release for Terraform modules k8s-base.

Supported Kubernetes Versions
Exostellar currently only supports Kubernetes version 1.32 - 1.33.

  • Starting from Kubernetes 1.32, Exostellar has migrated to Amazon Linux 2023 (AL2023)–based images, following AWS recommendations.

 

Prerequisites

Before running the Terraform commands on Exostellar Terraform modules - Kubernetes-native, ensure that your environment meets the following requirements:

1. Local tools

  1. Terraform: Version 1.8+

  2. AWS CLI: Version 2.0+

  3. Helm: Version 3.14.2+

  4. kubectl: Version 1.29+

 

2. IAM Permissions

The following block contains the IAM (Identity and Access Management) permissions to securely allow only the required access for Terraform modules and other operations (like Helm login to public ECR) in this document.

1. Terraform Modules Policy

Set the following IAM policy on the IAM user or role for deploying the Terraform modules:

Any modifications to this policy could cause internal application failures. Changes are not recommended, but if necessary, proceed with caution.

Ensure the local user has the following IAM permissions:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "iam:PassRole", "Resource": [ "arn:aws:iam::${aws_account_id}:role/${aws_resource_prefix}${eks_cluster_name}-xspot-worker" ] }, { "Effect": "Allow", "Action": [ "ec2:DescribeRouteTables", "ec2:DescribeSubnets", "ec2:DescribeVpcAttribute", "ec2:DescribeVpcs" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "eks:AssociateAccessPolicy", "eks:CreateAccessEntry", "eks:DeleteAccessEntry", "eks:DescribeAccessEntry", "eks:DescribeCluster", "eks:DisassociateAccessPolicy", "eks:ListAssociatedAccessPolicies", "eks:TagResource" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "iam:AddRoleToInstanceProfile", "iam:AttachRolePolicy", "iam:CreateInstanceProfile", "iam:CreateOpenIDConnectProvider", "iam:CreatePolicy", "iam:CreateRole", "iam:DeleteInstanceProfile", "iam:DeleteOpenIDConnectProvider", "iam:DeletePolicy", "iam:DeleteRole", "iam:DeleteRolePolicy", "iam:DetachRolePolicy", "iam:GetInstanceProfile", "iam:GetOpenIDConnectProvider", "iam:GetPolicy", "iam:GetPolicyVersion", "iam:GetRole", "iam:GetRolePolicy", "iam:ListAttachedRolePolicies", "iam:ListInstanceProfilesForRole", "iam:ListPolicyVersions", "iam:ListRolePolicies", "iam:PutRolePolicy", "iam:RemoveRoleFromInstanceProfile", "iam:TagInstanceProfile", "iam:TagPolicy", "iam:TagRole" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "route53:ChangeTagsForResource", "route53:CreateHostedZone", "route53:DeleteHostedZone", "route53:GetChange", "route53:GetHostedZone", "route53:ListResourceRecordSets", "route53:ListTagsForResource" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:DeleteBucket", "s3:DeleteObject", "s3:DeleteObjectVersion", "s3:GetAccelerateConfiguration", "s3:GetBucketAcl", "s3:GetBucketCORS", "s3:GetBucketLocation", "s3:GetBucketLogging", "s3:GetBucketObjectLockConfiguration", "s3:GetBucketPolicy", "s3:GetBucketPublicAccessBlock", "s3:GetBucketRequestPayment", "s3:GetBucketTagging", "s3:GetBucketVersioning", "s3:GetBucketWebsite", "s3:GetEncryptionConfiguration", "s3:GetLifecycleConfiguration", "s3:GetObject", "s3:GetObjectVersion", "s3:GetReplicationConfiguration", "s3:ListAccessPoints", "s3:ListBucket", "s3:ListBucketVersions", "s3:PutBucketAcl", "s3:PutBucketPolicy", "s3:PutBucketPublicAccessBlock", "s3:PutBucketTagging", "s3:PutBucketVersioning" ], "Resource": "*" } ] }

Replace the placeholders ${aws_account_id}, ${aws_resource_prefix}, and ${eks_cluster_name} in the policy.

 

2. Helm Auth Policy

Set the following IAM policy on the user or role for AWS CLI to fetch the auth token for authorizing Helm to access the public ECR to fetch the Exostellar Helm charts:

Ensure the local user has the following IAM permissions for fetching the auth token:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ecr-public:GetAuthorizationToken" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "sts:GetServiceBearerToken" ], "Resource": "*" } ] }

 

3. Private Zone Record Deletion Policy

Set the following IAM policy on the user or role used by the AWS CLI to allow deletion of Exostellar-related records from the Route53 private hosted zone. This permission is required to complete the manual cleanup step before running terraform destroy.

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "route53:ChangeResourceRecordSets", "Resource": "arn:aws:route53:::hostedzone/*" } ] }

 

3. AWS Authentication

If the AWS CLI is already logged in to your EKS cluster, you may “skip this step”.

Log in to AWS using the CLI in any of the methods recommended by AWS here.

It is recommended to set the AWS region to match your cluster’s region.

 

Introduction

There are 2 flows in Exostellar Terraform modules:

Deploy a new EKS cluster and set up Exostellar — the Kubernetes-native (k8s-base) architecture — on it using Terraform.

 

Standalone flow is not yet supported in Exostellar Terraform modules - Kubernetes-native.

 

Set up Exostellar — the Kubernetes-native (k8s-base) architecture — on your existing EKS cluster using Terraform.

 

The Kubernetes-native (k8s-base) architecture is a newer edition from Exostellar. It deploys all Exostellar components inside the EKS cluster (except the worker node, which must run as an EC2 instance for migration and scaling). This design improves performance and optimizes cost efficiency.

For more details on this, refer to this doc from the latest release version: README.md

 

Deploy

Standalone flow is not yet supported in Exostellar Terraform modules - Kubernetes-native.

Existing cluster flow is the recommended way to install the Exostellar setup. You are the owner of your EKS cluster, and Exostellar Terraform modules manages only the Exostellar-related resources on AWS and Kubernetes.

Prerequisites

1. GitHub PAT

Skip this step if you already have a GitHub PAT token with read:packages permission.

Since some of the Docker images are hosted in ghcr.io (GitHub Container Registry), you need to create a PAT token with read:packages permission.

  1. Go to https://github.com/settings/tokens. Log in.

  2. Go to Personal access tokens → Tokens (classic) → Generate new token → Generate new token (classic).

  3. Set Note (name), expiration, and select read:packages. Submit.

  4. Copy the PAT token. This will be used in inputs later in this page.

 

2. Create EKS Cluster

A functional EKS cluster, with control plane and managed nodes, should be up and running, and configured in ~/.kube/config file.

Exostellar Terraform Modules (Kubernetes-native) support the following Amazon EKS versions: 1.32 – 1.33

We recommend using EKS 1.33 to take advantage of the latest features and reduce the risk of issues related to end-of-life (EOL) versions.

Create the EKS cluster using eksctl:

eksctl create cluster \ --name <cluster-name> \ --region <aws-region> \ --version 1.33 \ --timeout 1h \ --nodegroup-name <nodegroup-name> \ --node-ami-family AmazonLinux2023 \ --node-type m5.large \ --nodes 3 \ --nodes-min 2 \ --nodes-max 4 \ --ssh-access \ --ssh-public-key <your-ssh-key-pair-name>

It’s recommended to use managed node groups based on AmazonLinux2023 to stay aligned with the x-compute nodes that will be created by Exostellar.

 

3. Install AWS CNI

AWS CNI (amazon-vpc-cni-k8s) is generally installed on the EKS cluster by default as an add-on. This is mandatory for Exostellar to function.

Check if AWS CNI is installed on the cluster:

kubectl get daemonset/aws-node -n kube-system

Output should be similar to:

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE aws-node 3 3 3 3 3 <none> 6h

If missing, use the following steps:

helm repo add eks https://aws.github.io/eks-charts helm repo update helm install aws-vpc-cni --namespace kube-system eks/aws-vpc-cni

Note: Exostellar Terraform modules (Kubernetes-native) are only tested against the official AWS-maintained CNI (amazon-vpc-cni-k8s), hence the same is recommended to users.

If you need Exostellar Terraform modules (Kubernetes-native) to support any other CNI out of the box, please contact Exostellar Customer Support (contact details are provided at the end of this page).

 

4. Install AWS CSI

AWS CSI (aws-ebs-csi-driver), if missing, can either be installed as add-ons (managed by AWS) or Helm charts (managed by the user) in the cluster. This is mandatory for Exostellar to function.

Check if AWS CNI is installed on the cluster:

kubectl get daemonset/ebs-csi-node -n kube-system

Output should be similar to:

NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE ebs-csi-node 3 3 3 3 3 kubernetes.io/os=linux 5h52m

If missing, use the following steps:

helm repo add aws-ebs-csi-driver https://kubernetes-sigs.github.io/aws-ebs-csi-driver helm repo update helm install aws-ebs-csi-driver \ --namespace kube-system \ aws-ebs-csi-driver/aws-ebs-csi-driver \ --version 2.48.0

Exostellar currently supports the AWS CSI (aws-ebs-csi-driver)'s Helm chart only up to version v2.48.0. Later version of it will be enabled in future releases of Exostellar.

Note: Exostellar Terraform modules (Kubernetes-native) are only tested against the official AWS-maintained CSI (aws-ebs-csi-driver), hence the same is recommended to users.

If you need Exostellar Terraform modules (Kubernetes-native) to support any other CSI out of the box, please contact Exostellar Customer Support (contact details are provided at the end of this page).

4.1. (Alternative) IRSA for AWS CSI (EBS CSI driver)

The Exostellar Terraform modules will create the IAM role and IAM policy required for setting up IRSA for the AWS EBS CSI driver.

 

Alternatively, you may set up IRSA for AWS CSI (EBS CSI driver) manually. If so, ensure the following for compatibility with Exostellar:

  1. The IAM role name must be [<aws_resource_prefix>]<eks_cluster_name>-ebs-csi (prefix is optional).

  2. The IAM policy name must also be [<aws_resource_prefix>]<eks_cluster_name>-ebs-csi (prefix is optional), using the policy from [here][EBS CSI driver IAM policy].

  3. The ServiceAccount ebs-csi-controller-sa must be annotated with the IAM role ARN. This is covered in the next step (“Configure AWS CNI and AWS CSI for Exostellar”) since the configuration is the same regardless of how IRSA is created.

  4. Disable the flag create_irsa_for_ebs_csi_driver in the module so the Terraform modules skip creating the IAM role and policy.

 

5. Configure AWS CNI and CSI

Run the script to configure AWS CNI and AWS CSI:

./configure-aws-cni-and-aws-csi.sh -n "<eks_cluster_name>"

Refer to the script’s help output for more information about its usage and input parameters:
./configure-aws-cni-and-aws-csi.sh -h

The same script is found at scripts/common/configure-aws-cni-and-aws-csi.sh in github.com/exostellar/terraform-exostellar-modules or in the release archive.

1. Prevent AWS CNI from running on x-compute nodes by adding the following condition to the DaemonSet.

  • Check the AWS CNI daemonSet's node affinity:

    kubectl get daemonset aws-node -n kube-system \ -o jsonpath='{.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[*]}' \ | jq

  • Verify the output contains the following condition:

    { "key": "eks.amazonaws.com/nodegroup", "operator": "NotIn", "values": ["x-compute"] }

  • If missing, patch the daemonSet:

    kubectl get daemonset aws-node -n kube-system -o json \ | jq '.spec.template.spec.affinity.nodeAffinity .requiredDuringSchedulingIgnoredDuringExecution .nodeSelectorTerms[0].matchExpressions += [ { "key": "eks.amazonaws.com/nodegroup", "operator": "NotIn", "values": ["x-compute"] } ]' \ | kubectl apply -f -

 

2. Disable AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG on container aws-node in AWS CNI DaemonSet.

  • Check the current value:

    kubectl get daemonset aws-node -n kube-system -o json \ | jq -r '.spec.template.spec.containers[] | select(.name=="aws-node") | .env[] | select(.name=="AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG") | "\(.name)=\(.value)"'

  • Expected output:

    AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=false

  • If missing or set to true, update it:

    kubectl set env daemonset/aws-node -n kube-system -c aws-node AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=false

 

3. Prevent AWS CSI from running on x-compute nodes by adding the following condition to the DaemonSet.

  • Check the AWS CSI daemonSet's node affinity:

    kubectl get daemonset ebs-csi-node -n kube-system \ -o jsonpath='{.spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[*]}' \ | jq

  • Verify the output contains the following condition:

    { "key": "eks.amazonaws.com/nodegroup", "operator": "NotIn", "values": ["x-compute"] }

  • If missing, patch the daemonSet:

    kubectl get daemonset ebs-csi-node -n kube-system -o json \ | jq '.spec.template.spec.affinity.nodeAffinity .requiredDuringSchedulingIgnoredDuringExecution .nodeSelectorTerms[0].matchExpressions += [ { "key": "eks.amazonaws.com/nodegroup", "operator": "NotIn", "values": ["x-compute"] } ]' \ | kubectl apply -f -

 

4. Annotate the EBS CSI driver's service account with the IAM role ARN for IRSA. And restart the EBS CSI driver add-on or pods (if installed using Helm chart).

The EBS CSI driver’s IAM role will be created by the Terraform modules after this manual step.

If you want to manually handle the IRSA for AWS EBS CSI driver, follow the steps in v4.0.1 Installing with Terraform Modules (Kubernetes-native)

  • Check if the annotation is already present:

    kubectl get sa ebs-csi-controller-sa \ -n kube-system \ -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'

  • Annotate the service account:

    kubectl annotate serviceaccount/ebs-csi-controller-sa \ -n kube-system \ "eks.amazonaws.com/role-arn=arn:aws:iam::<aws-account-id>:role/<cluster-name>-ebs-csi-driver-role"

    • Note: Replace <aws-account-id> and <cluster-name> in the above command with your AWS account ID and cluster name, respectively.

  • Restart driver pods:

    kubectl delete pod -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver kubectl wait pods -n kube-system -l app.kubernetes.io/name=aws-ebs-csi-driver --for=condition=ready --timeout=300s

When the EBS CSI driver (AWS CSI) IAM role or policy is removed (through terraform destroy, the force-deletion script, or manual deletion), the EBS CSI driver pods reconcile and revert to a state without IRSA, even though the ServiceAccount annotation persists.

To avoid this, restart the EBS CSI pods before running terraform apply, even if you are deploying again on the same EKS cluster.

 

@Bhargav Ravuri TODO: Check if this can be prevented or failed fast.

 

 

6. Clean-up Old Exostellar CRDs

If Exostellar modules were previously deployed on the cluster and the CRDs were not properly removed, version mismatches may occur. These mismatches can lead to internal failures that are not detectable at the Terraform level. To avoid this, it is recommended to run the following script to clean up the CRDs and their associated CRs before (re)deploying the Exostellar Terraform modules.

Quick check:

kubectl get crd \ exoautoscalers.autoscale.k8s.exo \ exonodeclaims.exokarpenter.sh \ exonodeclasses.karpenter.k8s.exo \ exonodepools.exokarpenter.sh \ xnodes.core.exostellar.io

If Exostellar CRDs found, run the script to delete them:

./force-delete-exostellar-crs-and-crds.sh

Refer to the script’s help output for more information about its usage and input parameters:
./force-delete-exostellar-crs-and-crds.sh -h

The same script is found at scripts/common/force-delete-exostellar-crs-and-crds.sh in github.com/exostellar/terraform-exostellar-modules or in the release archive.

 

7. Helm Login to Container Registries

Login to both:

Exostellar’s Helm charts are hosted on Amazon ECR Public.

Run the following AWS CLI command to retrieve the authentication token required for logging Helm into ECR Public.

aws ecr-public get-login-password --region "us-east-1" \ | helm registry login --username "AWS" --password-stdin "public.ecr.aws"

ECR Public API operations are supported only in us-east-1. Regardless of your deployment region, you must use us-east-1 when running the Helm login command for ECR Public.

This operation requires the IAM permissions: v4.0.1 Installing with Terraform Modules (Kubernetes-native)

Some additional Exostellar Helm charts and Docker images are hosted on a private GHCR (GitHub Container Registry).

Create a PAT token in the GitHub as explained above (v4.0.1 Installing with Terraform Modules (Kubernetes-native)), and use it as the login password in the following command:

echo "<github-pat>" | helm registry login ghcr.io --username "<github-username>" --password-stdin

Please contact Exostellar Customer Support (contact details are provided at the end of this page) for getting the access to Exostellar’s private GHCR.

Steps to deploy Existing Cluster Flow (Kubernetes-native):

1. Clone the repo: https://github.com/exostellar/terraform-exostellar-modules/. Switch to latest main, if not already.

Cloning the repository is a temporary workaround until a released version of the Terraform modules (Kubernetes-native) is available. Once released, the module can be referenced directly via the release source path.

@danh @Bhargav Ravuri ^^

2. Update the inputs as per your requirement.

  • Pick either example:

  1. Create a main.tf file with the following content anywhere in your local.

  2. Set the source path to the cloned repository’s existing-cluster-full module.

module "existing_cluster_flow" { source = "<path/to/root/of/cloned/repo>/modules/k8s-base/existing-cluster-full" eks_cluster = "my-exostellar-cluster" aws_region = "us-east-1" license_filepath = "/path/to/exo-license-file" worker_ami_id = "ami-XXXXXXXXXXXXXXXXX" } output "exo_console_url" { description = "Exostellar's console URL." value = module.existing_cluster_flow.exo_console_url } output "private_zone_id" { description = "Route 53 private hosted DNS zone's ID." value = module.existing_cluster_flow.private_zone_id }

Temporary: Use a “relative” path (not absolute) in source.

There is a known Terraform limitation when using local paths as module sources. If a module contains submodules that use local relative paths (as in our case), Terraform expects the parent module's source path to be relative, not absolute. Therefore, the source path in the module below has been changed to a relative path.

Most inputs are optional, and the Terraform modules use default values. Unless you need to customize something specific, using the minimal example (instead of this) should suffice.

Create a main.tf file using the content below, and set the source path to the cloned repository’s existing-cluster-full module, similar to the minimal example.
Alternatively, you can directly use the examples/existing-cluster-flow/main.tf file from the cloned repository.

module "existing_cluster_flow" { # Use the latest existing-cluster-full module release as source. This is the recommended approach to import the # existing-cluster-full module from anywhere, for production. # # In case of running in this repository, you may use the relative path to the existing-cluster-full module's # directory. # Note: This is not recommended for production as the local existing-cluster-full module may contain untested changes. # source = "../../../modules/k8s-base/existing-cluster-full" # # TODO :: Bhargav :: Fix the path in the released version URL with the next release. source = "../../../modules/k8s-base/existing-cluster-full" #################### # Mandatory Inputs # #################### # EKS Cluster Inputs: # # EKS cluster's name. # # Note: It must be between 1 and 42 characters in length. eks_cluster = "my-exostellar-cluster" # AWS region for deploying resources. Eg. 'us-east-1'. aws_region = "us-east-1" # Xmanager Inputs: # # The Exostellar's license file name/path. Default is "". # # Note: license_filepath is not technically mandatory, but it is required for the xcomputed to function. If you leave # it empty (""), the Terraform modules will skip adding the license. You can still add it later by logging in to the # Exostellar console and uploading the license from the **Settings** page. license_filepath = "/path/to/exo-license-file" # Xspot Inputs: # # Xspot worker AMI ID. worker_ami_id = "ami-XXXXXXXXXXXXXXXXX" ################### # Optional Inputs # ################### # EKS Cluster Inputs: # # Create an IAM OIDC provider for the EKS cluster. Default is true. # # An OIDC provider is required to grant the necessary IAM roles and permissions for Exostellar components to function # properly. Each EKS cluster can have only one OIDC provider associated with it. The OIDC provider URL is # automatically available from the EKS cluster, but the corresponding IAM OIDC provider resource must be created, # either manually or by enabling this flag to let Exostellar create it for you. # # If your EKS cluster already has an existing OIDC provider, set this to false so Exostellar uses the existing one # instead of creating a new one. create_oidc_provider_for_cluster = true # Xmanager Inputs: # # Xmanager images tag. Default is "". # # Note: If left empty (""), the latest compatible xmanager images will be used by the Helm chart. xmanager_images_tag = "" # Xmanager Helm chart's repository. Default is "oci://ghcr.io/exostellar/". xmanager_helm_chart_repo = "oci://ghcr.io/exostellar/" # Xmanager Helm chart's name. Default is "exo". xmanager_helm_chart_name = "exo" # Enable Exo CRDs like exoautoscaler (i.e., manage through Terraform)? Default is true. # # If Exo CRDs are already present in the cluster, run `force-delete-exostellar-crs-and-crds.sh` before # `terraform apply` to remove existing CRs and CRDs. # # Note: When this flag is disabled, Helm will still install the CRDs if they are missing from the cluster. However, # these CRDs are not managed by Terraform and will not be removed during `terraform destroy`, which can lead to CRD # version mismatches in subsequent runs. Therefore, it is recommended to enable this flag. exo_crd_enabled = true # Xmanager CRDs Helm chart's name. Default is "exo-crd". xmanager_crds_helm_chart_name = "exo-crd" # Xmanager Helm chart's version. Default is "4.0.1". xmanager_helm_chart_version = "4.0.1" # Install cert-manager in the cluster? Note: cert-manager is a prerequisite for exo Helm chart. Default is true. install_cert_manager = true # Image registry inputs: # # Exostellar image registry URL. Default is "". # # Note: If left empty (""), the Helm charts will attempt to pull images from public registries without authentication. images_registry = "" # Exostellar image registry username. Default is "". # # Note: If all 3 inputs (image_registry_username, image_registry_email, image_registry_password) are provided, # login to the image registry will be performed to pull images from private registries. image_registry_username = "" # Exostellar image registry email. Default is "". # # Note: If all 3 inputs (image_registry_username, image_registry_email, image_registry_password) are provided, # login to the image registry will be performed to pull images from private registries. image_registry_email = "" # Exostellar image registry password. Default is "". # # Note: If all 3 inputs (image_registry_username, image_registry_email, image_registry_password) are provided, # login to the image registry will be performed to pull images from private registries. image_registry_password = "" # Exostellar image registry URL. Default is "ghcr.io". image_registry_url = "ghcr.io" # Exostellar image pull secret name. Default is "ghcr-secret". image_pull_secret_name = "ghcr-secret" # Xspot Inputs: # # Xspot images tag. Default is "". # # Note: If left empty (""), the latest compatible xspot images will be used by the Helm chart. xspot_images_tag = "" # Enable hyperthreading in Xspot? Default is true. xspot_enable_hyperthreading = true # Enable ballooning in Xspot? Default is true. xspot_enable_balloon = true # Enable Infrastructure Optimizer (IO) for xspot? Default is true. enable_infrastructure_optimizer = true # Enable Workload Optimizer (WO) for xspot? Default is true. enable_workload_optimizer = true # Exostellar Karpenter (xkarpenter) Inputs: # # Enable xkarpenter? Default is true. # # Note: If this is disabled, all the following are disabled automatically: # 1. xkarpenter namespace creation. # 2. IAM role and policy for xkarpenter. # 3. IAM role and policy for xnode-controller. # 4. xkarpenter CRDs management through Terraform. # 5. xkarpenter resources (default ExoNodeClass and ExoNodePool). xkarpenter_enabled = true # Enable xkarpenter CRDs (i.e., manage through Terraform)? Default is true. # # If xkarpenter CRDs are already present in the cluster, run `force-delete-exostellar-crs-and-crds.sh` before # `terraform apply` to remove existing CRs and CRDs. # # Note: When this flag is disabled, Helm will still install the CRDs if they are missing from the cluster. However, # these CRDs are not managed by Terraform and will not be removed during `terraform destroy`, which can lead to CRD # version mismatches in subsequent runs. Therefore, it is recommended to enable this flag. xkarpenter_crd_enabled = true # Enable xkarpenter resources (default ExoNodeClass and ExoNodePool)? Default is true. xkarpenter_resources_enabled = true # Exostellar's Karpenter (xkarpenter)'s Helm chart's repository URL. Default is # "oci://ghcr.io/exostellar/xkarpenter/karpenter". xkarpenter_helm_chart_repository = "oci://ghcr.io/exostellar/xkarpenter/karpenter" # Exostellar's Karpenter (xkarpenter)'s resources (ExoNodeClass and ExoNodePool) Helm chart's repository URL. # Default is "oci://ghcr.io/exostellar/xkarpenter/karpenter-resources". xkarpenter_resources_helm_chart_repository = "oci://ghcr.io/exostellar/xkarpenter/karpenter-resources" # Exostellar's Karpenter (xkarpenter) CRDs Helm chart's repository URL. Default is # "oci://ghcr.io/exostellar/xkarpenter/karpenter-crd". xkarpenter_crds_helm_chart_repository = "oci://ghcr.io/exostellar/xkarpenter/karpenter-crd" # Exostellar's Karpenter (xkarpenter)'s Helm chart's version. Default is "v2.0.7". # Note: To keep xkarpenter and xkarpenter Resources compatible, same version is used with both charts. xkarpenter_version = "v2.0.7" # Exostellar's Karpenter (xkarpenter)'s namespace. Default is "exokarpenter". xkarpenter_namespace = "exokarpenter" # Guest kernel version for xspot nodes. Default is "5.15.185". guest_kernel_version = "5.15.185" # Pod resource requests and limits. Default is # {requests: { cpu: 1, memory: "1Gi" }, limits: { cpu: 1, memory: "1Gi" }}. pod_resource_limits = { requests = { cpu = 1 memory = "1Gi" } limits = { cpu = 1 memory = "1Gi" } } # Xspot controller's configurations. # # Xspot controller's namespace. Default is "exo-system". xspot_control_plane_namespace = "exo-system" # CNI and CSI Inputs: # # Exostellar's CNI chart's repository URL. Default is "oci://public.ecr.aws/u8h5n6o4/exostellar-cni/exo-aws-vpc-cni". exo_cni_chart_repository = "oci://public.ecr.aws/u8h5n6o4/exostellar-cni/exo-aws-vpc-cni" # Exostellar's CNI chart's version. Default is "1.20.0". exo_cni_chart_version = "1.20.0" # Exostellar's CNI chart's namespace. Default is "kube-system". exo_cni_chart_namespace = "kube-system" # Exostellar's CSI chart's repository URL. Default is # "oci://public.ecr.aws/u8h5n6o4/exostellar-csi/exo-aws-ebs-csi-driver". exo_csi_chart_repository = "oci://public.ecr.aws/u8h5n6o4/exostellar-csi/exo-aws-ebs-csi-driver" # Exostellar's CSI chart's version. Default is "1.46.0". exo_csi_chart_version = "1.46.0" # Exostellar's CSI chart's namespace. Default is "kube-system". exo_csi_chart_namespace = "kube-system" # Create an IRSA setup for the EBS CSI Driver (AWS CSI)? Default: true. # # IRSA (IAM Roles for Service Accounts) enables EKS pods to securely assume IAM roles via OIDC, providing keyless and # scoped access to AWS services. # Reference: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html # # The EBS CSI driver (AWS CSI) requires IRSA so its pods can assume an IAM role with the necessary EC2 permissions to # manage EBS volumes on behalf of Kubernetes, without exposing or sharing AWS credentials. # Reference: https://github.com/kubernetes-sigs/aws-ebs-csi-driver # # Although Exostellar’s CSI (Exo-CSI) handles CSI operations within Exostellar, it still depends on the AWS EBS CSI # driver for actual volume provisioning in AWS. # # If your AWS EBS CSI driver already has an existing IAM role with IRSA setup, set this to false so that Exostellar # uses the existing configuration instead of creating a new one. create_irsa_for_ebs_csi_driver = true # Storage Inputs: # # Postgres database size. Default is "10Gi". postgres_database_size = "10Gi" # Postgres database storage class name. Default is "gp2-expandable". postgres_database_storage_class_name = "gp2-expandable" # Cost Calculation database size. Default is "10Gi". cost_calculation_database_size = "10Gi" # Cost Calculation database storage class name. Default is "gp2-expandable". cost_calculation_database_storage_class_name = "gp2-expandable" # Grafana database size. Default is "2Gi". grafana_database_size = "2Gi" # Grafana database storage class name. Default is "gp2-expandable". grafana_database_storage_class_name = "gp2-expandable" # Keycloak database size. Default is "2Gi". keycloak_database_size = "2Gi" # Keycloak database storage class name. Default is "gp2-expandable". keycloak_database_storage_class_name = "gp2-expandable" # Loki database size. Default is "10Gi". loki_database_size = "10Gi" # Loki database storage class name. Default is "gp2-expandable". loki_database_storage_class_name = "gp2-expandable" # Prometheus database size. Default is "2Gi". prometheus_database_size = "2Gi" # Prometheus database storage class name. Default is "gp2-expandable". prometheus_database_storage_class_name = "gp2-expandable" # Redis database size. Default is "2Gi". redis_database_size = "2Gi" # Redis database storage class name. Default is "gp2-expandable". redis_database_storage_class_name = "gp2-expandable" # Miscellaneous Inputs: # # Prefix to use for all AWS resources. Combined length of (aws_resource_prefix + eks_cluster) must be between # 1 and 42. Default is "". # # Note: This will not add any trailing extra characters like '-'. aws_resource_prefix = "" # Optional ARN of an IAM policy to use as a permissions boundary for the X-Spot controller IAM role created by this module. # permissions_boundary_arn_xspot_controller = "arn:aws:iam::XXXXXXXXXXX:policy/MyPermissionsBoundary" # Optional ARN of an IAM policy to use as a permissions boundary for the X-Spot worker IAM role created by this module. # permissions_boundary_arn_xspot_worker = "arn:aws:iam::XXXXXXXXXXX:policy/MyPermissionsBoundary" # Optional ARN of an IAM policy to use as a permissions boundary for the Karpenter IAM role created by this module. # permissions_boundary_arn_karpenter = "arn:aws:iam::XXXXXXXXXXX:policy/MyPermissionsBoundary" # Optional ARN of an IAM policy to use as a permissions boundary for the X-Node Controller IAM role created by this module. # permissions_boundary_arn_xnode_controller = "arn:aws:iam::XXXXXXXXXXX:policy/MyPermissionsBoundary" # Optional ARN of an IAM policy to use as a permissions boundary for the Loki S3 IAM role created by this module. # permissions_boundary_arn_loki_s3 = "arn:aws:iam::XXXXXXXXXXX:policy/MyPermissionsBoundary" # Optional ARN of an IAM policy to use as a permissions boundary for the X-Computed IAM role created by this module. # permissions_boundary_arn_xcomputed = "arn:aws:iam::XXXXXXXXXXX:policy/MyPermissionsBoundary" # Disable OpenAPI validation for this release. # disable_openapi_validation = true }

 

3. Terraform Deploy

Run the following commands from your main.tf’s directory:

terraform init terraform plan -input=false terraform apply -auto-approve

On a successful deploy, the last few lines of the output should look similar this:

Apply complete! Resources: 49 added, 0 changed, 0 destroyed. Outputs: exo_console_url = "a762939e3a2ea47a095d5d2dca8bcf94-224035494e211b86.elb.us-east-2.amazonaws.com" private_zone_id = "Z08539222RAUEF20TX9PW"

This operation requires the IAM permissions: v4.0.1 Installing with Terraform Modules (Kubernetes-native)

5. Add License

Skip this step if license_filepath is already provided in the module.

An Exostellar license is required for the xcompute to work.

Please contact Exostellar Customer Support (contact details are provided at the end of this page) for getting an Exostellar license file.

  1. Open the Exostellar Console URL from the terraform apply output in your browser.

  2. Log in using the temporary credentials:
    Username: admin@exostellar.io
    Password: password

  3. You will be prompted to change the password. Update it and log in again.

  4. Navigate to Settings → Licenses → Add Licenses, and upload the license file.

 

Post-deployment Validation (Optional)

Run the following checks to validate your deployment is working as expected.

Steps:

  1. Deploy pod .

    kubectl apply -f csi-validation.yaml

  2. Monitor Node Creation.
    Run the following command:

    watch kubectl get pod/csi-validation-pod -owide

    Verify the following:

    1. The pod status transitions from Pending to Running.

    2. The pod is scheduled on a node whose name ends with x-compute.internal.

After deploying the pod, monitor for the controller (Kubernetes) Deployment’s creation:

kubectl get deploy -n exo-system -l app.kubernetes.io/name=xspot-controller

After approximately 30–60 seconds, the output should look similar to the following:

NAME READY UP-TO-DATE AVAILABLE AGE controller-exo62d27ae59d854668809505ee45698529-b60cf41e 1/1 1 1 2m

 

Then watch all the related resources:

watch kubectl get po,no,exonodeclass,exonodepool,exonodeclaim,pvc,sc

After approximately 5–10 minutes, the output should look similar to the following:

NAME READY STATUS RESTARTS AGE pod/csi-validation-pod 1/1 Running 0 4m25s NAME STATUS ROLES AGE VERSION node/ip-192-168-112-161.us-east-2.x-compute.internal Ready <none> 98s v1.33.5-eks-ecaa3a6 node/ip-192-168-14-195.us-east-2.compute.internal Ready <none> 173m v1.33.7-eks-70ce843 node/ip-192-168-37-2.us-east-2.compute.internal Ready <none> 173m v1.33.7-eks-70ce843 node/ip-192-168-68-163.us-east-2.compute.internal Ready <none> 173m v1.33.7-eks-70ce843 NAME AGE exonodeclass.karpenter.k8s.exo/default-class 5m3s NAME NODECLASS exonodepool.exokarpenter.sh/default-pool default-class NAME TYPE ZONE NODE READY AGE exonodeclaim.exokarpenter.sh/default-pool-85jrb m5.xlarge us-east-2c ip-192-168-112-161.us-east-2.x-compute.internal True 4m26s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/csi-validation-pod-ebs-volume Bound pvc-d12add60-f07d-4118-befd-906428e6d050 1Gi RWO gp2 <unset> 4m31s NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE storageclass.storage.k8s.io/gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 179m storageclass.storage.k8s.io/gp2-expandable kubernetes.io/aws-ebs Delete WaitForFirstConsumer true 14m

At a high level, you should observe the following sequence:

  1. A new node joins the cluster.

    • The node name ends with x-compute.internal.

    • The node transitions to the Ready state.

  2. ExoNodeClaim association.

    • A single ExoNodeClaim is created for the workload.

    • The newly created node is associated with that ExoNodeClaim.

  3. Volume provisioning.

    • An EBS volume is dynamically provisioned.

    • The corresponding PVC transitions to Bound.

  4. Pod readiness.

    • The pod status changes from Pending to Running.

  1. Delete the pod after confirming the above.

    kubectl delete -f csi-validation.yaml

Steps:

  1. Deploy pod (You can run this alongside csi-validation.yaml to save time).

    kubectl apply -f cni-validation-1.yaml

  2. Monitor node creation.
    Run the following command:

    watch kubectl get pod/cni-validation-pod-1 -owide

    Verify the following:

    1. The pod status transitions from Pending to Running.

    2. The pod is scheduled on a node whose name ends with x-compute.internal.

  3. Exec into the Controller Pod.

    kubectl exec -it `kubectl get deploy -n exo-system -l "app.kubernetes.io/name=xspot-controller" -oname` -c exo-d -n exo-system -- bash

  4. Check for xspot processes (Can be exited using CTRL+c).

    xspot ps -t

    Output will be similar to:

    ====License==== Status: active Expiration: N/A Head Node Licenses: X-WO, X-IO ====Config==== SchedulerEnabled: true OnDemandTypes: [c5.9xlarge c5.18xlarge c5.2xlarge r5.large m5.4xlarge m5.large m5.16xlarge r5.24xlarge c5.large c5.xlarge m5.xlarge r5.xlarge r5.8xlarge r5.2xlarge c5.12xlarge r5.12xlarge r5.4xlarge m5.2xlarge c5.4xlarge m5.24xlarge m5.12xlarge m5.8xlarge c5.24xlarge r5.16xlarge] SpotTypes: [c5d.xlarge c5.9xlarge c5.18xlarge c5.2xlarge r5d.8xlarge r5.large m5d.12xlarge m5.4xlarge m5.large m5.16xlarge c5d.2xlarge r5d.2xlarge r5.24xlarge c5.large c5.xlarge m5d.8xlarge m5d.4xlarge m5.xlarge r5.xlarge r5.8xlarge m5d.2xlarge c5d.large r5d.16xlarge r5d.24xlarge r5.2xlarge c5d.4xlarge c5.12xlarge r5d.large r5.12xlarge r5.4xlarge c5d.24xlarge m5d.xlarge m5d.16xlarge m5.2xlarge c5d.18xlarge c5.4xlarge r5d.4xlarge r5d.xlarge m5d.24xlarge m5d.large m5.24xlarge m5.12xlarge m5.8xlarge c5.24xlarge r5d.12xlarge r5.16xlarge c5d.9xlarge c5d.12xlarge] BootingTimeout: 5 DefaultSandboxCpu : 4 DefaultSandboxMem : 4096 ====Sandboxes==== SANDBOX ID WORKER ID TARGET WORKER ID STATUS TARGET CPU TARGET MEMORY PROV CPU PROV MEMORY IP ADDRESS CREATED TAGS vm-38345fbd- 3ceece3b Running 2 2928 2/4 2928/8939 192.168.110.163 6 minutes ago ====Workers==== WORKER ID STATUS INSTANCE ID INSTANCE TYPE REMAINING MEM REMAINING CPU IP ADDRESS UP TIME DEADLINE SPOT PRICE 17b95057 Booting i-0b25d3e782cbb85bc m5.large 4096 2 8s 3h9m18s true 0.0283 3ceece3b Ready i-05d1c220f1a636347 r5.large 7769 0 192.168.100.80 3m12s 3h33m32s true 0.0412 6459e0b9 Ready i-06399097f8e6ed4b2 m5.xlarge 10697 4 192.168.101.185 6m0s 1h59m34s true 0.0546

  5. When the Sandbox goes to Running status, delete the associated worker:

    xspot rm -f <worker_id>

  6. Monitor for sandboxes and workers. The sandbox should migrate from delete worker to a new one, and start running again.

    xspot ps -t

    Output will be similar to:

    ====Sandboxes==== SANDBOX ID WORKER ID TARGET WORKER ID STATUS TARGET CPU TARGET MEMORY PROV CPU PROV MEMORY IP ADDRESS CREATED TAGS vm-38345fbd- c91ad3b5 Running 1 2928 1/4 2928/8939 192.168.110.163 14 minutes ago ====Workers==== WORKER ID STATUS INSTANCE ID INSTANCE TYPE REMAINING MEM REMAINING CPU IP ADDRESS UP TIME DEADLINE SPOT PRICE 9d31555e Ready i-07e31b76a7e17ec32 r5.large 10697 2 192.168.99.110 4m36s --- false 0.126 c91ad3b5 Ready i-0ba6ec79db0f79df5 r5.large 7769 1 192.168.102.101 3m36s 3h5m52s true 0.0412

  7. Deploy pod .

    kubectl apply -f cni-validation-2.yaml

  8. Monitor both pods until they reach the Running state, then continue observing them for an additional 2 minutes to ensure stability.

    kubectl get pod/cni-validation-pod-1 pod/cni-validation-pod-2 -owide

    Output will be similar to:

    NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES cni-validation-pod-1 1/1 Running 0 21m 192.168.103.16 ip-192-168-110-163.us-east-2.x-compute.internal <none> <none> cni-validation-pod-2 1/1 Running 0 2m38s 192.168.102.240 ip-192-168-110-163.us-east-2.x-compute.internal <none> <none>

  9. Delete the pods after confirming the above.

    kubectl delete -f cni-validation-1.yaml -f cni-validation-2.yaml

Destroy

Steps to delete the terraform-exostellar-modules standalone flow:

Standalone flow is not yet supported in Exostellar Terraform modules - Kubernetes-native.

1. Delete Controllers

The xspot controllers (Kubernetes Deployments) will automatically terminate after 10 minutes of inactivity (i.e., no workloads running). However, if you plan to tear down the environment before that, ensure you delete the controllers manually before running terraform destroy.

  1. Check if there are xspot controller Deployments:

    kubectl get deploy -n exo-system -l "app.kubernetes.io/name=xspot-controller"

  2. If they exist, delete them:

    kubectl delete deploy -n exo-system -l "app.kubernetes.io/name=xspot-controller"

 

2. Remove Xspot Controller’s Records from Route53 Private Hosted Zone

The xspot controllers DNS records will automatically be removed after 10 minutes of inactivity (i.e., no workloads and xspot controllers running). However, if you plan to tear down the environment before that, ensure you remove the records manually before running terraform destroy.

Run the following commands in the directory where your main.tf and Terraform state files exist:

# Retrieve the private hosted zone ID from the Terraform outputs: # Note: Run this command in the directory containing your main.tf and Terraform state files. ZONE_ID=$(terraform output -raw private_zone_id) # Delete all DNS records except for the SOA and NS types (i.e., the xspot controller records). aws route53 change-resource-record-sets \ --hosted-zone-id "$ZONE_ID" \ --change-batch "$(aws route53 list-resource-record-sets --hosted-zone-id "$ZONE_ID" | jq -c ' { Changes: [ .ResourceRecordSets[] | select(.Type != "SOA" and .Type != "NS") | {Action: "DELETE", ResourceRecordSet: .} ] }')" | cat

This operation requires the IAM permissions: v4.0.1 Installing with Terraform Modules (Kubernetes-native)

 

3. Terraform Destroy

Run the following commands from your main.tf’s directory:

terraform destroy -auto-approve -refresh=true

On a successful deploy, the last few lines of the output should look similar this:

Destroy complete! Resources: 49 destroyed.

This will delete the Exostellar components (like xcomputed, xKarpenter, etc.), while preserving your original EKS cluster.