• What is kro (Kube Resource Orchestrator)

• What is ACK (AWS Controllers for Kubernetes)

• EKS capabilities advantages in building a platform for developers

• How the workload for platform engineers can be reduced (operational efficiency)

The primary focus of this chapter is to cover kro and ACK capabilities and how to deploy an application using ACK and Kubernetes APIs abstracted with kro. In the demo, I'll use a single account setup, but don't worry, later in this series, I will cover multi-account setups and more.

Okay, let me try to simplify kro and ACK. Imagine your Kubernetes cluster is a high-end restaurant kitchen.

The Developers are the Hungry Customers They don't want to know how the stove works or where you bought the carrots. They just want a meal (an app with a database) delivered to their table, fast.

ACK is the Fully Stocked Pantry ACK connects your kitchen to the massive warehouse of ingredients (AWS Resources).

kro is the Head Chef’s Recipe Card (The Menu) kro is the tool the Head Chef (Platform Engineer) uses to combine those raw ingredients into a finished dish. Abstracting API’s in the Kubernetes cluster and creates single unit, reusable API’s for the developers.

What are kro and ACK?

ACK (AWS Controllers for Kubernetes)

ACK allows you to define AWS resources directly as Kubernetes objects. Instead of logging into the AWS Console or running Terraform, you apply a YAML file for an S3Bucket or Table (DynamoDB), and the ACK controller provisions it for you.

• User creates the Kubernetes manifest files for Kubernetes objects with default api objects, also AWS Resources using ACK.

• Not using any IaC tools like Cloudformation or terraform to create AWS resources. Using a Kuberntes manifest file we can create the AWS resources.

• In order to do that, we need to give the right IAM permission for the ACK. Don’t worry, we will be revisiting this part with more details later.

Take a look in the below manifest which creates a lambda execution role using ACK. When we deploy the manifest, IAM role will be created in the AWS Account. So simply put, using a Kubernetes manifest we can create AWS Resources without any IaC tools. While this is a simplified example, I will expand into the advanced details later in this post.

apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
  name: lambda-execution-role
  namespace: default
spec:
  name: lambda-execution-role
  assumeRolePolicyDocument: |
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "lambda.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
  policies:
    - arn:aws:iam::<accountif>:policy/lambda-execution-policy
  tags:
    - key: App
      value: hello-world-lambda
    - key: Environment
      value: test

Think back to the high-end restaurant kitchen analogy I mentioned earlier, here is how it applies,

• Need a DynamoDB Table? ACK brings you a crate of raw potatoes.

• Need an S3 Bucket? ACK brings you a bag of flour.

The Problem: You can't serve a customer a raw potato and a bag of flour. It’s too messy, complicated, and they don't know how to cook it. That is where kro comes into the picture.

kro (Kube Resource Orchestrator)

If ACK provides the ingredients, kro provides the recipe. kro is an open-source project that lets you create custom, single unit, reusable API’s. It acts as the abstracter, bundling multiple resources (like a Deployment, Service, and DynamoDB Table [via ACK API’s]) into a high-level API. kro uses CEL (Common Expression Language), the same language used by Kubernetes webhooks, for logical operations.

• kro is an abstracter, it manages groups of resources as a single, reusable custom API.

• Building on that same high-end restaurant kitchen analogy,

  • You define a recipe called "The Microservice Special".

  • The recipe says: "Take 1 bag of flour (S3 via ACK), 2 potatoes (DynamoDB via ACK), and cook them with a side of Compute (Deployment)."

  • The Magic: Now, you provide the customer (Developer) a Menu. They don't order "flour and potatoes"; they just point to "The Microservice Special."

Resource Graph Definition (RGD) Cluster-scoped :
A Resource Graph Definition is a reusable template that defines: A developer-facing schema (API) A resource graph with dependencies. Here's a simplified example showing the core structure:

apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: webappstack.kro.run
spec:
  schema:
    apiVersion: v1alpha1
    kind: WebAppStack
    spec:
      name: string
      team: string 
      image: string | default="nginx"
      replicas: integer | default=2
      bucket:
        enabled: boolean | default=false
        name: string | default=""
        region: string | default="us-west-2"
    status:
      deploymentStatus: ${deployment.status.conditions}
      bucketStatus: ${s3-bucket.status.ackResourceMetadata.arn}

  resources:
  - id: deployment
    template:
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: ${schema.spec.name}
      spec:
        replicas: ${schema.spec.replicas}
        selector:
          matchLabels:
            app: ${schema.spec.name}
        template:
          spec:
            containers:
              - name: ${schema.spec.name}
                image: ${schema.spec.image}              
# ACK S3 Bucket Resource
  - id: s3-bucket
    includeWhen:
      - ${schema.spec.bucket.enabled}
    template:
      apiVersion: s3.services.k8s.aws/v1alpha1
      kind: Bucket
      metadata:
        name: ${schema.spec.bucket.name}
        namespace: ${schema.metadata.namespace}
        labels:
          team: ${schema.spec.team}
      spec:
        name: ${schema.spec.bucket.name}
        createBucketConfiguration:
          locationConstraint: ${schema.spec.bucket.region}

Highlighted Components:

• Abstraction Layer: The schema is the developer interface; they don't see Kubernetes complexity

• Dependency Resolution: The platform resolves references

• Conditional Resources: The includeWhen directive shows conditional resource creation based on developer input. As per to the above example, if developer needs an s3bucket simply can set this value to true.

• Status Aggregation: The status section exposes runtime information back to developers

Resource Group Instance (Namespace-scoped):

Developers create instances using the RGD schema. They focus on the desired state, leaving the implementation to the controllers. Developers simply need to apply the Resource Group Instance YAML.

apiVersion: kro.run/v1alpha1
kind: WebAppStack
metadata:
  name: blog-api
  namespace: production
spec:
  name: blog-api
  team: backend-team
  image: my-registry.io/blog-api:v1.2.3
  replicas: 3
  containerPort: 8080
  ingress:
    enabled: true
  # Request the ACK Bucket
  bucket:
    enabled: true
    name: blog-api-assets-prod-001

Platform Engineering Benefits:

• Self-Service: Developers provision infrastructure without platform team tickets

• Consistency: All web apps follow the same patterns and best practices

• Governance: Platform team controls what can be created via the RGD

• Abstraction: Developers think in application terms (replicas, image), not Kubernetes resources

If the RGD is the recipe; the instance is the customer order.

If you came this far you might be thinking “where is EKS Capabilities, all I read is kro and ACK”. Don’t worry, we are getting in to that soon. However, before we jump into the EKS Capabilities, let’s ensure we’re all on the same page regarding the basics of how these two tools function independently.

The Old Way: Self-Managing ACK and kro

Before EKS Capabilities, platform engineers installed and managed ACK and the kro themselves.

Additional operational burden:

• Running ACK, kro pods

• Managing ACK, kro upgrades and compatibility

The Reality:

Platform engineers spent significant time on:

• Controller installation and configuration

• Security patches and vulnerability management

• Capacity planning for controller workloads

• Debugging controller reconciliation loops

This operational overhead diverts resources such as time, from developing new tools and enhancing the platform

You can watch this episode from The Zacs’ Show, where we share how to install ACK and kro controllers to an EKS cluster: https://www.youtube.com/watch?v=Fex-xxKTMC4

The New Way: EKS Capabilities

EKS Capabilities provides fully managed ACK and KRO, running in AWS-owned infrastructure outside your cluster. It also includes Argo CD for GitOps (to be covered in upcoming posts).

What Changed:

1. No In-Cluster Controllers: ACK and kro run in AWS-managed infrastructure

2. GitOps Ready: ArgoCD is included as a managed capability (keep an eye out in future chapters for a detailed explanation)

3. Automatic Lifecycle Management: AWS handles scaling, patching, and upgrades

But now:

• No controller pods in your cluster

• No manual upgrades or patches

• Focus on building platform abstractions

Platform Engineering Benefits:

• No controller management

• More time on developer experience

• AWS managed security patches

• AWS handles availability and scaling

This shift allows platform engineers to concentrate on creating abstractions that empower developers to help themselves, instead of managing the underlying infrastructure tools.

Let's get hands-on.

Demo Time: Building a Full-Stack Application with EKS Capabilities

We'll deploy a full-stack application that demonstrates (kro and ACK) EKS Capabilities in action. The platform team defines a Resource Graph Definition (RGD) that abstracts infrastructure complexity, and the development team provisions their application with a single Kubernetes manifest.

The Scenario:

• Platform team: Creates a reusable Resource Graph Definition (RGD) that sets up a complete application stack.

• Development team: Uses the RGD to deploy their application.

• Result: A functioning application with AWS resources, Kubernetes workloads, and networking all from one simple manifest.

When you apply the Resource Group Instance, you'll see:

1. Automatic Resource Creation: KRO manages the creation of multiple resources in the correct order.

2. Dependency Resolution: Resources that rely on others (like an IAM Role referencing an IAM Policy) are created in sequence.

3. Status Propagation: The RGD status fields are automatically filled as resources become available.

4. Conditional Resources: The Ingress resource is created only when ingress.enabled: true.

5. End-to-End Integration: Your application pods automatically receive AWS credentials through Pod Identity and can access DynamoDB.

6. Readiness: Using readyWhen: ensures a resource is fully ready before deployment begins.

Resources created in this Demo

PS: You might notice Feijoa mentioned in this demo. Feijoa is a fruit available in New Zealand 🇳🇿 and it's my favorite fruit too 😛

Here's the complete inventory of resources that will be provisioned:

KRO Resources

• ResourceGraphDefinition feijoaappstack.kro.run - The platform abstraction template

  • We are going to create a single API called feijoaappstack.

ACK Resources

• IAM Policy iam.services.k8s.aws/v1alpha1- Grants DynamoDB permissions

• IAM Role iam.services.k8s.aws/v1alpha1 - Assumable by EKS pods

• PodIdentityAssociation eks.services.k8s.aws/v1alpha1- Links IAM role to Kubernetes ServiceAccount

• DynamoDB Table dynamodb.services.k8s.aws/v1alpha1 - Application data store

Kubernetes Native Resources

• ServiceAccount - Used by pods for Pod Identity

• Deployment - Application workload with configurable replicas

• Service - ClusterIP service exposing the deployment

• PodDisruptionBudget - Ensures availability during disruptions

• IngressClass - Defines ALB ingress controller (created once)

• Ingress - Application Load Balancer (conditional, only if ingress.enabled: true)

Demo Steps:

Step 1: EKS Cluster

• I used an EKS cluster with Auto Mode enabled, therefore, I don’t have to setup the core add-ons and other dependencies. If you are new to EKS Auto Mode check this : https://blog.awsfanboy.com/lets-explore-amazon-eks-auto-mode

Step 2: Enable EKS Capabilities

• There are several ways to do this. In this demo, I used the AWS Console to enable it.

• Go to the EKS Console and click on capabilities, and you will see this page.

• Select the kro and ACK capabilities, then create a Capability role here. For ACK, this role will create the AWS resources for you.

• Follow the "Create admin role" instructions to create the capabilities for both ACK and kro.

  

In this demo, we will deploy everything in the default namespace. However, in this series, I will also share how to deploy with namespace-specific permissions using IAMRoleSelector. This means we can assign specific IAM roles for each namespace, allowing developers with access to that namespace to deploy resources using a designated IAM role.

• Having completed these steps, you will see the capabilities are active now.

  

Now, if you check the available APIs in the EKS cluster, you will see over 200 available APIs.

You can see the resourcegraphdefinitions API from kro.

Also, you can see APIs available for AWS resources from ACK.

There are more ;) ,

Step 4: Configure RBAC for KRO

Important note: The kro capability can manage ResourceGraphDefinitions and their instances by default using the AmazonEKSKROPolicy, but it requires additional permissions to handle Kubernetes resources like ACK resources. These permissions need to be granted through access entry policies or Kubernetes RBAC.

For this demo, I am creating a ClusterRoleBinding with the cluster-admin ClusterRole. Ensure that /KRO is in uppercase, as the name is case-sensitive. Reference: Kubernetes RoleBinding Example. However, in a production environment, you can set more granular access. I will cover a real production setup in future posts in this series, so stay tuned.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kro-cluster-admin
subjects:
- kind: User # Replace the name: with with your IAM role
  name: arn:aws:sts::<account_id>:assumed-role/AmazonEKSCapabilityKRORole/KRO
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

Step 5: Create Resource Graph Definition (Platform Team)

Platform team creates the RGD template that defines the application stack abstraction.

# Apply the Resource Graph Definition
kubectl apply -f chapter-1/platform-team/feijoa-rgd.yaml

# Verify RGD was created
kubectl get resourcegraphdefinition feijoaappstack.kro.run

# Check RGD details
kubectl describe resourcegraphdefinition feijoaappstack.kro.run

If your RGD is ready, you will see the Status as active same as the below screenshot.

Step 6: Create Resource Group Instance (Application Team)

The application team uses the RGD to deploy their application with a single manifest.

# Apply the Resource Group Instance
kubectl apply -f chapter-1/dev-team/feijoa-instance.yml

# Watch resources being created
kubectl get feijoaappstack store -w

# Check instance status
kubectl get feijoaappstack store

Our Store instance is ready

Step 7: Verify Resource Creation and Dependencies

Confirm that all resources were created correctly and that dependencies are resolved.

We can list some of the resources created using kubectl get all, serviceaccount, ingress, pdb, table, roles.iam.services.k8s.aws, policies.iam.services.k8s.aws. The output will look something like this. Now, we can see that all the resources have been successfully created.

You can use the following commands to check and play around.

# Check RGD status fields are populated
kubectl get feijoaappstack store -o jsonpath='{.status}'

# Verify IAM role ARN is in PodIdentityAssociation
kubectl get podidentityassociation store -o jsonpath='{.spec.roleARN}'

# Verify DynamoDB table ARN in deployment env vars
kubectl get deployment store -o jsonpath='{.spec.template.spec.containers[0].env}'

# Verify ACK resources
kubectl get policy,role
kubectl get table
kubectl get podidentityassociation

# Verify Kubernetes resources
kubectl get deployment,service,serviceaccount
kubectl get ingress
kubectl get poddisruptionbudget

# Check all resources eg:
kubectl get all,serviceaccount,ingress,pdb,table,roles.iam.services.k8s.aws,policies.iam.services.k8s.aws

Step 8: Test the Application

Verify that the application is accessible and can interact with DynamoDB.

# Get Ingress URL
kubectl get ingress store-ingress -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'

Now that you have the ALB URL, you can access the app. Try adding some Feijoa fruits to the bucket.

We can see that DynamoDB is updating, which means everything is working end-to-end.

Step 9: Cleanup

To clean up the demo environment, make sure to delete the cluster resources first before disabling the EKS capabilities.

# Delete the instance (cascades to all resources)
kubectl delete feijoaappstack store

# Verify resources are deleted
kubectl get all,serviceaccount,ingress,pdb -l app=store

# Delete RGD (if needed)
kubectl delete resourcegraphdefinition feijoaappstack.kro.run

Once the resources are deleted, we can follow these steps to disable the EKS Capabilities from the EKS cluster.

Summary

In this demo, we have successfully:

• Enabled EKS Capabilities (ACK, kro)

• Configured IAM roles and access entries

• Set up RBAC for kro capability role

• Created a Resource Graph Definition (platform abstraction)

• Used the API we created to deploy a complete application stack

• Verified all resources and dependencies

• Tested the application end-to-end

Key Takeaway

From a single developer-friendly manifest, we managed 10 resources across AWS and Kubernetes, showing the strength of platform engineering with EKS Capabilities. With kro, we easily can create reusable single API units to set up multiple instances in a standard way for developers. Also, using ACK we can create the AWS resources and dependencies for the Kubernetes resources.

References:

• https://docs.aws.amazon.com/eks/latest/userguide/capabilities.html

• https://kro.run/

• https://aws-controllers-k8s.github.io/docs/intro/

However, this raised security concerns. Many of us, including myself, used KIAM and kube2iam to solve this problem by allowing granular access for Kubernetes applications. But managing these proxies wasn't easy either.

Then AWS IRSA (IAM roles for service accounts) simplified this complexity. However, for larger organizations with multiple AWS accounts, this also became a challenge because managing OIDC providers isn't easy. Before we dive in to EKS Pod Identity here is a simple understanding of KIAM, kube2iam and AWS IRSA.

EKS Pod Identity Launch

In 2023, AWS announced the launch of EKS Pod Identity, which simplifies the configuration of Kubernetes applications to obtain AWS IAM permissions. Unlike AWS IRSA, with EKS Pod Identity, we don't need the OIDC Provider dependency. You just need to install the EKS Pod Identity add-on and create the Pod Identity association. The diagram below explains the simple setup of EKS Pod Identity.

However, there were still some limitations with cross-account setup. There wasn't a direct way to do it, so we had to use some workarounds. We appreciate that AWS is great at listening to their customers.

Finally, AWS announced improvements for EKS Pod Identity cross-account access.

Internals of Cross-Account EKS Pod Identity

Let's break down how this works. In this scenario, Pods running in the EKS cluster within the Developer account can easily access resources in the Infrastructure account. This is done by having the IAM role in the Developer account assume a role in the Infrastructure account. It's as simple as that!

Let's explore two scenarios in this demo:

• As a Platform Engineer, Set up the external-dns add-on to create DNS records in Route53 across accounts.

• As a developer, learn how to access parameter store values from a different account.

Prerequisites:

• EKS Auto Mode Cluster with built-in EKS Pod Identity add-on. If you want to learn how to set up a EKS Auto Mode cluster, check my previous blog here.

• Two AWS Accounts

• Domain name configured in Route53

• Optional: ACM Certificate

Scenario 1: External DNS Cross-Account Access

We have an EKS Cluster running in the Developer Account, and the External DNS pods use the EKS Pod Identity to create DNS records in the Infrastructure Account.We already have a domain set up in Route53 within the Infrastructure Account.

Now let's create the IAM role and policies.

Infrastructure Account IAM

Include the IAM role ARN that we will create later in the developer account. Also, include the sts:ExternalId for more granular access..

Format for the sts:ExternalId

{AWS_Region}/{AWS_AccountNumber}/{EKS_Cluster_Name}/{NameSpace}/{ServiceAccountName}

For demonstration purposes, we are granting AmazonRoute53FullAccess access.

Create the IAM Role

Developer Account IAM

Skip adding permissions and create the IAM Role.

Go to the created IAM role and create an inline policy. Include the IAM role ARN that we previously created in the Infrastructure Account.

Create the policy

Install and configure the External DNS Addon in the EKS Cluster within the Developer Account.

Select External DNS

Select the EKS Pod Identity and choose the IAM Role we created. The External DNS add-on will be ready in a few minutes.

We needed to make one final configuration: edit the EKS Pod Identity Association for the External DNS.

Add the IAM role from the Infrastructure Account in the Target IAM Role section.

Now it's time to deploy a test application to check the External DNS Cross Account setup.

Here is the sample Kubernetes manifest we used for the application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: doggo-app
  labels:
    app: doggo-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: doggo-app
  template:
    metadata:
      labels:
        app: doggo-app
    spec:
      containers:
      - name: doggo-app
        image: awsfanboy/doggo-app:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: doggo-app-service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  selector:
    app: doggo-app
---
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: eks-auto-alb
spec:
  controller: eks.amazonaws.com/alb
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: doggo-app-ingress
  annotations:
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-southeast-2:xxxxxxxxxx:certificate/{ACM}
spec:
  ingressClassName: eks-auto-alb
  rules:
  - host: app.infra.awsfanboy.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: doggo-app-service
            port:
              number: 80

Deploy the application

List the ingress object

Logs of the external-dns pod show that the DNS records were successfully created.

Route53 records were created successfully.

App is running successfully

Scenario 2: App deployed on EKS Cluster with SSM Parameter access from Infrastructure account

We already have a parameter in SSM within the Infrastructure Account.

Now, let's create the IAM role and policies.

Infrastructure Account IAM

We are following the same steps we used for the External DNS add-on. Below are the screenshots for the Trust Relationship policy, IAM Policy of the the IAM Role to access the SSM Parameter store.

Developer Account IAM

We are following the same steps we used for the External DNS add-on. Below are the screenshots for the Trust Relationship policy, IAM Policy of the the IAM Role to assume the IAM role in the Infrastructure Account.

Create the EKS Pod Identity Association. Later, we are deploying a sample app into the default namespace with a Service Account named awsfanboy-sa.

Sample Kubernetes Manifest we used for deploying the sample application

apiVersion: v1
kind: ServiceAccount
metadata:
  name: awsfanboy-sa
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: awsfanboy
  labels:
    app: awsfanboy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: awsfanboy
  template:
    metadata:
      labels:
        app: awsfanboy
    spec:
      serviceAccountName: awsfanboy-sa
      containers:
      - name: awscli
        image: amazon/aws-cli:latest
        command: ["sleep"]
        args: ["infinity"]

Let's deploy the sample app.

Now let's access the pod and run some commands to check the cross-account access.

• aws sts get-caller-identity: This shows that the identity is using the IAM role created in the Infrastructure Account.

• • 

• aws ssm get-parameter --name <parameter_name> The pod can access the parameter from the SSM Parameter Store in the Infrastructure Account.

  • 

Conclusion

We can clearly see that this new enhanced and streamlined way of EKS Pod Identity for cross-account access can be configured within a few seconds, reducing the workload for the EKS platform team and developers.

Reference

• https://aws.amazon.com/blogs/aws/amazon-eks-pod-identity-simplifies-iam-permissions-for-applications-on-amazon-eks-clusters/

• https://aws.amazon.com/about-aws/whats-new/2023/11/amazon-eks-pod-identity/

• https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html

• https://aws.amazon.com/blogs/containers/amazon-eks-pod-identity-streamlines-cross-account-access/

Official announcement: https://aws.amazon.com/about-aws/whats-new/2025/02/aws-codepipeline-native-amazon-eks-deployment-support/

In this article, I will cover not only the Deploy stage but also discuss an end-to-end pipeline. Feel free to skip the Build section if you are already familiar with those services.

Let's begin with setting up the EKS Cluster

We need an EKS Cluster to test this out. Also, a note on; if the API server endpoint access is private, ensure the private subnets should be able to access internet via a NAT Gateway. Therefore, make sure the Route tables are configured correctly.

In this Demo, my EKS Cluster API server endpoint access is private, I have created an inbound rule in the Cluster security group

You can follow my blog to create an EKS cluster with Auto Mode within few minutes.

Creating the ECR Repository

For this demo, we need a container registry to store the container image that we will build during the build stage. Later, we will deploy this image to the EKS cluster. You can use the following simple Terraform script to create the registry or create it directly through the console.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

resource "aws_ecr_repository" "doggo_app_ecr" {
  name                 = "doggo-app-ecr"
  image_tag_mutability = "MUTABLE"
  image_scanning_configuration {
    scan_on_push = true
  }
}

output "ecr_repository_uri" {
  value       = aws_ecr_repository.doggo_app_ecr.repository_url
  description = "The URI of the ECR repository"
}

AWS CodePipeline Creation and Configuration

Create a Pipeline from the scratch, follow the steps as per to my screenshots.

Creation option, select Build custom pipeline.

For this demo, under the Service role, create a New service role. Since we are using an EKS cluster with a private API server endpoint, we will need to update this roles policy later to allow access to read the EKS cluster's private subnets. You can proceed for now; we will handle this step later.

A quick note before we start the source stage: I have created this repository https://github.com/awsfanboy/doggo-container-app with all the resources needed for this demo. Make sure to update the values.yaml file with the image URL.

This Helm chart is ready for an EKS Auto Mode cluster. If your cluster is not enabled for EKS Auto Mode, you will need to install add-ons like the ALB Ingress Controller.

image:
  repository: { Replace with your ECR repository URL}
 # example,  repository: 11111111111.dkr.ecr.ap-southeast-2.amazonaws.com/doggo-ap-ecr
  tag: latest
  pullPolicy: Always

Set up the source stage by connecting your GitHub repository or any other repository you plan to use.

In the build stage, choose Other build providers and select AWS ECR. Then, select the ECR repository name from the dropdown menu.

Skip the Test stage for this demo and proceed to the deploy stage.

So far, what we've done isn't new, and I'm sure you're familiar with the previous steps. However, I wanted to explain each step for this demo.

Deploy stage, let’s check these fields now.

• Select the deploy provider as Amazon EKS.

• Since we have already created an EKS cluster in the same region, you can select the cluster from the dropdown.

• For Deployment configuration type, choose either helm or kubectl. In this demo, we are using a Helm chart to deploy.

• Provide the release name and Helm chart location.

Thats all for this and create the pipeline.

The pipeline will fail because there are more steps to follow later, but don't worry about it. Once we complete the setup, we can rerun the pipeline.

Cluster access from CodePipeline

This is really an important part, let’s breakdown the changes that we are going to do.

• Since we are using an EKS cluster with a private API server endpoint, we need to update the existing CodePipeline role to grant permissions for ec2:CreateNetworkInterface, ec2:CreateNetworkInterfacePermission, ec2:DeleteNetworkInterface, and some ec2:Describe permissions.

If your cluster endpoint is public, you can skip this step.

• The CodePipeline service role needs to authenticate and authorize access permissions through EKS access entries. Additionally, the CodePipeline service role should have the eks:DescribeCluster permission to access the cluster.

Update the CodePipeline service role

Find your CodePipeline service role from the IAM roles and edit the existing role.

Update the existing policy by adding the following policy document. Make sure to replace the Subnet ARNs and EKS Cluster ARN.

You can find the subnets in the Networking tab of the EKS cluster console.

            "Sid": "EksClusterPolicy",
            "Effect": "Allow",
            "Action": "eks:DescribeCluster",
            "Resource": "arn:aws:eks:ap-southeast-2:111111111111:cluster/awsfanboy-dev"
        },
        {
            "Sid": "EksVpcClusterPolicy",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeDhcpOptions",
                "ec2:DescribeNetworkInterfaces",
                "ec2:DescribeRouteTables",
                "ec2:DescribeSubnets",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeVpcs"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": "ec2:CreateNetworkInterface",
            "Resource": "*",
            "Condition": {
                "StringEqualsIfExists": {
                     "ec2:Subnet": [
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-0c147f0e842c43726",
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-068e82a6b56cc1742",
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-0b12ab5e6baf6028d"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:CreateNetworkInterfacePermission",
            "Resource": "*",
            "Condition": {
                "ArnEquals": {
                    "ec2:Subnet": [
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-0c147f0e842c43726",
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-068e82a6b56cc1742",
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-0b12ab5e6baf6028d"
                    ]
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": "ec2:DeleteNetworkInterface",
            "Resource": "*",
            "Condition": {
                "StringEqualsIfExists": {
                     "ec2:Subnet": [
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-0c147f0e842c43726",
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-068e82a6b56cc1742",
                        "arn:aws:ec2:ap-southeast-2:111111111111:subnet/subnet-0b12ab5e6baf6028d"
                    ]
                }
            }
        }

Update the CodePipeline service role to access ECR.

Add the AmazonEC2ContainerRegistryPowerUser AWS managed policy to the CodePipeline service role so that AWS ECR build provider can push the container image to the ECR.

EKS Cluster Authentication and Authorization

We can create this easily via EKS access entries in the EKS Cluster console

Create access entry

Search for your CodePipeline service role and select it, then keep the type as Standard and click Next.

Now we need to attach the policy to authorize permissions for the CodePipeline service role. For this demo, I am using AmazonEKSClusterAdminPolicy. Keep the Access scope as Cluster, then click Add policy, followed by clicking Next to create the access entry.

That’s it now we can go ahead and rerun the pipeline.

Run the Pipeline

Navigate to the pipeline and click release pipeline.

If all good, you will see your pipeline executed successfully.

You can expand the deploy action and check the logs.

Verify the deployment

Since this is a private cluster, I am using the CloudShell with a VPC environment , where which is created in the same VPC where I have deployed the EKS cluster. So I can run kubectl and helm commands. We don’t have to install kubectl but helm. Once done update the kubeconfig

~ $ aws eks update-kubeconfig --name awsfanboy-dev

Install helm:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 > get_helm.sh
chmod 700 get_helm.sh
./get_helm.sh

List the Helm release and get the ingress URL.

You can also list the Ingress URL from the EKS console by navigating to the Resources tab and selecting the Ingress object.

Copy the Load Balancer URL and access it from your browser. You should see that our Doggo App Version 1 is running.

Eager to learn and curious about the deployment action for a private cluster, I looked at the CloudTrail logs and found the trail logs where userIdentity from CodePipeline appeared for NetworkInterface events.

Conclusion

Previously, for EKS deployment, we had to use CodeBuild with a lot of configurations to manage Kubernetes deployments. But now, with native CodePipeline support, we can easily and directly deploy to EKS clusters. My favourite part is that we can deploy to an EKS cluster with a private endpoint without any customizations, except for the IAM policy, which is really awesome.

If you have any questions or comments, please let me know.

Amazon Q: https://aws.amazon.com/q/ AWS Infrastructure Composer: https://aws.amazon.com/infrastructure-composer/

Problem:

Let's discuss the problem I'm attempting to tackle. IP exhaustion, which occurs when given subnets run out of IPs, is a problem that may arise if you are using Amazon EKS and your workload is growing.

Unless you have IPAM, AWS CloudWatch metrics do not support them at the time I am writing this blog. Monitoring your available IP addresses in subnets without the use of IPAM is what I'm attempting to accomplish here.

Solution:

AWS Services involved in this solution:

• AWS Lambda

• Event Bridge Scheduler

• Amazon CloudWatch Metrics

• Amazon CloudWatch Alarm

• Amazon SNS

Lambda Function

I was able to create this in a matter of minutes with the help of Amazon Q Developer, however, I obviously needed to make a few little adjustments. This is very beneficial if you understand the basics and what you are doing. Instead of configuring AWS services blindly, I recommend everyone to better understand AWS services.

Full Python Script here:

import boto3
import os
from botocore.exceptions import ClientError

def lambda_handler(event, context):
    vpc_id = os.environ['VPC_ID']
    subnet_ids = os.environ['SUBNET_IDS'].split(',')
    namespace = os.environ['NAMESPACE']

    ec2 = boto3.client('ec2')
    cloudwatch = boto3.client('cloudwatch')

    try:
        response = ec2.describe_subnets(
            Filters=[
                {'Name': 'vpc-id', 'Values': [vpc_id]},
                {'Name': 'subnet-id', 'Values': subnet_ids}
            ]
        )

        for subnet in response['Subnets']:
            subnet_id = subnet['SubnetId']
            available_ip_count = subnet['AvailableIpAddressCount']
            cidr_block = subnet['CidrBlock']
            total_ip_count = 2 ** (32 - int(cidr_block.split('/')[1])) - 5  # Subtract 5 for reserved IPs

            subnet_name = subnet_id  # Default to subnet ID if no name tag
            for tag in subnet.get('Tags', []):
                if tag['Key'] == 'Name':
                    subnet_name = tag['Value']
                    break

            utilization_percentage = ((total_ip_count - available_ip_count) / total_ip_count) * 100

            # Send metrics to CloudWatch
            cloudwatch.put_metric_data(
                Namespace=namespace,
                MetricData=[
                    {
                        'MetricName': 'AvailableIPAddresses',
                        'Dimensions': [
                            {'Name': 'SubnetName', 'Value': subnet_name},
                            {'Name': 'SubnetId', 'Value': subnet_id}
                        ],
                        'Value': available_ip_count,
                        'Unit': 'Count'
                    },
                    {
                        'MetricName': 'IPUtilizationPercentage',
                        'Dimensions': [
                            {'Name': 'SubnetName', 'Value': subnet_name},
                            {'Name': 'SubnetId', 'Value': subnet_id}
                        ],
                        'Value': utilization_percentage,
                        'Unit': 'Percent'
                    }
                ]
            )

            print(f"Metrics sent for Subnet: {subnet_name} (ID: {subnet_id})")

    except ClientError as e:
        print(f"An error occurred: {e}")
        return {
            'statusCode': 500,
            'body': str(e)
        }

    return {
        'statusCode': 200,
        'body': 'Subnet monitoring completed'
    }

Get IP address utilization:

Send metrics to CloudWatch:

Use AWS Infrastructure Composer to design the infrastructure.

This further enables you design your infrastructure visually, generate Infrastructure as Code and deploy it using AWS SAM (AWS Serverless Application Model) https://aws.amazon.com/serverless/sam/.

How to Deploy

Prerequisites

• AWS CLI installed and configured with appropriate permissions

• AWS Toolkit for Visual Studio Code installed and configured

• AWS SAM CLI installed

Deployment Steps

Repository for entire code and instructions on how to deploy: https://github.com/awsfanboy/aws-subnet-ip-address-utilization-monitor

• Modify the template.yaml file to adjust default parameter values or add/remove resources as needed. eg: VPC ID, Subnet Name, Subnet ID, CloudWatch Metric Namespace.

• (Optional) Update the lambda_function.py file in the src directory.

• Build the SAM application: sam build

• Deploy the SAM application: sam deploy --guided

• This will start an interactive deployment process. You'll be prompted to provide values for the parameters defined in the template. You can accept the default values or provide your own.

• During the deployment, you'll be asked to confirm the creation of IAM roles and the changes to be applied. Review and confirm these.

• SAM will output the ARNs of the created Lambda function and SNS topic once the deployment is complete.

Parameters

    VpcId: The ID of the VPC to monitor
    SubnetIds: Comma-separated list of subnet IDs to monitor
    SubnetName1: Name of the first subnet
    SubnetName2: Name of the second subnet
    CWMetericNamespace: The CloudWatch metric namespace
    AlertEmail: Email address to receive alerts

Resources Created

• Lambda function for monitoring subnets

• EventBridge rule to trigger the Lambda function every minute

• SNS topic for sending alerts

• CloudWatch alarms for each monitored subnet

Customization

• To monitor more than two subnets, duplicate the SubnetUtilizationAlarm resource in the template and adjust the SubnetIds parameter.

• Modify the Lambda function code in src/lambda_function.py to implement your specific monitoring logic.

• Adjust the alarm thresholds and evaluation periods in the SubnetUtilizationAlarm resources as needed.

Cleanup

• To remove all resources created by this stack: sam delete

• Follow the prompts to confirm the deletion of resources.

Demo

I have an Amazon EKS cluster running a deployment with 6 replicas. Worker nodes are running on 2 Subnets. IP address utilization is looking good.

The alarm state is OK.

Okay! let's increase the number of replicas from 6 to 600.

Let's check metrics from the CloudWatch and ooops! now we can see that IP utilization is high.

Now, let's check the Alarms in the CloudWatch. Now the state changed from OK to ALARM state.

Let's check my emails

I can see there are 2 emails in my inbox.

Cost

I calculated the cost using calculator.aws, and it appears to be not bad though.

What Next?

These notifications can be sent to Slack, PagerDuty, and other platforms.

Conclusion

I hope my automation will help someone who doesn't want to use IPAM to monitor IP address utilization in subnets, and I truly wish we could access these metrics straight from CloudWatch.

If you have any suggestions for improvement or if you would like to use anything you currently have in a different way, please feel free to share.

During AWS re:Invent 2024, one of my favorite announcements was Amazon EKS Auto Mode. As a huge AWSFanBoy, I was very impressed and decided to try out.

In this blog post, I explain what AWS offers in EKS, the challenges of managing an EKS cluster, and what is Amazon EKS Auto Mode is. I aim to keep it simple for easy understanding. Don't worry, I've included a link to a demo that Jones Zachariah Noel N and I did on our podcast The Zacs' Show Talking AWS. I've also added a video showing a cluster upgrade with EKS Auto Mode.

Before we dive into EKS Auto Mode, let's quickly review the previous EKS offerings.

EKS Cluster Architecture and Responsibility Model without EKS Auto Mode

Disclaimer: I have included some screenshots from the official re:Invent slide deck, which is publicly available here - Link

• AWS manages the control plane, etcd instances.

• Customers have to manage add-ons, worker nodes, and other services like load balancers for ingress.

• Without the core add-ons, we cannot run any workloads in a newly created EKS cluster.

• Customers are responsible for node patching, node scaling, add-on updates and more.

Some of the challenges with managing an Amazon EKS cluster

Deploying an EKS cluster is not enough to run your workloads; you need at least these core add-ons: Kubeproxy, CoreDNS, and VPC CNI to have an application-ready cluster. Keeping these add-ons up to date and managing them also requires time and effort from engineers.

Managing compute resources is one of the biggest challenges. The most commonly used add-ons for auto-scaling nodes are Cluster Autoscaler or Karpenter. However, managing another add-on requires additional effort and time.

When you have multiple EKS clusters for internal developer teams or customers, upgrading them can be challenging. We need to ensure that core add-ons are updated along with the clusters, check for API deprecations, and more.

After upgrading the clusters, we need to gradually update the nodes and remove the old ones. Additionally, when AWS releases a new AMI for the nodes, we must ensure the clusters use nodes with the latest AMI. This is important for applying security patches as well.

What is Amazon EKS Auto Mode

Now we can create application-ready EKS clusters with essential Kubernetes capabilities. AWS handles all the complex aspects of managing your Kubernetes setup - including computing, storage, and networking.

Amazon EKS Cluster Architecture and Responsibility Model for an EKS Cluster with EKS Auto Mode

With EKS Auto Mode enabled, AWS now manages the Cluster EC2 instances and Cluster Capabilities. Compared to the previous image, it's clear that customers have fewer responsibilities. Let me explain the Cluster EC2 instances and Cluster Capabilities.

Cluster EC2 instances

In EKS Auto Mode Node autoscaling is managed by Karpenter. Karpenter is excellent for node scaling. It is not only auto-scales nodes but also finds the right and optimized node for your workload, helping to save costs. We will discuss Karpenter separately in another article. If you are not familiar with Karpenter, you can read more about it here. All the instances launched by EKS Auto Mode run Bottlerocket OS.

Cluster Capabilities

Networking

There are three essential add-ons under Networking that are managed by AWS.

• Kubeproxy

• CoreDNS

• VPC CNI

So we no longer need to manage these add-ons. These add-ons run as systemd services on the nodes, so you won't see any of these pods running on the cluster. You'll see more in the demo later.

Listing pods in all namespaces, I don't see any of the pods running.

Load balancing

AWS manages the LB controller. For your ingress resource, you can use an ALB, and for services, you can use a NLB. You don’t have to install these controllers since AWS is managing them for you; you just need to annotate your service/ingress.

Storage

With EKS Auto Mode, the EBS CSI driver comes included, allowing us to set up EBS storage for pods that need persistent volumes.

Try it out yourself in just a few minutes.

I am using eksctl command to deploy an EKS cluster, which is really easy since it will deploy the dependancies for you cluster such as networking resource as VPC, Subnets etc.

1. Install eksctl - https://eksctl.io/installation/

2. Deploy an EKS cluster with EKS Auto Mode

   • eksctl create cluster --name=hogwarts --enable-auto-mode --version=1.30

3. Once cluster is created successfully, run the following commands

   • List all the pods in all namespaces - kubectl get pods -A

   • List all the nodes - kubectl get nodes

   • You will not see any resources

4. Now lets check the manifest file that you are going to deploy in the manifest.yaml file

   • This manifest file will deploy

     • Deployment

     • Pod Disruption Budget

     • Service

     • IngressClass

     • Ingress

         ---
         apiVersion: apps/v1
         kind: Deployment
         metadata:
           name: zacs-show-deployment
           labels:
             app: zacs
         spec:
           replicas: 36
           selector:
             matchLabels:
               app: zacs
           template:
             metadata:
               labels:
                 app: zacs
             spec:
               containers:
               - name: zacs
                 image: awsfanboy/doggo-app
                 ports:
                 - containerPort: 80
         ---
         apiVersion: policy/v1
         kind: PodDisruptionBudget
         metadata:
           name: zacs-pdb
         spec:
           minAvailable: 80%
           selector:
             matchLabels:
               app: zacs
         ---
         apiVersion: v1
         kind: Service
         metadata:
           name: zacs-show-service
         spec:
           selector:
             app: zacs
           ports:
             - protocol: TCP
               port: 80
               targetPort: 80
           type: NodePort
         ---
         apiVersion: networking.k8s.io/v1
         kind: IngressClass
         metadata:
           name: eks-auto-alb
         spec:
           controller: eks.amazonaws.com/alb
         ---
         apiVersion: networking.k8s.io/v1
         kind: Ingress
         metadata:
           name: zacs-ingress
           annotations:
             alb.ingress.kubernetes.io/target-type: ip
             alb.ingress.kubernetes.io/scheme: internet-facing
         spec:
           ingressClassName: eks-auto-alb
           rules:
             - http:
                 paths:
                   - path: /
                     pathType: Prefix
                     backend:
                       service:
                         name: zacs-show-service
                         port:
                           number: 80
       

5. Deploy it by running kubectl apply -f manifest.yaml.

   • 

6. Let's list all the resources by running kubectl get all.

   • 

7. Check the nodes and nodeclaims - kubectl get nodes , kubectl get nodeclaims

   • 

     Karpenter launched a new node for my workload.

8. AWS manages the compute, as I mentioned earlier. You can find the API resources by running the command kubectl api-resources | grep karpenter.

   • 

9. Get the ingress URL by running kubectl get ingress.

   • 

10. Allow some time for the DNS propagation and the creation of the ALB, then access the ALB URL in your browser.

    • Voila! You can see my doggo app is running.

    • 

11. See, that's it. We didn't install any add-ons, and within a few minutes, you have a cluster ready to run your application.

12. To cleanup run eksctl delete cluster --name=hogwarts

Cluster Upgrades

I have included a demo where I show an EKS cluster with Kubernetes version 1.30 and upgrade it to version 1.31.

Here is the podcast where Jones and myself did a deep dive into Amazon EKS Auto Mode on our show, "The Zacs' Show Talking AWS"

Conclusion

Amazon EKS Auto Mode reduces operational tasks like managing core add-ons, patching nodes, dynamic scaling, and cluster upgrades. You can create a cluster anytime, and if you want to run tests, EKS Auto Mode helps by setting up an application-ready cluster in just a few minutes. Currently, only the main core add-ons are included by default, but I really wish, more add-ons will be available in the future. However, you can still install and configure other add-ons on an EKS Auto Mode cluster.

Whats next ?

I will publish another article on how to migrate an existing EKS cluster to an EKS Auto Mode cluster.

Useful links

• https://www.youtube.com/watch?v=a_aDPo9oTMo

• https://docs.aws.amazon.com/eks/latest/userguide/automode.html

• https://aws.amazon.com/eks/auto-mode/