Fixing Permission Denied Deployments to EKS from GitLab CI/CD with IRSA

Tech Note: Always backup your configuration files before applying any changes to production environments.

Fixing Permission Denied Deployments to EKS from GitLab CI/CD with IRSA: A Comprehensive Guide

As a Senior Cloud Solution Architect and Software Engineer, I frequently encounter challenges related to secure and efficient deployments in cloud-native environments. One common hurdle is the "Permission Denied" error during deployments to Amazon EKS clusters orchestrated by GitLab CI/CD pipelines, especially when leveraging IAM Roles for Service Accounts (IRSA). This guide provides a deep dive into diagnosing, troubleshooting, and resolving such issues, ensuring your CI/CD pipelines run smoothly and securely.

Understanding IRSA and GitLab CI/CD Integration

IAM Roles for Service Accounts (IRSA) is a critical AWS feature that allows Kubernetes service accounts to assume IAM roles. This provides fine-grained, secure access to AWS services from pods running within your EKS cluster, eliminating the need for long-lived AWS credentials. When integrating with GitLab CI/CD, your pipeline typically uses a Kubernetes context to interact with the EKS cluster, often configured to use a specific service account that, in turn, is associated with an IAM role via IRSA.

How it Works (Simplified)

  • GitLab CI/CD Runner: Deploys kubectl commands to the EKS cluster.
  • Kubernetes Service Account: The runner authenticates with EKS using a Kubeconfig that points to a specific Kubernetes Service Account (SA).
  • IRSA Association: This SA is annotated with an AWS IAM Role ARN (eks.amazonaws.com/role-arn).
  • AWS STS: When a pod (or in this case, the context for the CI/CD runner interacting with the cluster) uses this SA, AWS STS exchanges the Kubernetes token for temporary AWS credentials from the associated IAM Role.
  • AWS API Calls: These temporary credentials are then used to make API calls to AWS services (e.g., managing EC2 instances, S3 buckets, or in this context, EKS itself for deploying resources).

Symptom Analysis & Root Causes

"Permission Denied" errors manifest in various ways, often indicating that the IAM role associated with your EKS service account lacks the necessary permissions to perform specific actions on Kubernetes resources or AWS services.

Common Symptoms

  • Error from server (Forbidden): deployments.apps "my-app" is forbidden: User "arn:aws:iam::xxxxxxxxxxxx:user/i-am-role-name" cannot create/get/update deployments.apps in namespace "my-namespace"
  • kubectl apply -f my-manifest.yaml fails with permission errors.
  • Pipeline logs show errors like User: arn:aws:sts::xxxxxxxxxxxx:assumed-role/MyGitLabIRSARole/i-xxxxxxxxxxxx is not authorized to perform: eks:DescribeCluster on resource: arn:aws:eks:REGION:ACCOUNT_ID:cluster/my-cluster
  • Helm deployments failing with similar authorization errors.

Typical Root Causes

  • Incorrect IAM Role Permissions: The most common issue. The IAM role linked to your Kubernetes service account does not have the necessary AWS IAM policies attached to allow actions on EKS or other AWS resources.
  • Missing Kubernetes RBAC Permissions: Even if the IAM role is correct, the Kubernetes Service Account itself might not have the necessary Role-based Access Control (RBAC) permissions (Roles and RoleBindings) within the EKS cluster to create, update, or delete Kubernetes resources.
  • IAM Role Trust Policy Misconfiguration: The trust policy of the IAM role might not allow the EKS OIDC provider to assume the role. This prevents IRSA from working entirely.
  • Service Account Annotation Errors: The Kubernetes service account might not be correctly annotated with the eks.amazonaws.com/role-arn pointing to the correct IAM role.
  • OIDC Provider Not Configured or Mismatched: The EKS cluster's OIDC Identity Provider (IdP) might not be created or its ARN might not match in the IAM role's trust policy.
  • Kubeconfig Context Issues: The GitLab CI/CD runner's kubeconfig might not be pointing to the correct cluster, context, or user (service account).
  • Namespace Mismatch: Trying to deploy to a namespace where the service account lacks permissions, or the service account itself exists in a different namespace than intended.
  • AWS STS Throttling: Less common, but possible with very high request volumes, leading to temporary credential acquisition failures.

Step-by-Step Resolution Guide

Follow these steps to systematically diagnose and resolve "Permission Denied" errors for EKS deployments from GitLab CI/CD using IRSA.

Step 1: Verify EKS OIDC Identity Provider Configuration

Ensure your EKS cluster has an OIDC Identity Provider configured and that it's accessible. This is fundamental for IRSA to function.

aws eks describe-cluster --name your-cluster-name --query "cluster.identity.oidc.issuer" --output text

This command should return the OIDC issuer URL. If not, you need to create one:

eksctl utils associate-iam-oidc-provider --region your-region --cluster your-cluster-name --approve

Step 2: Inspect the Kubernetes Service Account

Check if the Kubernetes Service Account (SA) used by your GitLab CI/CD pipeline is correctly annotated with the IAM Role ARN.

kubectl get sa your-service-account-name -n your-namespace -o yaml

Look for the eks.amazonaws.com/role-arn annotation. It should point to the full ARN of the IAM role.

apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNT_ID:role/YourGitLabIRSARole name: your-service-account-name namespace: your-namespace # ...

If the annotation is missing or incorrect, update your service account manifest and re-apply:

kubectl apply -f your-service-account.yaml

Step 3: Review IAM Role Trust Policy

The IAM role must trust the EKS OIDC provider to allow assuming the role.

aws iam get-role --role-name YourGitLabIRSARole --query 'Role.AssumeRolePolicyDocument'

The output should contain a trust policy similar to this, replacing OIDC_ISSUER_URL and AWS_ACCOUNT_ID with your specific values:

{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/OIDC_ISSUER_URL" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "OIDC_ISSUER_URL:sub": "system:serviceaccount:your-namespace:your-service-account-name" } } } ] }

Key elements to check:

  • "Federated": "arn:aws:iam::ACCOUNT_ID:oidc-provider/OIDC_ISSUER_URL" - Must match your EKS OIDC provider.
  • "OIDC_ISSUER_URL:sub": "system:serviceaccount:your-namespace:your-service-account-name" - The sub claim must precisely match your service account and namespace.

If incorrect, update the trust policy using the AWS CLI or console.

Step 4: Validate IAM Role Permissions (Managed Policies & Inline Policies)

Ensure the IAM role has the necessary permissions to interact with EKS and deploy resources. This is where most "Permission Denied" issues originate.

Identify the specific actions that are failing from your pipeline logs. For example, if it's failing to create deployments, it needs appropriate Kubernetes API permissions.

aws iam list-attached-role-policies --role-name YourGitLabIRSARole aws iam list-role-policies --role-name YourGitLabIRSARole

Examine each policy. Common permissions required for deployments include:

  • AmazonEKSClusterPolicy (Managed Policy) for basic EKS cluster interaction.
  • Custom policies allowing actions like:
    • "eks:DescribeCluster"
    • "ecr:GetAuthorizationToken", "ecr:BatchCheckLayerAvailability", "ecr:GetDownloadUrlForLayer", "ecr:BatchGetImage" (for pulling images from ECR).
    • Specific Kubernetes API actions depending on what your application deploys (e.g., "kubernetes:CreatePod", "kubernetes:GetDeployment", "kubernetes:UpdateService"). For general deployment, consider specific actions rather than broad "kubernetes:*".

Update the IAM role policies to grant the missing permissions. Always follow the principle of least privilege.

Step 5: Verify Kubernetes RBAC Permissions (Roles & RoleBindings)

Even with correct IRSA configuration, the Kubernetes Service Account itself needs appropriate RBAC permissions within the cluster.

Check the Role and RoleBinding associated with your Service Account in the target namespace.

kubectl get role -n your-namespace kubectl get rolebinding -n your-namespace

Example of a Role and RoleBinding for a deployer Service Account:

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: deployer-role namespace: your-namespace rules: - apiGroups: ["", "apps", "extensions"] resources: ["deployments", "services", "pods", "replicasets", "configmaps", "secrets", "ingresses"] verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: deployer-rolebinding namespace: your-namespace subjects: - kind: ServiceAccount name: your-service-account-name namespace: your-namespace roleRef: kind: Role name: deployer-role apiGroup: rbac.authorization.k8s.io

Ensure the Role grants the necessary verbs on the correct resources in the target apiGroups, and that the RoleBinding links this Role to your ServiceAccount.

Step 6: Debugging in GitLab CI/CD

If all configurations seem correct, enable verbose logging in your GitLab CI/CD pipeline steps that interact with EKS.

  • Add --v=6 or --v=8 to kubectl commands to get detailed output.
  • Ensure environment variables like KUBECONFIG are correctly set within your CI/CD job.
  • Temporarily try to explicitly set AWS credentials in a test job (for debugging only, do not use in production) to rule out IRSA specific issues vs. general IAM permission issues.
# Example .gitlab-ci.yml snippet for debugging deploy_job: image: name: registry.gitlab.com/gitlab-org/cloud-deploy/kubectl:latest entrypoint: ["/bin/bash", "-c", "trap 'exit ${ret}' EXIT; set -eo pipefail"] script: - export KUBECONFIG=$KUBE_CONFIG # Ensure KUBECONFIG is correctly set if you manage it - # If using GitLab's EKS integration, the context might be set automatically - kubectl config use-context "${CI_KUBERNETES_CLUSTER_NAME}" || true # Example: adapt to your setup - kubectl get pods -n your-namespace --v=8 - kubectl apply -f deployment.yaml -n your-namespace --v=8

Best Practices for Prevention & Performance Optimization

Adopting these best practices will minimize permission-related issues and enhance the security and efficiency of your EKS deployments.

  • Least Privilege Principle: Always grant the minimum necessary permissions to your IAM roles and Kubernetes service accounts. Regularly review and audit these permissions.
  • Automate IRSA Configuration: Use tools like eksctl or Terraform/CloudFormation to provision and manage your EKS OIDC provider, IAM roles, and service account associations. This reduces manual errors and ensures consistency.
  • Dedicated Service Accounts: Create specific Kubernetes Service Accounts for each application or deployment pipeline, rather than using a general-purpose SA. Each SA should ideally have its own IRSA-linked IAM role.
  • Namespace-Scoped RBAC: Define Kubernetes Roles and RoleBindings at the namespace level where possible, rather than using ClusterRoles. This limits the blast radius of any permission misconfiguration to a single namespace.
  • Version Control All Configurations: Store all your Kubernetes manifests (Service Accounts, Roles, RoleBindings) and infrastructure-as-code (IAM roles, OIDC providers) in Git. This allows for change tracking, peer review, and easy rollbacks.
  • Regular Audits: Periodically audit your IAM policies and Kubernetes RBAC configurations to identify overly permissive settings or stale permissions that are no longer needed.
  • Centralized Logging and Monitoring: Ensure comprehensive logging from your EKS cluster (CloudWatch Logs, Kubernetes audit logs) and GitLab CI/CD pipelines. Integrate with monitoring tools to quickly identify and alert on deployment failures and permission issues.
  • GitLab CI/CD Variables for Secrets: Store sensitive information like AWS region, cluster names, and Kubeconfig details as protected GitLab CI/CD variables, not directly in your .gitlab-ci.yml, for enhanced security.

Frequently Asked Questions

Q1: How can I test IAM role permissions without running the entire GitLab CI/CD pipeline?

A1: You can temporarily configure your local kubectl to use the EKS service account and its associated IAM role. First, get the token for the service account:

TOKEN=$(kubectl create token your-service-account-name -n your-namespace --duration=20m)

Then, use a tool like aws-iam-authenticator or directly modify your kubeconfig to use this token for a specific user/context, and then attempt the failing kubectl commands. Alternatively, for just verifying the IAM role itself (without Kubernetes RBAC), you can use aws sts assume-role-with-web-identity with a dummy token in a local script and then execute AWS CLI commands with the returned temporary credentials.

Q2: My deployments worked before, but now they are failing with "Permission Denied". What changed?

A2: This often indicates a recent change to one of the permission-related components. Common culprits include:

  • IAM Policy Updates: Someone might have modified or detached a policy from your IAM role.
  • Kubernetes RBAC Changes: A Role or RoleBinding might have been updated or deleted.
  • EKS Cluster Upgrade: EKS upgrades can sometimes require adjustments to OIDC or IAM configurations, though this is less common directly for IRSA unless the OIDC provider URL changes.
  • Service Account Manifest Changes: The eks.amazonaws.com/role-arn annotation might have been accidentally removed or altered.
  • Application Requirements: The application you're deploying might now require new permissions (e.g., access to a new AWS service or a new Kubernetes resource type) that haven't been granted.

Review recent changes in your version control system for all related configurations (IAM, Kubernetes manifests, CI/CD pipeline definitions).

Q3: Can I use a single IAM role for multiple service accounts?

A3: Yes, you can. The IAM role's trust policy can be configured to allow multiple service accounts to assume it by using a wildcard or listing multiple StringEquals conditions for the "OIDC_ISSUER_URL:sub" claim (e.g., "OIDC_ISSUER_URL:sub": ["system:serviceaccount:ns1:sa1", "system:serviceaccount:ns2:sa2"]). However, for enhanced security and adherence to the principle of least privilege, it's generally recommended to use dedicated IAM roles for dedicated service accounts, especially across different applications or security boundaries. This approach makes auditing, troubleshooting, and revoking permissions much easier and more granular.

Popular posts from this blog

Debugging ImagePullBackOff in Kubernetes EKS with AWS ECR authentication issues

Debugging ImagePullBackOff in Kubernetes EKS with AWS ECR Authentication Issues Encountering ImagePullBackOff is a common hurdle for developers managing containerized applications on Kubernetes, especially within AWS EKS when images are stored in Amazon Elastic Container Registry (ECR). This error signifies that Kubernetes could not pull a required container image, frequently stemming from underlying authentication and authorization problems with ECR. A robust cloud hosting server environment like EKS demands precise configuration for seamless operation. This guide will walk through diagnosing and resolving these authentication issues, ensuring your applications deploy smoothly on your scalable cloud infrastructure . Brief Introduction & Symptom Analysis The ImagePullBackOff status indicates that a Kubernetes Pod is repeatedly failing to pull an image. When combined with AWS ECR, this often points to EKS worker nodes lacking the necess...
Read more

Fixing EKS Pod CrashLoopBackOff Due to Readiness Probe Failures

Tech Note: Always backup your configuration files before applying any changes to production environments. Fixing EKS Pod CrashLoopBackOff Due to Readiness Probe Failures In modern cloud-native architectures, Kubernetes (K8s) is the de-facto standard for container orchestration, with Amazon Elastic Kubernetes Service (EKS) being a popular choice for managed K8s on AWS. However, even in highly resilient environments, pods can sometimes enter a CrashLoopBackOff state. One of the most common and often misunderstood reasons for this is a failing readiness probe . This comprehensive guide, crafted by a Senior Cloud Solution Architect and Software Engineer, will demystify readiness probe failures on EKS, provide a structured troubleshooting methodology, and offer best practices for prevention. Symptom Analysis & Root Causes Understanding the symptoms and underlying causes is the first critical step in resolving any complex issue. A CrashLoopBackOff state in...
Read more

Resolve Nginx `upstream prematurely closed connection` with SSL termination for Docker containers

Resolve Nginx `upstream prematurely closed connection` with SSL termination for Docker containers Encountering the Nginx error `upstream prematurely closed connection` when running applications within Docker containers, especially with SSL termination configured, can be a frustrating experience for any Cloud Engineer or developer. This guide provides a comprehensive, step-by-step approach to diagnose and resolve this common issue, ensuring your services remain robust and highly available on any cloud hosting server . Understanding and mitigating this error is crucial for maintaining a reliable scalable cloud infrastructure and an efficient VPS server management strategy. Brief Introduction & Symptom Analysis The `upstream prematurely closed connection` error indicates that Nginx, acting as a reverse proxy, initiated a connection to an upstream server (your Dockerized application) but the upstream closed the connection before Nginx expec...
Read more